Maintenance

New desktop migration process:

• Nuke Windows from new desktop.
• Install Ubuntu and all the software you can remember needing.
• Create a single encrypted partition on the spinning disk. Easy to do now in the Disks app.
• Pull that disk out, and migrate data from existing desktop encrypted drive to it.
• Put the drive back in the new machine.
• Create all required symlinks so that everything sensitive is on the encrypted drive, but the system drive is clean.
• Start work, and fix everything that's broken, install missing things, etc. as you go.

This .htaccess snippet will redirect all traffic to a site to https:

Ubuntu 18.04 seems to have a harder time auto-installing/configuring our office printer. As a result users were unable to perform certain printing tasks, like duplex printing.
It turns out that the auto-config is based on ZeroConf being turned on at the printer AND the computer ends.

In order to fix it, I turned off ZeroConf (Rendezvous/Bonjour) on the printer (which broke the iMacs printer setup) and went through the process of manually configuring the printer settings on the lab machines. I then bundled it in to the hcmc-conf package and rolled it out to the lab.

For reference, here's how I manually configured the printer:

1) Get the right PPD for the printer. I've stored a copy of the current one here
2) In the gnome control centre go to the Printers section and click the "Additional Printer Settings..." button

3) Click the "Add" button

4) Select "Enter URI" from the menu on the left and enter "ipp://printer.hcmc.uvic.ca:631/ipp" in the "Enter device URI" field. Click "Forward"

5) The system will look for drivers and present you with some choices.

6) Select "Provide PPD file" and browse for the PPD file you downloaded in step 1. Click "Forward".

7) Continue through the setup to customize your default print settings.

Migrated first JSRDocs_dev then JSRDocs forward to the new AdaptiveDB codebase, and tested. All seems good. That's the last of the important database projects that I'm aware of that will need porting. Backup of the db taken today; backups stored both locally and in the folder of the live site.

So far I've updated both VPN dbs and all four active Landscapes dbs to the latest codebase, meaning that they will run on both PHP 5.6 and PHP 7. For VPN and the Powell St db, I've put the following stuff in .htaccess files to ensure that we're actually running on the new server and taking advantage of PHP 7, because the speed payoff is significant:

setEnv UVPHP_VERSION 70

RewriteEngine On
RewriteBase /
RewriteCond     %{HTTP_HOST} ^hcmc\.uvic\.ca$ [NC]
RewriteRule     (.*)  https://webserver2.hcmc.uvic.ca%{REQUEST_URI}

Once the new server takes over from the old, we'll have to remove the redirects. I have one more database to port, the jsrDocs one.

My server teijenkins.hcmc.uvic.ca is also proxied as jenkins2.tei-c.org, and it needs a cert for that too. I had assumed that I would need to generate a second certificate, but actually it's easy to add a new domain to the existing cert. The VirtualHost has this:

ServerName teijenkins.hcmc.uvic.ca
ServerAlias *jenkins*.tei-c.org

and all I had to do was this:

sudo certbot --apache -d teijenkins.hcmc.uvic.ca -d jenkins2.tei-c.org

and it regenerated the existing cert with the other domain added to it. This cert runs till December; it's not clear from the feedback whether it will be renewed automatically, or whether I'll have to run certbot manually to get that to happen.

I've now set up mod_rewrite to force https on the Jenkins servers. We'll see if that breaks anything in the builds; my guess is that the occasional call to retrieve a product in a preceding build may fail when it hits the redirect.

Now that the required ports are opened up and working for both servers, I decided to see how straightforward it would be to get Let's Encrypt certs set up for the subdomains. The answer turns out to be that it's trivial:

  sudo add-apt-repository ppa:certbot/certbot
  sudo apt install python-certbot-apache
  sudo certbot --apache -d teijenkins.hcmc.uvic.ca
  sudo certbot renew --dry-run

The certbot did an automated challenge by temporarily tweaking the apache config and testing the tweak; then it downloaded and installed the certs, and even updated the virtual hosts file to point at them. The last line tests the renewal process, which has to happen every three months, and for which certbot has installed a cron job. If that fails, you get an email.

Spent a lot of the day on minor tweaks to the two Jenkins servers. The https problem on jenkins.hcmc.uvic.ca was a VLAN ACL issue, now fixed by NETS, so it's working OK now. Both machines now have 8GB of RAM, and the disk of the TEI Jenkins has been expanded by 50GB, so that should last for a little while. I haven't yet set up the Apache config on teiJenkins because I wanted to use the opportunity to document it properly, but it's now trying to build jobs (largely failing for various reasons relating to config).

These are my ongoing notes on how to get Jenkins up and running on a headless Ubuntu 18.04.

• Initial install: make sure you configure LVM to provide a big enough volume for what Jenkins needs in the root filesystem. By default, the Ubuntu installer will create a volume of only 4GB; if that happens, extend it as needed from the remaining free space.
• OPTIONAL: Get rid of cloud-init; it's installed by default but I see no value in it:

      wait until the VM boots
      login
      echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
      sudo apt-get purge cloud-init
      sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
      reboot
  

• Reconfigure the timezone to America/Vancouver, since it may be UTC by default:

  sudo dpkg-reconfigure tzdata

• You must

  sudo apt-add-repository universe

  before installing Jenkins, because it requires a package called daemon which is in that repo. Other stuff needs it too.

• sudo apt-get install ntp

• Install subversion. Although Jenkins has its Java implementation, some of our jobs use the command line app to get info.
• You must install Java 8 (openjdk-8-jdk), which is in the universe repo.
• Install jing. This will probably install a newer Java and make it the default, so

  sudo update-alternatives --config java

  , and set it back to 8.
• Install ant, ant-contrib, and libxml2-utils.
• Install linkchecker.
• Jenkins setup requires that you configure a relative path to the files. If you're runing a vm and proxying the port to your own machine, beware: don't choose the selected port on which you're viewing Jenkins. Instead, choose the port that you know Jenkins is running on IN THE VM. If you screw this up, you can fix it by editing jenkins.model.JenkinsLocationConfiguration.xml in /var/lib/jenkins.
• Install sendmail so Jenkins can send email.
• When setting up Jenkins, accept the suggested package of general-use plugins.
• In Jenkins, install log parser plugin and the ThinBackup plugin.
• In the Jenkins configuration, set the svn version to 1.7 (matches our revision server).
• In the Jenkins global security settings, choose "Project-based Matrix Authorization Strategy" and allow anonymous users to read the appropriate stuff.
• Once Jenkins set up and running, edit /etc/default/jenkins to add an appropriate value for the CSP policy. For example, for an utterly permissive policy, do this:

  JAVA_ARGS="$JAVA_ARGS -Dhudson.model.DirectoryBrowserSupport.CSP=\"\""

  to set the CSP allowing HTML to work properly. Consult docs on CSP to decide exactly what you need.
• As the jenkins user, create a directory /var/lib/jenkins/backups, then use that in the configuration of the ThinBackup plugin. I use this for the full backup schedule: H 12 * * 6 and this for the differential: H 12 * * 1-5
• For Mariage only: install imagemagick and libvips-tools.
• We usually use a configuration where the job workspaces are inside the job folder itself. To make this work, edit jenkins/config.xml, comment out the workspaceDir line, and replace it:

  <!--<workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>-->
    <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  
  

• For Moses project only:

  sudo apt install python3 python3-pip python3-lxml

  sudo -H pip3 install dicttoxml nltk numpy plotly jupyter

  Then install nltk punkt data:

  sudo python3

  import nltk

  nltk.download('punkt')

• Make sure that the folder nltk_data ended up in /usr/share. If it didn't, copy it there. It seems to end up in the wrong place often.