Maintenance

While GoAccess provides a shed load of detail about a site's monthly performance it's frequently the case that we need more cumulative stats - how many hits did the site get least year? how about compared to the year before? how many hits so far this year?

So, I've been working on an 'executive overview' kind of representation of site data using XSLT and I've been having an interesting problem. Saxon errors out on some files with:
Failed to read input file file:/path/to/file.html (java.nio.charset.MalformedInputException): Input length = 1

Looking at a such a file using file --mime-encoding /path/to/file.html produces something like 'file.html: us-ascii'

I suspect that the output is more a function of the characters that the 'file' application finds in a file. It's unlikely that GoAccess is intentionally encoding the files this way (reports usually come up 'us-ascii', but sometimes it's 'utf-8', sometimes it's 'iso-8859-1', and sometimes it's the literally unidentifiable 'unknown-8bit').

Looking at the specific point in the file that causes grief it looks like it's generally the result of intentionally malformed requests. GoAccess blithely includes the funky chars and XSLT coughs up a hairball.

So it looks like XSLT is out and I'll need to find another way of processing these files.

I edited my cdmirror script to accommodate building a PXE-boot bundle using a configurable target distro (it used to only build the current LTS, but now I can set it to build any distro) so I can now do tests on PXE builds of the soon-to-be-released 20.04.

It looks like the apt-mirror script project is dead. The repo directory structure has changed since 2018 and when I added 20.04 to the mirror it did not download a bunch of stuff.

So, I'm testing debmirror. Which also has problems, but is still being developed. The version of Debian I'm using on guava uses an old version of debmirror, so I grabbed the script off of the git repo and installed it locally.

Along with a wrapper script it seems to be much easier to use than apt-mirror was. Fingers crossed it works well when I start to run it as a cron job.

With the new mirror I've had an initial success with a 20.04 PXE build of one of our new lab NUCs.

Migrated first JSRDocs_dev then JSRDocs forward to the new AdaptiveDB codebase, and tested. All seems good. That's the last of the important database projects that I'm aware of that will need porting. Backup of the db taken today; backups stored both locally and in the folder of the live site.

So far I've updated both VPN dbs and all four active Landscapes dbs to the latest codebase, meaning that they will run on both PHP 5.6 and PHP 7. For VPN and the Powell St db, I've put the following stuff in .htaccess files to ensure that we're actually running on the new server and taking advantage of PHP 7, because the speed payoff is significant:

setEnv UVPHP_VERSION 70

RewriteEngine On
RewriteBase /
RewriteCond     %{HTTP_HOST} ^hcmc\.uvic\.ca$ [NC]
RewriteRule     (.*)  https://webserver2.hcmc.uvic.ca%{REQUEST_URI}

Once the new server takes over from the old, we'll have to remove the redirects. I have one more database to port, the jsrDocs one.

Now that the required ports are opened up and working for both servers, I decided to see how straightforward it would be to get Let's Encrypt certs set up for the subdomains. The answer turns out to be that it's trivial:

  sudo add-apt-repository ppa:certbot/certbot
  sudo apt install python-certbot-apache
  sudo certbot --apache -d teijenkins.hcmc.uvic.ca
  sudo certbot renew --dry-run

The certbot did an automated challenge by temporarily tweaking the apache config and testing the tweak; then it downloaded and installed the certs, and even updated the virtual hosts file to point at them. The last line tests the renewal process, which has to happen every three months, and for which certbot has installed a cron job. If that fails, you get an email.

Spent a lot of the day on minor tweaks to the two Jenkins servers. The https problem on jenkins.hcmc.uvic.ca was a VLAN ACL issue, now fixed by NETS, so it's working OK now. Both machines now have 8GB of RAM, and the disk of the TEI Jenkins has been expanded by 50GB, so that should last for a little while. I haven't yet set up the Apache config on teiJenkins because I wanted to use the opportunity to document it properly, but it's now trying to build jobs (largely failing for various reasons relating to config).

These are my ongoing notes on how to get Jenkins up and running on a headless Ubuntu 18.04.

• Initial install: make sure you configure LVM to provide a big enough volume for what Jenkins needs in the root filesystem. By default, the Ubuntu installer will create a volume of only 4GB; if that happens, extend it as needed from the remaining free space.
• OPTIONAL: Get rid of cloud-init; it's installed by default but I see no value in it:

      wait until the VM boots
      login
      echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
      sudo apt-get purge cloud-init
      sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
      reboot
  

• Reconfigure the timezone to America/Vancouver, since it may be UTC by default:

  sudo dpkg-reconfigure tzdata

• You must

  sudo apt-add-repository universe

  before installing Jenkins, because it requires a package called daemon which is in that repo. Other stuff needs it too.

• sudo apt-get install ntp

• Install subversion. Although Jenkins has its Java implementation, some of our jobs use the command line app to get info.
• You must install Java 8 (openjdk-8-jdk), which is in the universe repo.
• Install jing. This will probably install a newer Java and make it the default, so

  sudo update-alternatives --config java

  , and set it back to 8.
• Install ant, ant-contrib, and libxml2-utils.
• Install linkchecker.
• Jenkins setup requires that you configure a relative path to the files. If you're runing a vm and proxying the port to your own machine, beware: don't choose the selected port on which you're viewing Jenkins. Instead, choose the port that you know Jenkins is running on IN THE VM. If you screw this up, you can fix it by editing jenkins.model.JenkinsLocationConfiguration.xml in /var/lib/jenkins.
• Install sendmail so Jenkins can send email.
• When setting up Jenkins, accept the suggested package of general-use plugins.
• In Jenkins, install log parser plugin and the ThinBackup plugin.
• In the Jenkins configuration, set the svn version to 1.7 (matches our revision server).
• In the Jenkins global security settings, choose "Project-based Matrix Authorization Strategy" and allow anonymous users to read the appropriate stuff.
• Once Jenkins set up and running, edit /etc/default/jenkins to add an appropriate value for the CSP policy. For example, for an utterly permissive policy, do this:

  JAVA_ARGS="$JAVA_ARGS -Dhudson.model.DirectoryBrowserSupport.CSP=\"\""

  to set the CSP allowing HTML to work properly. Consult docs on CSP to decide exactly what you need.
• As the jenkins user, create a directory /var/lib/jenkins/backups, then use that in the configuration of the ThinBackup plugin. I use this for the full backup schedule: H 12 * * 6 and this for the differential: H 12 * * 1-5
• For Mariage only: install imagemagick and libvips-tools.
• We usually use a configuration where the job workspaces are inside the job folder itself. To make this work, edit jenkins/config.xml, comment out the workspaceDir line, and replace it:

  <!--<workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>-->
    <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  
  

• For Moses project only:

  sudo apt install python3 python3-pip python3-lxml

  sudo -H pip3 install dicttoxml nltk numpy plotly jupyter

  Then install nltk punkt data:

  sudo python3

  import nltk

  nltk.download('punkt')

• Make sure that the folder nltk_data ended up in /usr/share. If it didn't, copy it there. It seems to end up in the wrong place often.

I'm building a new apt Frankenserver out of spare parts, and have re-documented the process from scratch. Notes are in the utilities repo.

The upgrade for Jenkins on teiJenkins was being kept back, and it turned out this was because Ubuntu 14.04 has Java 7 by default. I added a PPA for Java 8, updated the alternatives (sudo update-alternatives --config java) to point to the new one, and was then able to install Java 8. Following that, the Jenkins update went ahead. I elected to keep my existing config for Jenkins rather than overwrite. It needed a reboot for Apache to find Jenkins again.