Maintenance

While GoAccess provides a shed load of detail about a site's monthly performance it's frequently the case that we need more cumulative stats - how many hits did the site get least year? how about compared to the year before? how many hits so far this year?

So, I've been working on an 'executive overview' kind of representation of site data using XSLT and I've been having an interesting problem. Saxon errors out on some files with:
Failed to read input file file:/path/to/file.html (java.nio.charset.MalformedInputException): Input length = 1

Looking at a such a file using file --mime-encoding /path/to/file.html produces something like 'file.html: us-ascii'

I suspect that the output is more a function of the characters that the 'file' application finds in a file. It's unlikely that GoAccess is intentionally encoding the files this way (reports usually come up 'us-ascii', but sometimes it's 'utf-8', sometimes it's 'iso-8859-1', and sometimes it's the literally unidentifiable 'unknown-8bit').

Looking at the specific point in the file that causes grief it looks like it's generally the result of intentionally malformed requests. GoAccess blithely includes the funky chars and XSLT coughs up a hairball.

So it looks like XSLT is out and I'll need to find another way of processing these files.

For the last couple of weeks I've been noticing some very weird behaviour in the two Jenkins build jobs where the svn repo has GitHub externals. The projects simply build all the time, whether or not there's been any change to any of the repos (svn or git).

We can't have that, of course; it's only happening to two repos right now, but it as more mini sub-projects get shared between repos, we'd end up with Jenkins simply queuing builds endlessly.

I'm still researching this, but I suspect it might have something to do with branch information; when (say) the DVPP repo is subscribed to the master branch of the staticSearch git repo, the master branch may not change, but the dev branch might; that means you end up with weird svn info like this (running svn info . in the external directory inside the Keats project):

URL: https://github.com/projectEndings/staticSearch.git/branches/master
Relative URL: ^/branches/master
Repository Root: https://github.com/projectEndings/staticSearch.git
Repository UUID: 04f9d699-67a6-817a-6227-ce57b7db7bff
Revision: 588
Node Kind: directory
Schedule: normal
Last Changed Author: martindholmes
Last Changed Rev: 570
Last Changed Date: 2020-01-13 11:41:56 -0800 (Mon, 13 Jan 2020)

You'll see that there are two revision numbers; the last-changed is 570, but the current revision is 588. Presumably that's because the master branch last changed in 570 but the dev branch changed more recently (as you'd expect). I have a suspicion that the svn polling mechanism in Jenkins interprets this to mean that there are new changes, and does a build. But it may be some other bug.

Anyway, I'm thinking that it might be a better alternative to pull static copies of the remote repo contents during the build process. The only disadvantage of this is that you can't live-edit the remote repo's code in the context of one of the projects using it, then commit from there back into the real repo. But that really just means that you have to have solid testing components in the original repo rather than relying on the consumer projects to do testing during active development.

The main front page and the Maintenance blog have been tweaked to be a little more useful. Most blogs have not been tweaked and look a bit rubbish. Here are the changes I made. I made it as terse as I could, but they're still kind of clunky.

------

1. Log in to blogs. Toolbar at top->Site->Site Dashboard
2. Select Blog (now called a Collection) from list
3. Look for a row of tabs with titles like 'Collection Dashboard', 'Contents', 'Comments' etc. Select 'Skin'
4. Look for a section titled 'Header Top Settings'. Uncheck 'Display Header Top'. This disables the weird listing of blogs at the top of the page.

5. You can also change some of the colours used for this Collection. The main page and the maintenance blog use the following colours:

   • General Settings section: 'Color Scheme'=#005493
   • Main Header Settings section: 'Site Title Color'=#ffffff; 'Menu Links Color'=#ffffff; 'Main Header BG Color'=#002754
6. There is a save button at the bottom of the page.

7. Next, go to the 'Widgets' tab. We'll be deleting some widgets - in some cases we'll empty sections completely.

   • 'Item Page' section: Seen by, Item Vote
   • 'Item Single' section: Item Location, About Author, Seen by
   • 'Menu' section: Pages, Link to: Collection owner contact form, Link to: Log in form
   • 'Page Top' section: All blogs
   • 'Register' section: Register, Content Block
   • 'Sidebar' section: Random photo, XML Feeds
   • 'Sidebar 2' section: remove everything
   • 'Login Required' section: Log in to your account
8. We'll be editing/adding some widgets too.

9. In the 'Menu' section we'll make the 'Home' link in the header point to the front page of the whole blog system instead of the Collection front page and create a log in link that does not cause confusion.

   • 'Edit' the Link to: Front Page widget and type 'Home' in the "Link text" field. Change "Collection ID" to 1. Save.
   • 'Edit' the Link to: Log in form widget by changing 'Link Type' from "Log in form (disp-login)" to "Go to any URL", type 'Log in' in the "Link text" field, and type 'https://hcmc.uvic.ca/blogs/index.php?disp=login' in the "URL" field. Save.
10. In the 'Sidebar 2' section we're adding a widget called "Public collctions list" - you'll find it at the bottom of the widgets list under 'Other'. Leave the settings alone.

Migrated first JSRDocs_dev then JSRDocs forward to the new AdaptiveDB codebase, and tested. All seems good. That's the last of the important database projects that I'm aware of that will need porting. Backup of the db taken today; backups stored both locally and in the folder of the live site.

So far I've updated both VPN dbs and all four active Landscapes dbs to the latest codebase, meaning that they will run on both PHP 5.6 and PHP 7. For VPN and the Powell St db, I've put the following stuff in .htaccess files to ensure that we're actually running on the new server and taking advantage of PHP 7, because the speed payoff is significant:

setEnv UVPHP_VERSION 70

RewriteEngine On
RewriteBase /
RewriteCond     %{HTTP_HOST} ^hcmc\.uvic\.ca$ [NC]
RewriteRule     (.*)  https://webserver2.hcmc.uvic.ca%{REQUEST_URI}

Once the new server takes over from the old, we'll have to remove the redirects. I have one more database to port, the jsrDocs one.

Now that the required ports are opened up and working for both servers, I decided to see how straightforward it would be to get Let's Encrypt certs set up for the subdomains. The answer turns out to be that it's trivial:

  sudo add-apt-repository ppa:certbot/certbot
  sudo apt install python-certbot-apache
  sudo certbot --apache -d teijenkins.hcmc.uvic.ca
  sudo certbot renew --dry-run

The certbot did an automated challenge by temporarily tweaking the apache config and testing the tweak; then it downloaded and installed the certs, and even updated the virtual hosts file to point at them. The last line tests the renewal process, which has to happen every three months, and for which certbot has installed a cron job. If that fails, you get an email.

Spent a lot of the day on minor tweaks to the two Jenkins servers. The https problem on jenkins.hcmc.uvic.ca was a VLAN ACL issue, now fixed by NETS, so it's working OK now. Both machines now have 8GB of RAM, and the disk of the TEI Jenkins has been expanded by 50GB, so that should last for a little while. I haven't yet set up the Apache config on teiJenkins because I wanted to use the opportunity to document it properly, but it's now trying to build jobs (largely failing for various reasons relating to config).

These are my ongoing notes on how to get Jenkins up and running on a headless Ubuntu 18.04.

• Initial install: make sure you configure LVM to provide a big enough volume for what Jenkins needs in the root filesystem. By default, the Ubuntu installer will create a volume of only 4GB; if that happens, extend it as needed from the remaining free space.
• OPTIONAL: Get rid of cloud-init; it's installed by default but I see no value in it:

      wait until the VM boots
      login
      echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
      sudo apt-get purge cloud-init
      sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
      reboot
  

• Reconfigure the timezone to America/Vancouver, since it may be UTC by default:

  sudo dpkg-reconfigure tzdata

• You must

  sudo apt-add-repository universe

  before installing Jenkins, because it requires a package called daemon which is in that repo. Other stuff needs it too.

• sudo apt-get install ntp

• Install subversion. Although Jenkins has its Java implementation, some of our jobs use the command line app to get info.
• You must install Java 8 (openjdk-8-jdk), which is in the universe repo.
• Install jing. This will probably install a newer Java and make it the default, so

  sudo update-alternatives --config java

  , and set it back to 8.
• Install ant, ant-contrib, and libxml2-utils.
• Install linkchecker.
• Jenkins setup requires that you configure a relative path to the files. If you're runing a vm and proxying the port to your own machine, beware: don't choose the selected port on which you're viewing Jenkins. Instead, choose the port that you know Jenkins is running on IN THE VM. If you screw this up, you can fix it by editing jenkins.model.JenkinsLocationConfiguration.xml in /var/lib/jenkins.
• Install sendmail so Jenkins can send email.
• When setting up Jenkins, accept the suggested package of general-use plugins.
• In Jenkins, install log parser plugin and the ThinBackup plugin.
• In the Jenkins configuration, set the svn version to 1.7 (matches our revision server).
• In the Jenkins global security settings, choose "Project-based Matrix Authorization Strategy" and allow anonymous users to read the appropriate stuff.
• Once Jenkins set up and running, edit /etc/default/jenkins to add an appropriate value for the CSP policy. For example, for an utterly permissive policy, do this:

  JAVA_ARGS="$JAVA_ARGS -Dhudson.model.DirectoryBrowserSupport.CSP=\"\""

  to set the CSP allowing HTML to work properly. Consult docs on CSP to decide exactly what you need.
• As the jenkins user, create a directory /var/lib/jenkins/backups, then use that in the configuration of the ThinBackup plugin. I use this for the full backup schedule: H 12 * * 6 and this for the differential: H 12 * * 1-5
• For Mariage only: install imagemagick and libvips-tools.
• We usually use a configuration where the job workspaces are inside the job folder itself. To make this work, edit jenkins/config.xml, comment out the workspaceDir line, and replace it:

  <!--<workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>-->
    <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  
  

• For Moses project only:

  sudo apt install python3 python3-pip python3-lxml

  sudo -H pip3 install dicttoxml nltk numpy plotly jupyter

  Then install nltk punkt data:

  sudo python3

  import nltk

  nltk.download('punkt')

• Make sure that the folder nltk_data ended up in /usr/share. If it didn't, copy it there. It seems to end up in the wrong place often.

The tftp boot daemon on the apt server has its configuration file in the file /etc/default/tftpd-hpa.
The config file points to the filesystem location of the deliverables and a few options.

The deliverables we're using in the netboot setup are:
grub (a directory)
initrd.gz (the primordial OS that gets booted)
linux (the kernel)

The grub directory contains:
grub.cfg (the config file that provides the menuentries you see on-screen among other things)
grubnetx64.efi.signed (the boot app that actually boots the machine - AKA bootx64.efi)
theme (a directory containing pngs and the theme.txt file that arranges the pngs on the screen)
unicode.pf2 font (I think this is fall back font that can be removed, but I haven't tested the theory).

If you need to start/stop/restart the tftp daemon:
sudo systemctl start tftpd-hpa
sudo systemctl stop tftpd-hpa
sudo systemctl restart tftpd-hpa

To watch it at work, do:
sudo tail -f /var/log/syslog | grep tftpd

While fiddling with adding a new distro to reprepro I had cause to remove one...

1. Remove reference to it from the $repodir/conf/distributions file
2. sudo reprepro -Vb /path/to/name-of-your-repo --delete clearvanished