Permalink 05:00:40 pm, by mholmes, 51 words, 140 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 45

JSRDocs project ported to latest AdaptiveDB code

Migrated first JSRDocs_dev then JSRDocs forward to the new AdaptiveDB codebase, and tested. All seems good. That's the last of the important database projects that I'm aware of that will need porting. Backup of the db taken today; backups stored both locally and in the folder of the live site.


Permalink 04:26:27 pm, by mholmes, 109 words, 15 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 180

Porting adaptivedbs forward for new server

So far I've updated both VPN dbs and all four active Landscapes dbs to the latest codebase, meaning that they will run on both PHP 5.6 and PHP 7. For VPN and the Powell St db, I've put the following stuff in .htaccess files to ensure that we're actually running on the new server and taking advantage of PHP 7, because the speed payoff is significant:


RewriteEngine On
RewriteBase /
RewriteCond     %{HTTP_HOST} ^hcmc\.uvic\.ca$ [NC]
RewriteRule     (.*)  https://webserver2.hcmc.uvic.ca%{REQUEST_URI}

Once the new server takes over from the old, we'll have to remove the redirects. I have one more database to port, the jsrDocs one.


Permalink 01:24:32 pm, by mholmes, 122 words, 153 views   English (CA)
Categories: Announcements; Mins. worked: 20

Certbot for two domains

My server teijenkins.hcmc.uvic.ca is also proxied as jenkins2.tei-c.org, and it needs a cert for that too. I had assumed that I would need to generate a second certificate, but actually it's easy to add a new domain to the existing cert. The VirtualHost has this:

ServerName teijenkins.hcmc.uvic.ca
ServerAlias *jenkins*.tei-c.org

and all I had to do was this:

sudo certbot --apache -d teijenkins.hcmc.uvic.ca -d jenkins2.tei-c.org

and it regenerated the existing cert with the other domain added to it. This cert runs till December; it's not clear from the feedback whether it will be renewed automatically, or whether I'll have to run certbot manually to get that to happen.


Permalink 08:10:18 am, by mholmes, 44 words, 28 views   English (CA)
Categories: Announcements; Mins. worked: 20

Jenkins servers: now forcing https

I've now set up mod_rewrite to force https on the Jenkins servers. We'll see if that breaks anything in the builds; my guess is that the occasional call to retrieve a product in a preceding build may fail when it hits the redirect.


Permalink 05:05:15 pm, by mholmes, 126 words, 6 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 120

Jenkins Servers: Let's Encrypt certs deployed

Now that the required ports are opened up and working for both servers, I decided to see how straightforward it would be to get Let's Encrypt certs set up for the subdomains. The answer turns out to be that it's trivial:

  sudo add-apt-repository ppa:certbot/certbot
  sudo apt install python-certbot-apache
  sudo certbot --apache -d teijenkins.hcmc.uvic.ca
  sudo certbot renew --dry-run

The certbot did an automated challenge by temporarily tweaking the apache config and testing the tweak; then it downloaded and installed the certs, and even updated the virtual hosts file to point at them. The last line tests the renewal process, which has to happen every three months, and for which certbot has installed a cron job. If that fails, you get an email.


Permalink 04:56:58 pm, by mholmes, 99 words, 12 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 240

Servers getting close

Spent a lot of the day on minor tweaks to the two Jenkins servers. The https problem on jenkins.hcmc.uvic.ca was a VLAN ACL issue, now fixed by NETS, so it's working OK now. Both machines now have 8GB of RAM, and the disk of the TEI Jenkins has been expanded by 50GB, so that should last for a little while. I haven't yet set up the Apache config on teiJenkins because I wanted to use the opportunity to document it properly, but it's now trying to build jobs (largely failing for various reasons relating to config).


Permalink 05:29:53 pm, by mholmes, 570 words, 23 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 240

Jenkins on 18.04

These are my ongoing notes on how to get Jenkins up and running on a headless Ubuntu 18.04.

  • Initial install: make sure you configure LVM to provide a big enough volume for what Jenkins needs in the root filesystem. By default, the Ubuntu installer will create a volume of only 4GB; if that happens, extend it as needed from the remaining free space.
  • OPTIONAL: Get rid of cloud-init; it's installed by default but I see no value in it:
        wait until the VM boots
        echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
        sudo apt-get purge cloud-init
        sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
  • Reconfigure the timezone to America/Vancouver, since it may be UTC by default:
    sudo dpkg-reconfigure tzdata
  • You must
    sudo apt-add-repository universe
    before installing Jenkins, because it requires a package called daemon which is in that repo. Other stuff needs it too.
  • sudo apt-get install ntp
  • Install subversion. Although Jenkins has its Java implementation, some of our jobs use the command line app to get info.
  • You must install Java 8 (openjdk-8-jdk), which is in the universe repo.
  • Install jing. This will probably install a newer Java and make it the default, so
    sudo update-alternatives --config java
    , and set it back to 8.
  • Install ant, ant-contrib, and libxml2-utils.
  • Install linkchecker.
  • Jenkins setup requires that you configure a relative path to the files. If you're runing a vm and proxying the port to your own machine, beware: don't choose the selected port on which you're viewing Jenkins. Instead, choose the port that you know Jenkins is running on IN THE VM. If you screw this up, you can fix it by editing jenkins.model.JenkinsLocationConfiguration.xml in /var/lib/jenkins.
  • Install sendmail so Jenkins can send email.
  • When setting up Jenkins, accept the suggested package of general-use plugins.
  • In Jenkins, install log parser plugin and the ThinBackup plugin.
  • In the Jenkins configuration, set the svn version to 1.7 (matches our revision server).
  • In the Jenkins global security settings, choose "Project-based Matrix Authorization Strategy" and allow anonymous users to read the appropriate stuff.
  • Once Jenkins set up and running, edit /etc/default/jenkins to add an appropriate value for the CSP policy. For example, for an utterly permissive policy, do this:
    JAVA_ARGS="$JAVA_ARGS -Dhudson.model.DirectoryBrowserSupport.CSP=\"\""
    to set the CSP allowing HTML to work properly. Consult docs on CSP to decide exactly what you need.
  • As the jenkins user, create a directory /var/lib/jenkins/backups, then use that in the configuration of the ThinBackup plugin. I use this for the full backup schedule: H 12 * * 6 and this for the differential: H 12 * * 1-5
  • For Mariage only: install imagemagick and libvips-tools.
  • We usually use a configuration where the job workspaces are inside the job folder itself. To make this work, edit jenkins/config.xml, comment out the workspaceDir line, and replace it:
  • For Moses project only:
    sudo apt install python3 python3-pip python3-lxml
    sudo -H pip3 install dicttoxml nltk numpy plotly jupyter
    Then install nltk punkt data:
    sudo python3
    import nltk
  • Make sure that the folder nltk_data ended up in /usr/share. If it didn't, copy it there. It seems to end up in the wrong place often.


Permalink 09:43:26 am, by mholmes, 102 words, 9 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 30

VirtualBox weirdity: shared folders using symlinks

After rebuilding my desktop, VirtualBox worked fine except that my Windows VM could not connect to the shared folder I had configured for it. After finding this page, I was able to run this command:

VBoxManage setextradata "Windows 10" VBoxInternal2/SharedFoldersEnableSymlinksCreate/vmShare 1

where "Windows 10" is the name of the VM, and "vmShare" is the name of the share, then restart the VM, and it worked. Obscure thing, so I document it here. The issue is that the path to the shared folder depends on a symlink on my system, since the shared folder lives on an encrypted data drive, not the system drive.


Permalink 05:05:41 pm, by mholmes, 61 words, 16 views   English (CA)
Categories: R & D, Activity log; Mins. worked: 120

Borked upgrade to Ubuntu 18.04

Decided to upgrade my desktop, and encountered a known common scenario whereby both gdm and lightdm seem to get borked and no GUI can be started. Eventually did a reinstall. Note to self: once email and browsers are properly set up and stable, move those home folders onto the spinning disk so that they don't have to get recreated when rebuilding.


Permalink 10:56:48 am, by sarneil, 129 words, 91 views   English (CA)
Categories: Announcements; Mins. worked: 90

BOM in htaccess file

I was creating an htaccess file to restrict access to a folder on nfs.hcmc.uvic.ca. I used BBedit to create the file. It's configurations said it was not include a Byte Order Mark in the file (i.e. encoding was set to UTF-8, and not UTF-8 with BOM). When I uploaded the file to the server and accessed it in the browser I got an internal server error. Sysadmin checked logs and told me there was a BOM (\xef\xbb\xbf) at the start of the file. That did not appear in BBEdit (even with Show Invisibles enabled). I opened the file in Nano and no BOM was visible in that editor either. I deleted the file and retyped it from scratch in Nano, and that worked.

:: Next Page >>


This blog is the location for all work involving software and hardware maintenance, updates, installs, etc., both routine and urgent, in the server room, the labs and the R&D rooms.


XML Feeds