HCMC Blogs

Did the bibliography and a few pages of the content. Going steadily.

Jenkins server gradually coming along, but eating up many hours.

Went through my TODO folder checking and removing obsolete stuff; started on some corrections to Kress, but hit some issues so contacted DG; then the final vol 25 article came in, so I did the word-processor work on it, and set up the initial framework file.

Late duty, and pushing forward with vm building and testing.

In between wrestling with vSphere and test-building vms, I've finished the first draft of my presentation on search engines for JADH. One more to go.

I now have a basic XSLT transform working, which does the hard stuff -- it creates tagging which can be used to render in a human-readable way multiple coloured underlines for nested codes. I still have to do the control panel and work on the CSS, but the hard bit is done.

Logging time from two late days, due to the pressure of work and Jenkins server crashes. This is a tough time.

These are my ongoing notes on how to get Jenkins up and running on a headless Ubuntu 18.04.

• Initial install: make sure you configure LVM to provide a big enough volume for what Jenkins needs in the root filesystem. By default, the Ubuntu installer will create a volume of only 4GB; if that happens, extend it as needed from the remaining free space.
• OPTIONAL: Get rid of cloud-init; it's installed by default but I see no value in it:

  
      wait until the VM boots
      login
      echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
      sudo apt-get purge cloud-init
      sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
      reboot
  

• Reconfigure the timezone to America/Vancouver, since it may be UTC by default:

  sudo dpkg-reconfigure tzdata

• You must

  sudo apt-add-repository universe

  before installing Jenkins, because it requires a package called daemon which is in that repo. Other stuff needs it too.

• sudo apt-get install ntp

• Install subversion. Although Jenkins has its Java implementation, some of our jobs use the command line app to get info.
• You must install Java 8 (openjdk-8-jdk), which is in the universe repo.
• Install jing. This will probably install a newer Java and make it the default, so

  sudo update-alternatives --config java

  , and set it back to 8.
• Install ant, ant-contrib, and libxml2-utils.
• Install linkchecker.
• Jenkins setup requires that you configure a relative path to the files. If you're runing a vm and proxying the port to your own machine, beware: don't choose the selected port on which you're viewing Jenkins. Instead, choose the port that you know Jenkins is running on IN THE VM. If you screw this up, you can fix it by editing jenkins.model.JenkinsLocationConfiguration.xml in /var/lib/jenkins.
• Install sendmail so Jenkins can send email.
• When setting up Jenkins, accept the suggested package of general-use plugins.
• In Jenkins, install log parser plugin and the ThinBackup plugin.
• In the Jenkins configuration, set the svn version to 1.7 (matches our revision server).
• In the Jenkins global security settings, choose "Project-based Matrix Authorization Strategy" and allow anonymous users to read the appropriate stuff.
• Once Jenkins set up and running, edit /etc/default/jenkins to add an appropriate value for the CSP policy. For example, for an utterly permissive policy, do this:

  JAVA_ARGS="$JAVA_ARGS -Dhudson.model.DirectoryBrowserSupport.CSP=\"\""

  to set the CSP allowing HTML to work properly. Consult docs on CSP to decide exactly what you need.
• As the jenkins user, create a directory /var/lib/jenkins/backups, then use that in the configuration of the ThinBackup plugin. I use this for the full backup schedule: H 12 * * 6 and this for the differential: H 12 * * 1-5
• For Mariage only: install imagemagick and libvips-tools.
• We usually use a configuration where the job workspaces are inside the job folder itself. To make this work, edit jenkins/config.xml, comment out the workspaceDir line, and replace it:

  <!--<workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>-->
    <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  
  

• For Moses project only:

  sudo apt install python3 python3-pip python3-lxml

  sudo -H pip3 install dicttoxml nltk numpy plotly jupyter

  Then install nltk punkt data:

  sudo python3

  import nltk

  nltk.download('punkt')

• Make sure that the folder nltk_data ended up in /usr/share. If it didn't, copy it there. It seems to end up in the wrong place often.

I now have Oxygen able to insert linked anchors and prompt for analysis codes. This is coming along nicely. I think we should have a pilot version ready by the end of next week. We'll need to write a fairly sophisticated two-step XSLT transformation to create an HTML version that's useful for reading/proofing, but there's nothing there that we haven't done before.

WJL's final commit of the day failed yesterday, because of a hash sum mismatch. We were unable to solve it last night, but started again this morning. My guess is that it was caused by a corrupted .svn folder on the local machine, or by a corruption on the server. In any case, the only solution was to svn-delete the whole folder, svn-add it again, and then copy backups of the files into it and svn-add them. Weird.