HCMC Blogs

Although it's a bit ugly. No time to get too fancy, unfortunately.

Made some progress today after discovering that installing some XML tools before the Sun JDK caused the OpenJDK to get installed, so I switched the order of some items. Also figured out a better way of enabling the partner repo that will work with other distros than Lucid, and made a couple of other tweaks. The full script is below (since our backup server is down at the moment). I've also built Jenkins4 as a Natty 11.04 server, and I'm testing the script on that to see if it might work there too. It seems to be running fine, and I've got as far as a working Jenkins once before retreating to create a useful restore point and fix some bugs. It's quite plausible it might work perfectly well on any Ubuntu between Lucid and Natty.

#!/bin/bash
#The Mighty Jenkins Builder Script.
#Note that this should be run as root (with sudo).
echo "Entering the Mighty Jenkins Builder Script."

uid=$(/usr/bin/id -u) && [ "$uid" = "0" ] ||
{ echo "This script must be run as root."; exit 1; }

echo "Running as root: good."

#First do updates.
echo "Doing system updates before starting on anything else."
apt-get update
apt-get upgrade

#Now add the repositories we want.
echo "Backing up repository list."
cp /etc/apt/sources.list /etc/apt/sources.list.bak

#Uncomment partner repos.
echo "Uncommenting partner repositories on sources list, so we can get Sun Java."
#Note: this is very crude, and also enables the CD-ROM source, which results in 
#errors. We need to make this more precise.
#sed -i -e "s/# deb/deb/g" /etc/apt/sources.list
#This is a better replacement:
sed -i -re '/partner/ s/^#//' /etc/apt/sources.list

#First Jenkins
echo "Adding Jenkins repository."
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list

#Next TEI.
echo "Adding TEI Debian repository."
gpg --keyserver wwwkeys.uk.pgp.net --recv-keys FEA4973F86A9A497
apt-key add ~/.gnupg/pubring.gpg
echo "deb http://tei.oucs.ox.ac.uk/teideb/binary ./" > /etc/apt/sources.list.d/tei.list

#Now we can start installing packages.
echo "Updating for new repositories."
apt-get update

echo "Installing Sun Java JDK."
apt-get install sun-java6-jdk &&
echo "Installing core packages we need."
apt-get install openssh-server libxml2 libxml2-utils devscripts xsltproc debhelper subversion trang &&

#TEI packages
echo "Installing TEI packages."
apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-sgml jing jing-trang-doc libjing-java rnv texlive-xetex &&
apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema tei-emacs saxon nxml-mode-tei tei-p5-xsl tei-p5-xsl2 tei-roma onvdl tei-oxygen zip &&

#I don't believe the following step is necessary, so it's been commented out for the moment.
#Waiting for info from SR and SY about why it was in the instructions.
#echo "Removing things that cause problems for TEI."
#apt-get remove `apt-cache search gcj | grep gcj | awk '{print $1}'`

#Setting up configuration for oXygen
mkdir /root/.com.oxygenxml
chown jenkins /root/.com.oxygenxml
chmod a+x /root/.com.oxygenxml
mkdir /root/.java
chown jenkins /root/.java
chmod a+x /root/.java
echo "Don't forget to put your licensekey.txt file in the folder /usr/share/oxygen so that oXygen is registered."

#More packages needed
echo "Installing packages needed for building TEI source."

#Various fonts and the like.
echo "Installing fonts we need."
apt-get install msttcorefonts ttf-arphic-ukai ttf-arphic-uming ttf-baekmuk ttf-junicode ttf-kochi-gothic ttf-kochi-mincho
echo "The Han Nom font is not available in repositories, so we have to download it from SourceForge."
cd /usr/share/fonts/truetype
mkdir hannom
cd hannom
wget -O hannom.zip http://downloads.sourceforge.net/project/vietunicode/hannom/hannom%20v2005/hannomH.zip
unzip hannom.zip
find . -iname "*.ttf" | rename 's/\ /_/g'
rm hannom.zip
fc-cache -f -v

#Jenkins
apt-get install jenkins

#Configuration for Jenkins
echo "Starting configuration of Jenkins."
echo "Getting the Hudson log parsing rules from TEI SVN."
cd /var/lib/jenkins
svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/P5/Utilities/hudson-log-parse-rules
chown jenkins hudson-log-parse-rules
echo "Getting all the job data from TEI SVN."
#Don't bring down the config.xml file for now; that contains security settings specific to 
#Sebastian's setup, and will prevent anyone from logging in. We leave the server unsecured,
#and make it up to the user to secure it.
#svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/config.xml
#chown jenkins config.xml
svn export --force https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/jobs/ jobs
chown -R jenkins jobs
echo "Installing Jenkins plugins."
cd plugins
wget --no-check-certificate http://updates.jenkins-ci.org/latest/copyartifact.hpi
chown jenkins copyartifact.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/emotional-hudson.hpi
chown jenkins emotiosudnal-hudson.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/greenballs.hpi
chown jenkins greenballs.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/jobConfigHistory.hpi
chown jenkins jobConfigHistory.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/plot.hpi
chown jenkins plot.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/log-parser.hpi
chown jenkins log-parser.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/scp.hpi
chown jenkins scp.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/WebSVN2.hpi
chown jenkins WebSVN2.hpi

echo "Restarting Jenkins server, so that it finds and initializes all the new plugins."
/etc/init.d/jenkins restart

echo "OK, we should be done. Now you have to:"
echo "1. Put your oXygen licence key in a file called licensekey.txt in the oXygen directory (/usr/share/oxygen/)."
echo "2. Go to the Jenkins interface on http://localhost:8080, and set up authentication. Read the Jenkins docs."
echo "That's it!"
read
exit

Went through the process a couple more times at home, and I have some questions still to be answered, and some additions to the script:

When it got to this line:

apt-get remove `apt-cache search gcj | grep gcj | awk '{print $1}'`

apt reported that it was uninstalling jing. I'd previously been puzzled about the absence of jing at the end of the script, because it's explicitly installed before this point, but it looks like this is the reason. I'm just wondering what that line is actually doing. I got it from the wiki:

and it looks like SY added that line in July last year:

http://wiki.tei-c.org/index.php?title=TEIDebian&diff=7877&oldid=7875

I can add another line to my script to reinstall jing, so it's no problem, but I've written to ask him why he did that. Perhaps it's not necessary to remove all that stuff, because installing jing just brings a lot of it back anyway.

If we don't delete that stuff, the addition would be:

echo "Reinstalling jing, which just got removed by the previous line."
apt-get install jing

We can also restart Jenkins at the end of the script, so the user doesn't have to:

echo "Restarting Jenkins server, so that it finds and initializes all the new plugins."
/etc/init.d/jenkins restart

Leaving a little early.

Had the signs delivered, and taped and waterproofed them all with JN. They're ready to be deployed on Monday. We can put them out ourselves, which is going to make things a little simpler.

Another minor site update.

Did most of a diagram of the OAI process today.

The full install script now seems to be working.

Sitting with K to debug an out-of-memory error with nVivo, I noticed that instead of closing nVivo to force it to write its file to disk, she was closing the VM. That would have the opposite effect: the file would not be written to the disk until the VM was restarted and nVivo, at some point, closed. That's a bit of a fragile situation, and we might want to look at a better approach to projects which depend on an app which doesn't actually save when it says it's saving.

Found some stuff had not got installed due to a couple of typos, and other things which used to be installed automatically as dependencies in the TEI Debian package now have to be explicitly installed. Updated the script. Finally got a successful job to run in Jenkins.

One remaining problem is kindlegen, which is proprietary and requires you to agree to terms before you can download it, so can't be installed from the command line. Still figuring out what to do about that.