Maintenance

I've now got all important data off my old Lucid box (the Drive2 data), onto my new machine. We can now take out Drive2 from that machine and use it in a cradle, and repurpose the machine.

Problems solved this afternoon:

• rnv was failing to download and build; ampersands in the url query string needed to be backslash-escaped, and the file name needed to be specified with the -O flag.
• The log parse rules were not actually being used, even though they were downloaded, and were referred to in the job configs. I think this is because a file called hudson.plugins.logparser.LogParserPublisher.xml also needed to be there, to specify that the other file exists somehow. That's the theory anyway; not tested yet.
• Emailing could never have worked without going to a lot of trouble to mess with Jenkins's setup, so I'm simply sidestepping it and leaving it up to the user to set it up if they want to. I'm removing mine and SR's email addresses from the job configs.
• The Priority Sorter is working, but didn't actually solve the sequencing problem for the first build. Jinks managed to complete OxGarage, Roma and Stylesheets1 before Stylesheets completed, so it just had time to start and fail on P5-Test before the required artifacts were there. I'm consideriong making P5-Test a downstream job from Stylesheets, which should solve it once and for all.
• Core files are now in the TEI repo, as they should be, and the hudson log parse rules have been moved from P5/Source to Documents/Editing/Jenkins, where all the rest of the stuff is.

• Added the OxGarage messages to the log parser rules file, so they no longer show up as errors.
• Re-organized the structure of the build process so EULAs come first, and the rest of the build should be able to proceed unattended, using the -y flag on apt-get install commands.
• Moved the requirement for an Oxygen license to the beginning of the process, so that the user must provide it before the build can proceed. This prevents the Stylesheets build from failing first time out, because of the missing license.
• Added more useful information messages, particularly before the build.
• Tested the Priority Sorter plugin for Jenkins, which should enable us to put P5 builds at the end of the queue, so that we can be sure that Stylesheets will build first.

The last thing to do is to XSLT the job configs to insert the user's email address instead of SR's and mine.

It turns out that the response at the command line when you try to run rnv is completely misleading. It leads you to believe that the executable is missing, whereas it's clearly there. Actually the problem is caused by the fact that the rnv installed from the TEI repo is 32-bit, and won't run on a 64-bit kernel.

So instead of installing it from the repos, I'm downloading and building it instead:

apt-get install libexpat-dev
wget http://downloads.sourceforge.net/project/rnv/Sources/1.7.10/rnv-1.7.10.zip
unzip rnv-1.7.10.zip
cd rnv-1.7.10
./configure
make
make install

In view of this, I don't think rnv should be in the TEI repos at all, unless it can be done in such a way as to provide the right build for the host architecture.

I now have a more convenient setup for creating and testing Jinks builds. This is how it works:

• There's a sort of "seed" vm called PreJenkins in /home/mholmes/VirtualBox VMs/
• That's a fully-updated vanilla install of Precise server.
• In my tei/jenkins directory, there's a script called vboxmanage_Jenkins2012A.sh. When you run that script, it clones the vanilla server to create a new VM called Jenkins2012A. It also configures that VM so that you can see its port 8080 (Jenkins) on the host's 7070, and so that you can ssh into it on port 2012.
• After running the above script, you start the newly-created VM. You log in as hcmc, and sudo su, then you run a script you'll find there called make_jenkins.sh (also in my local tei/jenkins directory on my host).
• The make_jenkins.sh script first scps a copy of the jenkins_builder_script_2012.sh from the host into the hcmc home directory (pulling this in, rather than having it there in the seed, enables me to tweak the script easily while working on it). It runs the script.
• At this point, you're seeing what the ordinary user of the jenkins build script would see.
• Once the build is complete, and you exit from the build script, make_jenkins.sh scps a copy of our Oxygen license from the host into the right location, so that Oxygen is registered.
• Now Jenkins should be running on the new machine. You can now run a script on the host called connect_to_Jenkins2012A.sh, which will send Firefox to the right port, and ssh into the machine. The connection to the machine is done like this:

  ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l hcmc -p 2012 localhost
  

  which precludes checking of the machine's key, or storage of its key; this avoids the problem where every time you clone the pre-seed to create a new VM, it has a different key, and the host complains that the key in known_hosts has changed.
• Then you wait and watch to see if Jenkins builds the jobs OK.

This is what remains to be done:

• Fix the rnv problem. It's installed now from the TEI packages, and is clearly there, but make and bash are unable to find it:

   rnv
  -bash: /usr/bin/rnv: No such file or directory
  

  Very weird indeed. There are executable copies in /usr/bin and in /usr/bin/X11, just like on my local machine, where they work fine.
• Figure out what to do about job configs which include mine and SR's email addresses. We should probably XSTL the config.xml files during the setup process to replace SR's email with the user's own, which we can ask for.
• Solve the problem whereby P5-Test builds and fails before Stylesheets has built for the first time. Stylesheets needs to build once successfully before the P5 jobs can build; thereafter, they're independent. There must be a way to force Jenkins to build Stylesheets before P5-Test the first time out.
• List the error messages and warnings that show up ONLY on the first checkout/build of the jobs. There are lots of these in OxGarage, and possibly elsewhere. These need to be put into the hudson-log-parse-rules file so that a first-time user of the script is not worried by a lot of errors that will never show up again, and aren't relevant.
• Document and publish the script, by updating the page on the TEI wiki.
• Get a real VM to replace our current teijenkins, and when it's working, repoint the domain and bring the old one down.

SM working on MoL has been seeing LibreOffice Writer crash frequently when editing a complex document with comments. I've worked with similar documents without problems, though.

I looked around for alternative word-processors, but neither Calligra Words nor AbiWord handle comments. Noting that Writer works fine for me on Precise + Gnome 3, and fine for CB with Gnome 2, I've now installed Gnome Shell on Radish, and we'll see if that solves the problem. Never liked Unity anyway.

In the process of doing this, I noticed that an HCMC style deb is failing to update itself when doing apt-get upgrade.

I've been working through the problems on the Jinks build with SR, and we're close to a working system. This is where we're at right now:

• Problems caused by the absence of a JDK have been solved by apt-get install openjdk-6-jdk (so we apparently don't actually need Oracle).
• There are still a few error messages about not being able to write to /root/.java/.com.oxygenxml.rk, which I'm puzzled by -- I even see them if I make the file world-writable. I'm inclined to suppress these messages in the log-parse rules.
• SR has added rnv to the TEI packages, so I'm now installing it as part of that group of installs; no need to get it from SourceForge.
• We still have a problem with some error messages in the OxGarage build, but they don't seem to be indicating a series issue.
• All of the P5 jobs have now completed apparently successfully, but there one set of errors on TEIP5:

  tar: p5odds.rng: Cannot stat: No such file or directory
  tar: p5odds.rnc: Cannot stat: No such file or directory
  tar: p5odds-examples.rng: Cannot stat: No such file or directory
  tar: p5odds-examples.rnc: Cannot stat: No such file or directory
  tar: Exiting with failure status due to previous errors

  and in fact none of those four files are in the release/xml/tei/odd directory of the archived artifacts, whereas they are there in my current Jenkins artifacts. So I think this build should be tagged as a failure. As it is, although the console says it will be tagged as a fail, it isn't. This will need a bit of investigation.
• I've created a vanilla Precise Server install which I can keep updated and use as the basis for script testing, by cloning it each time.

Tested projector with new build/Intel Graphics and wireless keyboard.
Turns out that the projector looks fabulous as long as you use its native resolution. The bad news is that its native resolution is 1024x768. Any other 4:3 resolution that works with the Intel Graphics chip looks terrible on the screen.
To get mirroring to work, you need to plug the HDMI plug in to the DVI port on the computer (via the HDMI-DVI converter) and use a VGA cable for the monitor itself. Any other combo is fraught.
Wireless keyboard/trackpad has a nice feel to it but is a real pain to use as it loses its connection regularly. Pressing the connect button MAY work, but it doesn't always. I tried elevating the receiver, which helps some (but not enough). Martin replaced the batteries and reports that performance is better.

If you want to do a presentation in B045 using our equipment:
Projector plugs in to the orange DVI converter and then in to the DVI port on the computer. Monitor plugs in to the computer via the VGA cable.
Connect wireless keyboard: plug the receiver in to the computer using a USB extension cable and elevate the receiver so it's within line-of-sight of the keyboard. Turn on the (keyboard) power and depress the connect button. It should work more-or-less right away.

Started testing on a VM, running my old script bit by bit and watching the results. I've got as far as a working Jinks server, and one or two builds work, but there will have to be many changes. These are my notes so far:

First time building Jenkins with Precise, I'm running only what I think I need to run, and keeping notes.

#START UNTESTED BIT -- TEST LATER ON DEFAULT INSTALL.#
#First do updates.
echo "Doing system updates before starting on anything else."
apt-get update
apt-get upgrade

#Now add the repositories we want.
echo "Backing up repository list."
cp /etc/apt/sources.list /etc/apt/sources.list.bak

#Uncomment partner repos.
echo "Uncommenting partner repositories on sources list, so we can get Sun Java."
sed -i -re '/partner/ s/^#//' /etc/apt/sources.list
#FINISH UNTESTED BIT#


#First Jenkins
echo "Adding Jenkins repository."
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list

#OK

#Next TEI.
echo "Adding TEI Debian repository."
     gpg --keyserver wwwkeys.uk.pgp.net --recv-keys FEA4973F86A9A497
#NOT OK: HAD TO USE:
#Next TEI.
gpg --keyserver wwwkeys.pgp.net --recv-keys FEA4973F86A9A497
echo "deb http://tei.oucs.ox.ac.uk/teideb/binary ./" > /etc/apt/sources.list.d/tei.list

#OK

#Now we can start installing packages.
echo "Updating for new repositories."
apt-get update

#OK

echo "Installing core packages we need."
apt-get install openssh-server libxml2 libxml2-utils devscripts xsltproc debhelper subversion trang &&
echo "Installing curl, required for some tei building stuff."
apt-get install curl &&

#OK. curl already installed.

#TEI packages
echo "Installing TEI packages."
	apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-sgml jing jing-trang-doc libjing-java rnv texlive-xetex &&
#NOT OK: rnv no longer in repos (but is in TEI, so no worries); linuxdoc-sgml should be linuxdoc-tools. HAD TO USE:

apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-tools jing jing-trang-doc libjing-java texlive-xetex

	apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema saxon nxml-mode-tei tei-p5-xsl tei-p5-xsl2 tei-p5-xslprofiles tei-roma onvdl tei-oxygen zip &&

#NOT OK: nxml-mode-tei has no installation candidate. HAD TO USE:
apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema saxon tei-p5-xsl tei-p5-xsl2 tei-p5-xslprofiles tei-roma onvdl tei-oxygen zip &&

#Setting up configuration for oXygen
mkdir /root/.com.oxygenxml
chmod a+x /root/.com.oxygenxml
mkdir /root/.java
chmod a+x /root/.java
echo "Don't forget to put your licensekey.txt file in the folder /usr/share/oxygen so that oXygen is registered."

#OK

#Various fonts and the like.
echo "Installing fonts we need."
apt-get install ttf-dejavu msttcorefonts ttf-arphic-ukai ttf-arphic-uming ttf-baekmuk ttf-junicode ttf-kochi-gothic ttf-kochi-mincho
echo "The Han Nom font is not available in repositories, so we have to download it from SourceForge."
cd /usr/share/fonts/truetype
mkdir hannom
cd hannom
wget -O hannom.zip http://downloads.sourceforge.net/project/vietunicode/hannom/hannom%20v2005/hannomH.zip
unzip hannom.zip
find . -iname "*.ttf" | rename 's/\ /_/g'
rm hannom.zip
fc-cache -f -v

#OK

#Configuration for Jenkins
echo "Starting configuration of Jenkins."
echo "Getting the Hudson log parsing rules from TEI SVN."
cd /var/lib/jenkins
svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/P5/Utilities/hudson-log-parse-rules
chown jenkins hudson-log-parse-rules
echo "Getting all the job data from TEI SVN."
#Don't bring down the config.xml file for now; that contains security settings specific to 
#Sebastian's setup, and will prevent anyone from logging in. We leave the server unsecured,
#and make it up to the user to secure it.
#svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/config.xml
#chown jenkins config.xml
svn export --force https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/jobs/ jobs
chown -R jenkins jobs
echo "Installing Jenkins plugins."
cd plugins
wget --no-check-certificate http://updates.jenkins-ci.org/latest/copyartifact.hpi
chown jenkins copyartifact.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/emotional-hudson.hpi
chown jenkins emotional-hudson.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/greenballs.hpi
chown jenkins greenballs.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/jobConfigHistory.hpi
chown jenkins jobConfigHistory.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/plot.hpi
chown jenkins plot.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/log-parser.hpi
chown jenkins log-parser.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/scp.hpi
chown jenkins scp.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/WebSVN2.hpi
chown jenkins WebSVN2.hpi

echo "Restarting Jenkins server, so that it finds and initializes all the new plugins."
/etc/init.d/jenkins restart

#OK


PROBLEMS AFTER STARTUP:

1. Builds fail because rnv is not installed. Have to get it from SourceForge:

cd ~
apt-get install libexpat-dev
wget http://downloads.sourceforge.net/project/rnv/Sources/1.7.10/rnv-1.7.10.zip
unzip rnv-1.7.10.zip
cd rnv-1.7.10
./configure
make
make install

2. NON-FATAL, but investigate: in Stylesheets:
	Unable to locate tools.jar. Expected to find it in /usr/lib/jvm/java-6-openjdk-amd64/lib/tools.jar

3. FATAL:

BUILD Build for P5, XSLT 2.0
test -d release/p5 || mkdir -p release/p5/xml/tei/stylesheet/
for i in  bibtex common2 docx dtd docbook epub epub3 fo2 html html5 latex2 nlm odds2 odt profiles/default rdf relaxng rnc slides tbx tite tools txt xhtml2 xsd ; do \
		tar cf - --exclude .svn $i | (cd release/p5/xml/tei/stylesheet; tar xf - ); \
	done
(cd odt;  mkdir TEIP5; saxon -o:TEIP5/teitoodt.xsl -s:teitoodt.xsl expandxsl.xsl ; cp odttotei.xsl TEIP5.ott teilite.dtd TEIP5; jar cf ../teioo.jar TEIP5 TypeDetection.xcu ; rm -rf TEIP5)

/bin/sh: 1: jar: not found
mkdir -p /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp catalog.xml /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp teioo.jar /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp: cannot stat `teioo.jar': No such file or directory
make[1]: *** [installp5] Error 1
make[1]: Leaving directory `/var/lib/jenkins/jobs/Stylesheets/workspace'

This took all day. I first attempted to set the home dir for mholmes on the second disk drive, but when I did that, I ended up with no bash profile. I ended up leaving it at /home/mholmes, but symlinking to specific folders on the second drive instead.

To move VirtualBox vms, I first deleted all snapshots, then moved the disks over (only the disks). Then I created new VMs for the HDs. All working normally, after lots of Windows updates and a bit of tweaking on the Win7 machine (which by default tried to attach the old IDE disk image to a SATA disk controller).

There is now a complete build system for Ubuntu 12.04 set up. Here's how it works.
A machine called papaya is set up with a mirror of the precise repo (main, restricted, security, extras, universe and multiverse), Google's repos for Earth and Chrome, Oracle's Virtualbox and a groovy icon set (Faenza). It also has a reprepro setup that runs a kind-of local ppa with a few home-brew apps for use in the HCMC labs: hcmc-desktop (a metapackage that installs a bunch of necessary software and sets up stuff like printers and so forth), hcmc-auth (for LDAP logins), hcmc-oxygen (xml editor) and hcmc-style (adjusts the candy-cane look to a greyscale look).

In order to manage the mirrors see the setup documention here. The mirror should automatically update itself every day. To add a new repo to be mirrored run the script called add-mirror.sh in the admin user's homedir. It's a wizard-kind-of-thing that leads you by the hand through the process.

In order to add a package to the hcmc repo there is a script in the admin homedir called uprepo.sh. It is extremely basic, adding anything it finds in the admin user's homedir/packages directory to the repo, ignoring anything that is already in the repo. It demands a passphrase (twice) for my gpg key (ask me for it) in order to add a package.

To remove a package from the hcmc repo there is a script called rmpkg.sh in the admin user's homedir. It takes a package name as an argument (e.g. hcmc-desktop) and also demands my gpg passphrase

In order to install a fresh Ubuntu 12.04 you can either do the vanilla install first, then run the Bob the Builder script, or you can grab the hcmc-mini.iso from http://apt.hcmc.uvic.ca/iso/precise/ and put it on a thumb drive using something like unetbootin, which is in the repos. The hcmc-mini.iso is a custom-built iso which has a set of preseeds built in to it so it sets up everything required in the HCMC labs. The great thing about using it is that it pulls all packages for the install directly from papaya, so there is no need to update the machine after the install. After the install you're left with a completely set up HCMC lab machine that's ready to go.
***** NOTE: the admin user that gets set up by hcmc-mini is preseeded with a LAME password because I have so far been unsuccessful in creating an md5 hash to store in the preseed - although it is *supposed* to work. I'll change the preseed if I can get it to work. In the meantime, chage the admin user's password after the build is finished.

The hcmc-mini.iso image is created by a script in the admin user's homedir called build-hcmc-mini.sh. It downloads a stock netboot image from an official Ubuntu source, mangles it to include the necessary preseeds, then repacks it in to a bootable iso image, storing it in /var/www/iso/precise.