Maintenance

I'm trying to figure out an easy way of putting a Jenkins config (config.xml) file into the directory where it's expecting it to be, in the hope that this will cause Jinks to use the correct log parse rules when running jobs. I can create a generic config file, but I want to make sure that it doesn't have the wrong Jenkins build number in it. Once I have the right build number from the current Jenkins instance, I'll be able to pass it as a parameter to an XSLT transformation. Here's one approach which works:

wget -O /tmp/jinks.html http://localhost:8080
JINKSBUILD=`grep -oP '(?<=Jenkins ver\. )(1.[0-9\.]+)(?=</a>)' /tmp/jinks.html`

I'm also going to investigate the option of using the Jenkins CLI interface.

I've now got all important data off my old Lucid box (the Drive2 data), onto my new machine. We can now take out Drive2 from that machine and use it in a cradle, and repurpose the machine.

Problems solved this afternoon:

• rnv was failing to download and build; ampersands in the url query string needed to be backslash-escaped, and the file name needed to be specified with the -O flag.
• The log parse rules were not actually being used, even though they were downloaded, and were referred to in the job configs. I think this is because a file called hudson.plugins.logparser.LogParserPublisher.xml also needed to be there, to specify that the other file exists somehow. That's the theory anyway; not tested yet.
• Emailing could never have worked without going to a lot of trouble to mess with Jenkins's setup, so I'm simply sidestepping it and leaving it up to the user to set it up if they want to. I'm removing mine and SR's email addresses from the job configs.
• The Priority Sorter is working, but didn't actually solve the sequencing problem for the first build. Jinks managed to complete OxGarage, Roma and Stylesheets1 before Stylesheets completed, so it just had time to start and fail on P5-Test before the required artifacts were there. I'm consideriong making P5-Test a downstream job from Stylesheets, which should solve it once and for all.
• Core files are now in the TEI repo, as they should be, and the hudson log parse rules have been moved from P5/Source to Documents/Editing/Jenkins, where all the rest of the stuff is.

It turns out that the response at the command line when you try to run rnv is completely misleading. It leads you to believe that the executable is missing, whereas it's clearly there. Actually the problem is caused by the fact that the rnv installed from the TEI repo is 32-bit, and won't run on a 64-bit kernel.

So instead of installing it from the repos, I'm downloading and building it instead:

apt-get install libexpat-dev
wget http://downloads.sourceforge.net/project/rnv/Sources/1.7.10/rnv-1.7.10.zip
unzip rnv-1.7.10.zip
cd rnv-1.7.10
./configure
make
make install

In view of this, I don't think rnv should be in the TEI repos at all, unless it can be done in such a way as to provide the right build for the host architecture.

I now have a more convenient setup for creating and testing Jinks builds. This is how it works:

• There's a sort of "seed" vm called PreJenkins in /home/mholmes/VirtualBox VMs/
• That's a fully-updated vanilla install of Precise server.
• In my tei/jenkins directory, there's a script called vboxmanage_Jenkins2012A.sh. When you run that script, it clones the vanilla server to create a new VM called Jenkins2012A. It also configures that VM so that you can see its port 8080 (Jenkins) on the host's 7070, and so that you can ssh into it on port 2012.
• After running the above script, you start the newly-created VM. You log in as hcmc, and sudo su, then you run a script you'll find there called make_jenkins.sh (also in my local tei/jenkins directory on my host).
• The make_jenkins.sh script first scps a copy of the jenkins_builder_script_2012.sh from the host into the hcmc home directory (pulling this in, rather than having it there in the seed, enables me to tweak the script easily while working on it). It runs the script.
• At this point, you're seeing what the ordinary user of the jenkins build script would see.
• Once the build is complete, and you exit from the build script, make_jenkins.sh scps a copy of our Oxygen license from the host into the right location, so that Oxygen is registered.
• Now Jenkins should be running on the new machine. You can now run a script on the host called connect_to_Jenkins2012A.sh, which will send Firefox to the right port, and ssh into the machine. The connection to the machine is done like this:

  ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l hcmc -p 2012 localhost
  

  which precludes checking of the machine's key, or storage of its key; this avoids the problem where every time you clone the pre-seed to create a new VM, it has a different key, and the host complains that the key in known_hosts has changed.
• Then you wait and watch to see if Jenkins builds the jobs OK.

This is what remains to be done:

• Fix the rnv problem. It's installed now from the TEI packages, and is clearly there, but make and bash are unable to find it:

   rnv
  -bash: /usr/bin/rnv: No such file or directory
  

  Very weird indeed. There are executable copies in /usr/bin and in /usr/bin/X11, just like on my local machine, where they work fine.
• Figure out what to do about job configs which include mine and SR's email addresses. We should probably XSTL the config.xml files during the setup process to replace SR's email with the user's own, which we can ask for.
• Solve the problem whereby P5-Test builds and fails before Stylesheets has built for the first time. Stylesheets needs to build once successfully before the P5 jobs can build; thereafter, they're independent. There must be a way to force Jenkins to build Stylesheets before P5-Test the first time out.
• List the error messages and warnings that show up ONLY on the first checkout/build of the jobs. There are lots of these in OxGarage, and possibly elsewhere. These need to be put into the hudson-log-parse-rules file so that a first-time user of the script is not worried by a lot of errors that will never show up again, and aren't relevant.
• Document and publish the script, by updating the page on the TEI wiki.
• Get a real VM to replace our current teijenkins, and when it's working, repoint the domain and bring the old one down.

I've been working through the problems on the Jinks build with SR, and we're close to a working system. This is where we're at right now:

• Problems caused by the absence of a JDK have been solved by apt-get install openjdk-6-jdk (so we apparently don't actually need Oracle).
• There are still a few error messages about not being able to write to /root/.java/.com.oxygenxml.rk, which I'm puzzled by -- I even see them if I make the file world-writable. I'm inclined to suppress these messages in the log-parse rules.
• SR has added rnv to the TEI packages, so I'm now installing it as part of that group of installs; no need to get it from SourceForge.
• We still have a problem with some error messages in the OxGarage build, but they don't seem to be indicating a series issue.
• All of the P5 jobs have now completed apparently successfully, but there one set of errors on TEIP5:

  tar: p5odds.rng: Cannot stat: No such file or directory
  tar: p5odds.rnc: Cannot stat: No such file or directory
  tar: p5odds-examples.rng: Cannot stat: No such file or directory
  tar: p5odds-examples.rnc: Cannot stat: No such file or directory
  tar: Exiting with failure status due to previous errors

  and in fact none of those four files are in the release/xml/tei/odd directory of the archived artifacts, whereas they are there in my current Jenkins artifacts. So I think this build should be tagged as a failure. As it is, although the console says it will be tagged as a fail, it isn't. This will need a bit of investigation.
• I've created a vanilla Precise Server install which I can keep updated and use as the basis for script testing, by cloning it each time.

Started testing on a VM, running my old script bit by bit and watching the results. I've got as far as a working Jinks server, and one or two builds work, but there will have to be many changes. These are my notes so far:

First time building Jenkins with Precise, I'm running only what I think I need to run, and keeping notes.

#START UNTESTED BIT -- TEST LATER ON DEFAULT INSTALL.#
#First do updates.
echo "Doing system updates before starting on anything else."
apt-get update
apt-get upgrade

#Now add the repositories we want.
echo "Backing up repository list."
cp /etc/apt/sources.list /etc/apt/sources.list.bak

#Uncomment partner repos.
echo "Uncommenting partner repositories on sources list, so we can get Sun Java."
sed -i -re '/partner/ s/^#//' /etc/apt/sources.list
#FINISH UNTESTED BIT#


#First Jenkins
echo "Adding Jenkins repository."
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list

#OK

#Next TEI.
echo "Adding TEI Debian repository."
     gpg --keyserver wwwkeys.uk.pgp.net --recv-keys FEA4973F86A9A497
#NOT OK: HAD TO USE:
#Next TEI.
gpg --keyserver wwwkeys.pgp.net --recv-keys FEA4973F86A9A497
echo "deb http://tei.oucs.ox.ac.uk/teideb/binary ./" > /etc/apt/sources.list.d/tei.list

#OK

#Now we can start installing packages.
echo "Updating for new repositories."
apt-get update

#OK

echo "Installing core packages we need."
apt-get install openssh-server libxml2 libxml2-utils devscripts xsltproc debhelper subversion trang &&
echo "Installing curl, required for some tei building stuff."
apt-get install curl &&

#OK. curl already installed.

#TEI packages
echo "Installing TEI packages."
	apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-sgml jing jing-trang-doc libjing-java rnv texlive-xetex &&
#NOT OK: rnv no longer in repos (but is in TEI, so no worries); linuxdoc-sgml should be linuxdoc-tools. HAD TO USE:

apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-tools jing jing-trang-doc libjing-java texlive-xetex

	apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema saxon nxml-mode-tei tei-p5-xsl tei-p5-xsl2 tei-p5-xslprofiles tei-roma onvdl tei-oxygen zip &&

#NOT OK: nxml-mode-tei has no installation candidate. HAD TO USE:
apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema saxon tei-p5-xsl tei-p5-xsl2 tei-p5-xslprofiles tei-roma onvdl tei-oxygen zip &&

#Setting up configuration for oXygen
mkdir /root/.com.oxygenxml
chmod a+x /root/.com.oxygenxml
mkdir /root/.java
chmod a+x /root/.java
echo "Don't forget to put your licensekey.txt file in the folder /usr/share/oxygen so that oXygen is registered."

#OK

#Various fonts and the like.
echo "Installing fonts we need."
apt-get install ttf-dejavu msttcorefonts ttf-arphic-ukai ttf-arphic-uming ttf-baekmuk ttf-junicode ttf-kochi-gothic ttf-kochi-mincho
echo "The Han Nom font is not available in repositories, so we have to download it from SourceForge."
cd /usr/share/fonts/truetype
mkdir hannom
cd hannom
wget -O hannom.zip http://downloads.sourceforge.net/project/vietunicode/hannom/hannom%20v2005/hannomH.zip
unzip hannom.zip
find . -iname "*.ttf" | rename 's/\ /_/g'
rm hannom.zip
fc-cache -f -v

#OK

#Configuration for Jenkins
echo "Starting configuration of Jenkins."
echo "Getting the Hudson log parsing rules from TEI SVN."
cd /var/lib/jenkins
svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/P5/Utilities/hudson-log-parse-rules
chown jenkins hudson-log-parse-rules
echo "Getting all the job data from TEI SVN."
#Don't bring down the config.xml file for now; that contains security settings specific to 
#Sebastian's setup, and will prevent anyone from logging in. We leave the server unsecured,
#and make it up to the user to secure it.
#svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/config.xml
#chown jenkins config.xml
svn export --force https://tei.svn.sourceforge.net/svnroot/tei/trunk/Documents/Editing/Jenkins/jobs/ jobs
chown -R jenkins jobs
echo "Installing Jenkins plugins."
cd plugins
wget --no-check-certificate http://updates.jenkins-ci.org/latest/copyartifact.hpi
chown jenkins copyartifact.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/emotional-hudson.hpi
chown jenkins emotional-hudson.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/greenballs.hpi
chown jenkins greenballs.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/jobConfigHistory.hpi
chown jenkins jobConfigHistory.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/plot.hpi
chown jenkins plot.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/log-parser.hpi
chown jenkins log-parser.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/scp.hpi
chown jenkins scp.hpi
wget --no-check-certificate http://updates.jenkins-ci.org/latest/WebSVN2.hpi
chown jenkins WebSVN2.hpi

echo "Restarting Jenkins server, so that it finds and initializes all the new plugins."
/etc/init.d/jenkins restart

#OK


PROBLEMS AFTER STARTUP:

1. Builds fail because rnv is not installed. Have to get it from SourceForge:

cd ~
apt-get install libexpat-dev
wget http://downloads.sourceforge.net/project/rnv/Sources/1.7.10/rnv-1.7.10.zip
unzip rnv-1.7.10.zip
cd rnv-1.7.10
./configure
make
make install

2. NON-FATAL, but investigate: in Stylesheets:
	Unable to locate tools.jar. Expected to find it in /usr/lib/jvm/java-6-openjdk-amd64/lib/tools.jar

3. FATAL:

BUILD Build for P5, XSLT 2.0
test -d release/p5 || mkdir -p release/p5/xml/tei/stylesheet/
for i in  bibtex common2 docx dtd docbook epub epub3 fo2 html html5 latex2 nlm odds2 odt profiles/default rdf relaxng rnc slides tbx tite tools txt xhtml2 xsd ; do \
		tar cf - --exclude .svn $i | (cd release/p5/xml/tei/stylesheet; tar xf - ); \
	done
(cd odt;  mkdir TEIP5; saxon -o:TEIP5/teitoodt.xsl -s:teitoodt.xsl expandxsl.xsl ; cp odttotei.xsl TEIP5.ott teilite.dtd TEIP5; jar cf ../teioo.jar TEIP5 TypeDetection.xcu ; rm -rf TEIP5)

/bin/sh: 1: jar: not found
mkdir -p /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp catalog.xml /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp teioo.jar /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp: cannot stat `teioo.jar': No such file or directory
make[1]: *** [installp5] Error 1
make[1]: Leaving directory `/var/lib/jenkins/jobs/Stylesheets/workspace'

This took all day. I first attempted to set the home dir for mholmes on the second disk drive, but when I did that, I ended up with no bash profile. I ended up leaving it at /home/mholmes, but symlinking to specific folders on the second drive instead.

To move VirtualBox vms, I first deleted all snapshots, then moved the disks over (only the disks). Then I created new VMs for the HDs. All working normally, after lots of Windows updates and a bit of tweaking on the Win7 machine (which by default tried to attach the old IDE disk image to a SATA disk controller).

My new box has arrived, and I've started building it up with the next LTS version of Ubuntu (not actually due for release till April). I've installed with hcmc admin logon only, added gnome-shell, and started adding applications. I'll keep hacking at this till 12.04 seems pretty stable, and then switch from my old machine by moving HDs to the new one. In the meantime, there's a lot of testing to do.

Working with RVDB in Belgium, I've been helping to investigate problems building and running the eXist WAR distro (trunk) on Windows. It appears to be the case that the WAR distro does not run correctly on Windows unless an additional stax jar file is added (this jar was recently removed from trunk, as it was thought to be unnecessary). We have now clarified and confirmed the symptoms and solution, and RVDB wil make a detailed bug report. We don't run eXist under Windows ourselves, but 70% of eXist downloads came from Windows users over the last few years, so it's important that the upcoming new version work well on Windows.