Archives for: 2012


Permalink 09:40:45 am, by mholmes, 82 words, 190 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 15

Boot partition out of space

teijenkins found itself unable to complete an update because it was out of disk space. This was initially surprising, but I discovered that it was actually the small boot partition that was out of space, because of the accumulation of old kernels. This command:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

cribbed from this page cleaned out the old kernels and solved the problem.


Permalink 05:12:02 pm, by Greg, 57 words, 158 views   English (CA)
Categories: Labs, Activity log; Mins. worked: 120

More TSM

Got the colonist machine sorted out, at least to the point that they have the most recent data on the VM and have a backup scheme working. That is, this appears to work.
Next step is to make sure that the backup is working, how long it takes, and whether the backup is like rsync or not.


Permalink 01:59:17 pm, by Greg, 247 words, 287 views   English (CA)
Categories: Labs, Activity log, Documentation; Mins. worked: 180

TSM backup in virtual machine

The Colonist crew needed a reliable backup.
I've got TSM installed in their VM and configured to only back up their stuff.
For future reference
1) The opt file that you get from the helpdesk is intended to provide complete backup for an entire machine. We only wanted to back up a single user's directory. I edited the opt file to accommodate this requirement.
2) If TSM is installed by an admin, but the backup application is going to be run by a non-admin, the app will complain that the non-admin user cannot write to the error log, and the app won't run unless the error log is writeable. So, move the error log.
3) Some files cannot be backed up while the user is logged in. Fortunately we don't care about backing up such things as NTUSER and so forth. Remove them.

Note: includes and excludes can be managed in the preferences editor in TSM. You can also edit the dsm.opt file directly (it's in the 'baclient' folder of TSM's install directory.
I removed all of the end user settings provided by the helpdesk's file and replaced it with this:

* ==== END USER SETTINGS ============================================
ERRORLOGNAME "C:\Documents and Settings\All Users\Documents\tsmerror.log"
EXCLUDE "*:\...\*"
INCLUDE "C:\Documents and Settings\WorkerBee\...\*"
Exclude "C:\Documents and Settings\WorkerBee\NTUSER.DAT"
Exclude "C:\Documents and Settings\WorkerBee\NTUSER.DAT.LOG"
Exclude.Dir "C:\Documents and Settings\WorkerBee\Local Settings\Application Data\Microsoft"

The edits should be self-explanatory.

Permalink 12:35:27 pm, by mholmes, 25 words, 197 views   English (CA)
Categories: R & D, Documentation; Mins. worked: 0

Type Unicode characters in Gnome 3

Putting this on the blog because I keep forgetting it:

Type Control+Shift+U, then release and type the hex of the character, then return.


Permalink 09:47:23 am, by sarneil, 193 words, 208 views   English (CA)
Categories: Activity log; Mins. worked: 60

controlling Yubikey LastPass behaviour on browser launch

When I launched a browser (Firefox, Safari) I was immediately presented with the YubiKey authentication panel. I wanted instead to manually launch the login and authentication process.

To prevent that automatic login (i.e. to manually choose to login to lastpass and then provide authentication):
LastPass_Icon/Preferences/Advanced/Open Login Dialog When Start Browser = unchecked
LastPass_Icon/Preferences/Account Settings/YubiKeys/Permit Offline Access = disallow

The default setting for the first is checked and for the second is allow. The first item launches the login dialog, the second allows lastpass to grab the master password from the local cache, and opens the Yubikey dialog. My first attempt was to set the first item to unchecked and leave the second at allow, but that had no apparent effect. Took a couple of emails with their tech support to figure it out.

UPDATE: When I restart the computer and browser launches, I still get the Yubikey authentication box. Firefox and Safari are obviously getting my lastpass master password from somewhere, but the remember password settings in the browsers are disabled and I can't find anything in the OSX keychain, so I'm at a loss.


Permalink 09:33:49 am, by Greg, 246 words, 182 views   English (CA)
Categories: Servers, Activity log; Mins. worked: 90

Met with RE about servers

As Martin notes, we met with RE about the near future of several servers.
a) The new machine, aka Peach, will replace Pear and Grape as soon as it's configured. RE is setting it up with a single Tomcat install, but we're going to try running multiple instances of it instead of having an installed codebase for each instance like we've had in the past. Once we're ready to test it, we'll get Grape's webapps and Pear's webapps on to it and test it for a week or so. When we're happy that things are working we'll bring down Grape and Pear.
b) when Grape comes down, it will be rebuilt as an additional node in our webcluster - this should happen by Christmas.
c) when Pear comes down it will become a BCP (bulk-copy) machine for Peach and Mango (the DB server), providing us with a rapid-recovery scheme. This build should be done by the end of February.
d) Once the CFI machine is purchased it will be replacing arugula as the NFS/disk box, and arugula will be added to the cluster. Timeline ASAP.
e) Lettuce will wait until we have a chance to put time in to the MoEML map refresh. We expect to begin in the new year. When Lettuce is no longer required for MoEML we'll add it to the cluster.
f) RE will look in to Cress and let us know what he thinks we should do with it.

Permalink 09:21:48 am, by Greg, 79 words, 220 views   English (CA)
Categories: Servers, Activity log; Mins. worked: 10

ehcache on ise machine

Got a call from sysadmin this morning about the ise machine filling its fs with It has been addressed in cocoon.xconf (see here) but we aren't sure how long it's been since last dealing with the problem.
So, sysadmin has noted it in the machine's config log(?) and I'm noting it here that tomcat was restarted this morning in order to remove the file, which had filled the filesysytem (the file was about 4GB).


Permalink 04:28:53 pm, by mholmes, 7 words, 241 views   English (CA)
Categories: Servers, Activity log; Mins. worked: 90

Meeting with RE re servers

GN will blog the outcomes. Posting time.


Permalink 09:53:52 am, by mholmes, 127 words, 210 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 120

Updates to JavaScript code for HotPot exercises

A change in the user agent string for Firefox 17 has triggered an error message in HotPot and Quandary exercises. An update to the applications will be required, which we'll do at home, but in the meantime all the existing exercises we provide to departments needed to be updated, including Greek, Latin, Spanish, XSLT (DHSI), Hulquminum and Arabic. Those have all now been updated.

This is basically what I did:

  • Search and replace with regex on the dir trees in Oxygen:
    ([\t ]*alert\('Your browser may not be able to handle this page.'\);)
  • rsync of the HTML files to the site -- e.g.:
    rsync --verbose --progress --stats --recursive --include '*/' --include '*.htm' --exclude '*' arabic/
Permalink 09:15:59 am, by Greg, 53 words, 185 views   English (CA)
Categories: Labs, Activity log; Mins. worked: 120

Colonist TSM

I'm setting up a TSM node for the Colonist project and am working on getting it running under an XP virtual machine.
The node is set up, with Martin and Stewart as co-admins.
The software is installed, but I can't get it to send any data to for some reason.

Permalink 09:13:54 am, by Greg, 76 words, 139 views   English (CA)
Categories: Labs, Activity log; Mins. worked: 60

Dead drive

Forgot to post this when it happened, but beet's SSD died last week, and I have been unable to recover any data. Unfortunately, there was some data loss for one project. Not unrecoverable, but an annoyance nonetheless.
While waiting for a replacement drive I put a old-school drive in beet and got it up and running. The replacement SSD has arrived and I'm now waiting for an opportunity to put the new drive in to beet.


Permalink 04:34:04 pm, by Greg, 186 words, 320 views   English (CA)
Categories: Labs, Activity log, Activity log, Tasks; Mins. worked: 60

Spontaneous reboots

We've been having spontaneous reboots on several machines in the last two months or so.
We've had an electrician double-check the power we're getting and all is well.
Looking in to potential computer-based issues I discover that many people experience this kind of thing with SSDs on Sabayon, Debian, Ubuntu, CentOS, and likely other distros. There do not seem to be any real solutions, but suggestions include the usual:
1) Get the latest firmware for the drive. Right now we have at least 2 different model of SSD in our machines (haven't checked Martin's yet): OCZ Vertex (96GB running f/w v1.6) and Vertex2 (115GB running f/w v1.29 or 1.33). Firmware and general info on OCZ drives can be found here. Firmware is here
2) Adjust fstab and /etc/rc.local like this

I'll come back to this next week.

Figured it out. The problem was I stupidly changed the security update config to reboot after a security update gets done. It explains everything. The file has been edited, the package updated. Now I wait with my fingers crossed...


Permalink 04:41:56 pm, by mholmes, 104 words, 171 views   English (CA)
Categories: Labs, R & D, Documentation, Documentation; Mins. worked: 15

Configuring Gnome 3

Some notes on how to configure Gnome 3, in preparation for trying to roll it out to the lab machines:


Permalink 09:17:11 am, by Greg, 45 words, 191 views   English (CA)
Categories: Labs, Activity log, Documentation; Mins. worked: 10

Force Mac OS to boot in 64 bit mode

Having built the Macs from a 32bit image I find that they are running in 32 bit mode. No surprise, but it *is* reversible. Run this in the terminal sudo nvram boot-args="arch=x86_64" and reboot. The machine should run in 64 bit mode from now on.


Permalink 09:50:22 am, by Greg, 181 words, 716 views   English (CA)
Categories: Servers, Activity log; Mins. worked: 60

VMware ESXi license problem

Due to my misunderstanding how the licensing works for fennel's ESXi server, our license lapsed the other day and we were unable to restart a VM.

For future reference, we now have to pay a fee for academic licenses for VMware products. We just got a 3 year subscription for ~$800 after taxes. What this provides is a 3 year lifespan on our "web store". Any approved HCMC user (right now this is only me) can go to the web store and download a 'free' copy of most VMware offerings. The ESXi server license I just downloaded says there is a 1 year term on the license (it looks like it's a full calendar year plus, it expires on Dec. 31/2013), so we'll need to head back to the store next December and get a whole new license - it doesn't get renewed.

Our WebStore URL:

Oct 18, 2013: I applied for, and received a new activation code for ESXi server and sent in a request to sysadmin to apply it.


Permalink 05:41:25 pm, by mholmes, 78 words, 191 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 120

Build of temporary Jinks server

GN and I repurposed Plum to replace the temporarily-suspended teijenkins machine, while licensing issues with VMWare are sorted out. This functioned as a fresh test of the Jenkins build script, which went great with two little hiccups (TEI packages have changed, and zip is no longer installed on Ubuntu Server by default). Those changes have been added to the build script in SVN. The new machine is running (but very slow to build, due to its Arm processor).


Permalink 10:44:31 am, by Greg, 258 words, 156 views   English (CA)
Categories: Servers, Activity log, Documentation; Mins. worked: 10

Extending a volume in linux

If you need to add a new drive to (and extend) the LVM setup:
1) Make sure the drive is free of partitions. Install the drive and power up.

2) Find out what the machine is calling the new drive. This will be something in the sd[a-z] range, and is not predictable - you'll need to check before moving on. For the purposes of this post, we'll say that it's called 'sdc'.

3) Create a physical volume on the new disk:

sudo pvcreate /dev/sdc

--- if you only need to extend an existing volume you can start here ---

4) We need to know what the existing volume-groupe and volume names are:

sudo vgdisplay

should return something that starts with 'VG Name vgpool', where 'vgpool' is the string we need.

5) To extend the volume group to include our new device:

sudo vgextend vgpool /dev/sdc1

6) Extend the logical volume by adding 20GB to the volume - note that we can access the volume at /dev/mapper/<volume-group-name>/<volume-name>

sudo lvextend -L+20G /dev/mapper/vgpool-lvrepo

** if you don't know how much disk you have left on any physical device, run this:

sudo hdparm -I /dev/sdc | grep GB

7) You also need to extend the filesystem, and you can't do it while the volume is mounted:

sudo umount /dev/mapper/vgpool-lvrepo

8) You need to run fsck before resizing a filesystem:

sudo e2fsck -f /dev/mapper/vgpool-lvrepo

9) Now you can resize the filesystem:

sudo resize2fs /dev/mapper/vgpool-lvrepo

6) And remount:

sudo mount /dev/mapper/vgpool-lvrepo /mountlocation


Permalink 04:50:36 pm, by mholmes, 149 words, 241 views   English (CA)
Categories: Servers, Documentation; Mins. worked: 0

Restarting Tomcat on Pear

The Tomcat setup has recently changed with the move of stable apps to Grape, so this is an updated set of instructions. MDH and GN have rights to run the tomcat script; SA probably does too.

At this point, tomcat will seem to have stopped, but it probably hasn't. Confirm by doing this:

  • ps aux | grep tomcat

If you see both tomcat running, you'll have to kill -9 tomcat. Since the process runs as hcmc, and you can't sudo-kill it, do this:

  • su hcmc
  • kill -9 [the process number]

Do ps aux again to check that the process is dead (you should not see tomcat running now). Assuming that worked, then

  • exit (to return to mholmes)
  • sudo ./tomcat start

Wait a couple of minutes, then check that tomcat is back up and that projects are working.


Permalink 02:01:03 pm, by mholmes, 248 words, 241 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 120

TEI Jenkins CI servers: new setup

This is my documentation of recent changes to the way the TEI Jenkins Continuous Integration Server VMs are configured.

The new VM whose machine name is teijenkins1204 is now running Jenkins on port 8080, and also Apache proxying Jenkins on port 80. JA (sysadmin) has pointed the DNS entry for at this machine, so the Jenkins instance is accessible on port 80 and 8080 there. The dual access allows for normal access to Jenkins on port 80 (convenient), but also for a situation in which we might want to put up a temporary warning page on port 80 while solving a problem with Jenkins accessed through 8080, then restore the proxy setup. I followed these instructions to set up proxying.

That machine is also running denyhosts.

The old machine hostname has been changed to teijenkinsdev, and JA has pointed at it. The Jenkins service has been disabled on this machine using sysv-rc-conf. When I ran that tool, it showed X characters for the jenkins service on run levels 2, 3, 4 and 5. I removed all of them; to re-enable the service, run it again and replace them. In the meantime, you can run Jenkins manually using sudo /etc/init.d/jenkins start. The plan is to reprovision that machine with more RAM and disk space, and then build it up with Ubuntu 12.10 to test and tweak the build script; doing this for every Ubuntu release will make it less messy to migrate to the next LTS when it comes out.


Permalink 02:27:29 pm, by mholmes, 88 words, 162 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 60

Exist WAR building

Spent some time with GN trying to get builds working on Plum. We found a couple of bugs in the code, which were reported and fixed, and I also added some new false positives to the log-parse file. Now we're back to apparently-working builds (although with warnings) for three out of the four builds OK, and the last one in process. I need to start testing recent builds with MoEML next week if I have a chance. We've commented out the svn module because that was causing problems.


Permalink 03:32:59 pm, by mholmes, 30 words, 319 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 20

Running roma locally pointing at SVN stylesheets and source

This seems to work:

roma --xsl=/home/mholmes/WorkData/tei/sf_repo/trunk/Stylesheets --localsource=/home/mholmes/WorkData/tei/sf_repo/trunk/P5/Source/guidelines-en.xml --isoschematron london_all.odd


Permalink 03:35:16 pm, by mholmes, 170 words, 149 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 30

Problem with Agenda (2)

MC changed the include_path and extension_dir settings -- it appears there was an update in May, at which time these should have been changed but weren't. However, the app still won't work. There are two remaining settings that look wrong to me:

MYSQL_INCLUDE     no value
MYSQL_LIBS     no value 

On a phpinfo.php for PHP5, these show as:

MYSQL_INCLUDE     -I/usr/local/include/mysql
MYSQL_LIBS     -L/usr/local/lib/mysql -lmysqlclient 

I'm waiting to see if a fix to these will do the job. If not, we'll have to start picking carefully through every one of the settings, trying to figure out what might be wrong. The only other thing I can think of is that PEAR might be broken somehow in this new install.

In the long run, we should get Agenda shifted from PHP4 to PHP5. If this update was done in May, and we're the first to report that the paths were broken, then clearly hardly anyone is using it any more.

Permalink 03:09:01 pm, by mholmes, 380 words, 236 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 120

Problem with Agenda application

This morning I was contacted by two different departments to report that the Agenda application used to schedule classes is not working. On investigation, I discovered that the login page is truncated immediately at the point where the first call to the database is made. After reading through the code, I couldn't see anything wrong, so I suspected a connection problem such as an ACL issue between and csmgenr2 (the mysql server). The db is there, functioning, and accessible through PhpMyAdmin.

I raised a ticket with sysadmin, and MC got back to me to say that there doesn't seem to be a connection issue, but the error in the logs was with a PHP include:

2012-08-29T08:43:23-07:00 user.notice php_cgi: PHP
Warning: main() [<a href='function.include'>function.include</a>]: Failed
opening 'DB.php' for inclusion
on line 11

The error comes when the db connection script tries to include DB.php, which is the PEAR library this project uses for accessing MySql. This sent me to check the PHP settings info in phpinfo.php, and I found a number of oddities:

  • The PHP version is 4.4.9,
  • the configure command output includes this: './configure' '--prefix=/usr/local/php-4.4.9', and
  • the configuration file is here: /usr/local/php-4.4.9/lib/php.ini.
  • HOWEVER, the extension_dir is /usr/local/php-4.4.8/lib/php/extension, and
  • the include_path is .:/usr/local/php-4.4.8/include/php:/usr/local/php-4.4.8/lib/php.

My current theory is that when PHP was updated from 4.4.8 to 4.4.9, those key settings in the master php.ini file should have been changed, but weren't; and as a result, the include of DB.php is failing. I've reported this back to sysadmin, and I'm waiting for a response. In the meantime, I've tried overriding the include_path setting in a local php.ini, but I can't get that to work; perhaps those settings can't be overridden.

In the meantime, I could no longer bear to look at the logo which proclaimed "Agenda / Organize. Mangage. Simplfy" (sic, seriously), so I've fixed that.


Permalink 08:29:41 am, by mholmes, 41 words, 242 views   English (CA)
Categories: Servers, Labs, R & D, Activity log, Documentation, Documentation; Mins. worked: 15

Switching from old repo to new in Subversion

Just blogging something I keep having to look up. This is how to switch your SVN repo from the old tapor URL to hcmc:

svn switch --relocate[reponame][reponame] 


Permalink 11:51:05 am, by mholmes, 79 words, 269 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 10

Accented characters on Gnome 3

Just blogging this so that the next time I forget it, I don't have to rediscover it again:

  • Set up a Compose Key -- I use Right Alt -- in Keyboard Layout/Options.
  • Type accented characters by holding down that key while you type the combining symbol (e.g. backtick) followed by the character (e.g. e), then release to get the char. NOTE: sometimes you have to type them in the other order (character first, then diacritic key).


Permalink 08:37:47 am, by sarneil, 282 words, 165 views   English (CA)
Categories: Notes; Mins. worked: 0

VirtualBox : new windows virtual machine on a Mac

After numerous tries with various approaches, I finally managed to end up with a 40 Gig virtual machine to host Windows.
The procedure that worked for me was to
- clone the existing hard drive using the function in the GUI of Virtual Box (and not the command line). VB created a new folder and put the new virtual machine file into that folder.
- go into the command line and make the virtual file 40 gigs in size
sudo VBoxManage modifyhd newVirtualMachine.vdi --resize 40960
- start up virtual box
- select the new virtual machine
- click on settings, click on disk, right click, choose "Remove"
- still on settings, right
- in Windows, start Windows Computer Management
- go to Disk/Storage management
- right click on existing small partition, select "extend"
- click OK in dialog box

When I tried using the command line to clone the virtual drive, I got the file, but the GUI in Virtual Box didn't see it.
I tried a couple of other partition managers before stumbling on the Windows Computer Management and either I couldn't get them to install properly or I couldn't get them to extend the smaller partition to a bigger one.

The most helpful website I found was though I did not create the hard drive and then clone the existing drive to the new drive as that page says, rather I cloned the existing drive and then resized it to 40G.

Finally, when I started windows in the new virtual machine, I got a "this copy of windows may not be authentic" warning. I followed the link on the warning and that "authenticated" my copy.


Permalink 02:32:42 pm, by mholmes, 131 words, 198 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 90

New Jenkins running and visible

Sysadmin fixed the ACLs (there were two sets of entries, apparently, which confused the process for a while), and now the machine is up and running with Jenkins on port 8080. I tried moving it to port 80, but was prevented by the Ubuntu rule which won't allow a service not running as root to run on a port below 1024 (and we definitely don't want Jinks running as root). So I may end up running Apache just so I can proxy it to port 80. Seems like overkill, but there doesn't seem to be an alternative solution.

I'm going to run the two machines side by side until I come back from vacation, so we can make sure the new one is stable before we switch to it and bring down the old one.


Permalink 02:31:08 pm, by mholmes, 59 words, 193 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 60

New Jenkins built and running, but currently invisibule

The new VM was set up today, so Greg built it with Ubuntu 12.04, and I ran my Jinks-builder script on it. Everything seemed to go perfectly, and presumably it's building away behind the scenes, but because the ACL prevents any access to it except on port 22, I can't get in to finish the setup yet. We'll get there eventually...

Permalink 08:23:26 am, by mholmes, 8 words, 642 views   English (CA)
Categories: Announcements; Mins. worked: 0

Showing date on Gnome 3 panel

gsettings set show-date true


Permalink 03:04:00 pm, by mholmes, 89 words, 189 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 180

Jenkins build script now working

I've completed the Jenkins build script, after some more tweaking and testing, and it's working fine. I've updated the wiki page about it, and GN has put in a request for a new VM we can build up to replace the old one. Learned quite a lot in the process of doing this. The horrible curl request in the previous post actually works, if you do it at the right time (after downloading all the jobs, setting them up, and restarting Jenkins so it knows the jobs are there).


Permalink 04:34:22 pm, by mholmes, 532 words, 187 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 20

For testing tomorrow

I think we may be able to do the form submit to generate configs like this:

curl -d "_.rawWorkspaceDir=%24%7BITEM_ROOTDIR%7D%2Fworkspace&_.rawBuildsDir=%24%7BITEM_ROOTDIR%7D%2Fbuilds&system_message=&_.numExecutors=2&_.quietPeriod=5&_.scmCheckoutRetryCount=0&namingStrategy=0&stapler-class=jenkins.model.ProjectNamingStrategy%24DefaultProjectNamingStrategy&stapler-class=jenkins.model.ProjectNamingStrategy%24PatternProjectNamingStrategy&_.namePattern=.*&slaveAgentPort.type=random&stapler-class=hudson.markup.RawHtmlMarkupFormatter&" http://localhost:8080/configSubmit
Permalink 04:17:53 pm, by mholmes, 71 words, 164 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 120

Jenkins: updated wiki page

Updated the wiki page after testing the build a couple more times, so now the wiki points to the current version of the script in SVN and gives appropriate instructions. However, I still haven't solved the log-parse-rules file issue. I will have to figure out some way to watch what happens when that Save button in the config screen is clicked, and mimick it using curl or wget. It's very messy.

Permalink 11:09:53 am, by mholmes, 132 words, 170 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation; Mins. worked: 30

Getting Jinks version number for config file

I'm trying to figure out an easy way of putting a Jenkins config (config.xml) file into the directory where it's expecting it to be, in the hope that this will cause Jinks to use the correct log parse rules when running jobs. I can create a generic config file, but I want to make sure that it doesn't have the wrong Jenkins build number in it. Once I have the right build number from the current Jenkins instance, I'll be able to pass it as a parameter to an XSLT transformation. Here's one approach which works:

wget -O /tmp/jinks.html http://localhost:8080
JINKSBUILD=`grep -oP '(?<=Jenkins ver\. )(1.[0-9\.]+)(?=</a>)' /tmp/jinks.html`

I'm also going to investigate the option of using the Jenkins CLI interface.


Permalink 04:08:42 pm, by Greg, 25 words, 179 views   English (CA)
Categories: Activity log; Mins. worked: 60

New monitors

Finished setting up all the new monitors after finding a bunch of HDMI cables on my chair this morning. Everyone now has a 3840x1080 desktop!


Permalink 03:24:42 pm, by mholmes, 37 words, 159 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 45

All data from old machine retrieved

I've now got all important data off my old Lucid box (the Drive2 data), onto my new machine. We can now take out Drive2 from that machine and use it in a cradle, and repurpose the machine.

Permalink 03:18:25 pm, by mholmes, 235 words, 149 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation; Mins. worked: 180

More fixes, still not quite there with Jenkins...

Problems solved this afternoon:

  • rnv was failing to download and build; ampersands in the url query string needed to be backslash-escaped, and the file name needed to be specified with the -O flag.
  • The log parse rules were not actually being used, even though they were downloaded, and were referred to in the job configs. I think this is because a file called hudson.plugins.logparser.LogParserPublisher.xml also needed to be there, to specify that the other file exists somehow. That's the theory anyway; not tested yet.
  • Emailing could never have worked without going to a lot of trouble to mess with Jenkins's setup, so I'm simply sidestepping it and leaving it up to the user to set it up if they want to. I'm removing mine and SR's email addresses from the job configs.
  • The Priority Sorter is working, but didn't actually solve the sequencing problem for the first build. Jinks managed to complete OxGarage, Roma and Stylesheets1 before Stylesheets completed, so it just had time to start and fail on P5-Test before the required artifacts were there. I'm consideriong making P5-Test a downstream job from Stylesheets, which should solve it once and for all.
  • Core files are now in the TEI repo, as they should be, and the hudson log parse rules have been moved from P5/Source to Documents/Editing/Jenkins, where all the rest of the stuff is.
Permalink 10:07:47 am, by mholmes, 152 words, 547 views   English (CA)
Categories: Announcements; Mins. worked: 120

Jenkins build progress

  • Added the OxGarage messages to the log parser rules file, so they no longer show up as errors.
  • Re-organized the structure of the build process so EULAs come first, and the rest of the build should be able to proceed unattended, using the -y flag on apt-get install commands.
  • Moved the requirement for an Oxygen license to the beginning of the process, so that the user must provide it before the build can proceed. This prevents the Stylesheets build from failing first time out, because of the missing license.
  • Added more useful information messages, particularly before the build.
  • Tested the Priority Sorter plugin for Jenkins, which should enable us to put P5 builds at the end of the queue, so that we can be sure that Stylesheets will build first.

The last thing to do is to XSLT the job configs to insert the user's email address instead of SR's and mine.


Permalink 05:13:18 pm, by mholmes, 129 words, 156 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation; Mins. worked: 45

rnv problem solved

It turns out that the response at the command line when you try to run rnv is completely misleading. It leads you to believe that the executable is missing, whereas it's clearly there. Actually the problem is caused by the fact that the rnv installed from the TEI repo is 32-bit, and won't run on a 64-bit kernel.

So instead of installing it from the repos, I'm downloading and building it instead:

apt-get install libexpat-dev
cd rnv-1.7.10
make install

In view of this, I don't think rnv should be in the TEI repos at all, unless it can be done in such a way as to provide the right build for the host architecture.

Permalink 04:16:36 pm, by mholmes, 604 words, 209 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation; Mins. worked: 240

More progress with Jinks 2012 build

I now have a more convenient setup for creating and testing Jinks builds. This is how it works:

  • There's a sort of "seed" vm called PreJenkins in /home/mholmes/VirtualBox VMs/
  • That's a fully-updated vanilla install of Precise server.
  • In my tei/jenkins directory, there's a script called When you run that script, it clones the vanilla server to create a new VM called Jenkins2012A. It also configures that VM so that you can see its port 8080 (Jenkins) on the host's 7070, and so that you can ssh into it on port 2012.
  • After running the above script, you start the newly-created VM. You log in as hcmc, and sudo su, then you run a script you'll find there called (also in my local tei/jenkins directory on my host).
  • The script first scps a copy of the from the host into the hcmc home directory (pulling this in, rather than having it there in the seed, enables me to tweak the script easily while working on it). It runs the script.
  • At this point, you're seeing what the ordinary user of the jenkins build script would see.
  • Once the build is complete, and you exit from the build script, scps a copy of our Oxygen license from the host into the right location, so that Oxygen is registered.
  • Now Jenkins should be running on the new machine. You can now run a script on the host called, which will send Firefox to the right port, and ssh into the machine. The connection to the machine is done like this:
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l hcmc -p 2012 localhost
    which precludes checking of the machine's key, or storage of its key; this avoids the problem where every time you clone the pre-seed to create a new VM, it has a different key, and the host complains that the key in known_hosts has changed.
  • Then you wait and watch to see if Jenkins builds the jobs OK.

This is what remains to be done:

  • Fix the rnv problem. It's installed now from the TEI packages, and is clearly there, but make and bash are unable to find it:
    -bash: /usr/bin/rnv: No such file or directory
    Very weird indeed. There are executable copies in /usr/bin and in /usr/bin/X11, just like on my local machine, where they work fine.
  • Figure out what to do about job configs which include mine and SR's email addresses. We should probably XSTL the config.xml files during the setup process to replace SR's email with the user's own, which we can ask for.
  • Solve the problem whereby P5-Test builds and fails before Stylesheets has built for the first time. Stylesheets needs to build once successfully before the P5 jobs can build; thereafter, they're independent. There must be a way to force Jenkins to build Stylesheets before P5-Test the first time out.
  • List the error messages and warnings that show up ONLY on the first checkout/build of the jobs. There are lots of these in OxGarage, and possibly elsewhere. These need to be put into the hudson-log-parse-rules file so that a first-time user of the script is not worried by a lot of errors that will never show up again, and aren't relevant.
  • Document and publish the script, by updating the page on the TEI wiki.
  • Get a real VM to replace our current teijenkins, and when it's working, repoint the domain and bring the old one down.


Permalink 05:34:39 pm, by mholmes, 97 words, 201 views   English (CA)
Categories: Labs, Activity log, Documentation; Mins. worked: 60

Problems with LibreOffice Writer

SM working on MoL has been seeing LibreOffice Writer crash frequently when editing a complex document with comments. I've worked with similar documents without problems, though.

I looked around for alternative word-processors, but neither Calligra Words nor AbiWord handle comments. Noting that Writer works fine for me on Precise + Gnome 3, and fine for CB with Gnome 2, I've now installed Gnome Shell on Radish, and we'll see if that solves the problem. Never liked Unity anyway.

In the process of doing this, I noticed that an HCMC style deb is failing to update itself when doing apt-get upgrade.


Permalink 03:18:37 pm, by mholmes, 302 words, 157 views   English (CA)
Categories: Servers, R & D, Activity log; Mins. worked: 180

Great progress on the Jinks 2012 build

I've been working through the problems on the Jinks build with SR, and we're close to a working system. This is where we're at right now:

  • Problems caused by the absence of a JDK have been solved by apt-get install openjdk-6-jdk (so we apparently don't actually need Oracle).
  • There are still a few error messages about not being able to write to /root/.java/.com.oxygenxml.rk, which I'm puzzled by -- I even see them if I make the file world-writable. I'm inclined to suppress these messages in the log-parse rules.
  • SR has added rnv to the TEI packages, so I'm now installing it as part of that group of installs; no need to get it from SourceForge.
  • We still have a problem with some error messages in the OxGarage build, but they don't seem to be indicating a series issue.
  • All of the P5 jobs have now completed apparently successfully, but there one set of errors on TEIP5:
    tar: p5odds.rng: Cannot stat: No such file or directory
    tar: p5odds.rnc: Cannot stat: No such file or directory
    tar: p5odds-examples.rng: Cannot stat: No such file or directory
    tar: p5odds-examples.rnc: Cannot stat: No such file or directory
    tar: Exiting with failure status due to previous errors
    and in fact none of those four files are in the release/xml/tei/odd directory of the archived artifacts, whereas they are there in my current Jenkins artifacts. So I think this build should be tagged as a failure. As it is, although the console says it will be tagged as a fail, it isn't. This will need a bit of investigation.
  • I've created a vanilla Precise Server install which I can keep updated and use as the basis for script testing, by cloning it each time.
Permalink 11:04:27 am, by Greg, 223 words, 168 views   English (CA)
Categories: Labs, Activity log; Mins. worked: 60

B045 setup for presentations

Tested projector with new build/Intel Graphics and wireless keyboard.
Turns out that the projector looks fabulous as long as you use its native resolution. The bad news is that its native resolution is 1024x768. Any other 4:3 resolution that works with the Intel Graphics chip looks terrible on the screen.
To get mirroring to work, you need to plug the HDMI plug in to the DVI port on the computer (via the HDMI-DVI converter) and use a VGA cable for the monitor itself. Any other combo is fraught.
Wireless keyboard/trackpad has a nice feel to it but is a real pain to use as it loses its connection regularly. Pressing the connect button MAY work, but it doesn't always. I tried elevating the receiver, which helps some (but not enough). Martin replaced the batteries and reports that performance is better.

If you want to do a presentation in B045 using our equipment:
Projector plugs in to the orange DVI converter and then in to the DVI port on the computer. Monitor plugs in to the computer via the VGA cable.
Connect wireless keyboard: plug the receiver in to the computer using a USB extension cable and elevate the receiver so it's within line-of-sight of the keyboard. Turn on the (keyboard) power and depress the connect button. It should work more-or-less right away.


Permalink 02:30:26 pm, by mholmes, 1005 words, 402 views   English (CA)
Categories: Servers, R & D, Activity log; Mins. worked: 120

Work on new TEI Jenkins server based on Precise

Started testing on a VM, running my old script bit by bit and watching the results. I've got as far as a working Jinks server, and one or two builds work, but there will have to be many changes. These are my notes so far:

First time building Jenkins with Precise, I'm running only what I think I need to run, and keeping notes.

#First do updates.
echo "Doing system updates before starting on anything else."
apt-get update
apt-get upgrade

#Now add the repositories we want.
echo "Backing up repository list."
cp /etc/apt/sources.list /etc/apt/sources.list.bak

#Uncomment partner repos.
echo "Uncommenting partner repositories on sources list, so we can get Sun Java."
sed -i -re '/partner/ s/^#//' /etc/apt/sources.list

#First Jenkins
echo "Adding Jenkins repository."
wget -q -O - | apt-key add -
echo "deb binary/" > /etc/apt/sources.list.d/jenkins.list


#Next TEI.
echo "Adding TEI Debian repository."
     gpg --keyserver --recv-keys FEA4973F86A9A497
#Next TEI.
gpg --keyserver --recv-keys FEA4973F86A9A497
echo "deb ./" > /etc/apt/sources.list.d/tei.list


#Now we can start installing packages.
echo "Updating for new repositories."
apt-get update


echo "Installing core packages we need."
apt-get install openssh-server libxml2 libxml2-utils devscripts xsltproc debhelper subversion trang &&
echo "Installing curl, required for some tei building stuff."
apt-get install curl &&

#OK. curl already installed.

#TEI packages
echo "Installing TEI packages."
	apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-sgml jing jing-trang-doc libjing-java rnv texlive-xetex &&
#NOT OK: rnv no longer in repos (but is in TEI, so no worries); linuxdoc-sgml should be linuxdoc-tools. HAD TO USE:

apt-get install psgml xmlstarlet debiandoc-sgml linuxdoc-tools jing jing-trang-doc libjing-java texlive-xetex

	apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema saxon nxml-mode-tei tei-p5-xsl tei-p5-xsl2 tei-p5-xslprofiles tei-roma onvdl tei-oxygen zip &&

#NOT OK: nxml-mode-tei has no installation candidate. HAD TO USE:
apt-get install trang-java tei-p5-doc tei-p5-database tei-p5-source tei-schema saxon tei-p5-xsl tei-p5-xsl2 tei-p5-xslprofiles tei-roma onvdl tei-oxygen zip &&

#Setting up configuration for oXygen
mkdir /root/.com.oxygenxml
chmod a+x /root/.com.oxygenxml
mkdir /root/.java
chmod a+x /root/.java
echo "Don't forget to put your licensekey.txt file in the folder /usr/share/oxygen so that oXygen is registered."


#Various fonts and the like.
echo "Installing fonts we need."
apt-get install ttf-dejavu msttcorefonts ttf-arphic-ukai ttf-arphic-uming ttf-baekmuk ttf-junicode ttf-kochi-gothic ttf-kochi-mincho
echo "The Han Nom font is not available in repositories, so we have to download it from SourceForge."
cd /usr/share/fonts/truetype
mkdir hannom
cd hannom
wget -O
find . -iname "*.ttf" | rename 's/\ /_/g'
fc-cache -f -v


#Configuration for Jenkins
echo "Starting configuration of Jenkins."
echo "Getting the Hudson log parsing rules from TEI SVN."
cd /var/lib/jenkins
svn export
chown jenkins hudson-log-parse-rules
echo "Getting all the job data from TEI SVN."
#Don't bring down the config.xml file for now; that contains security settings specific to 
#Sebastian's setup, and will prevent anyone from logging in. We leave the server unsecured,
#and make it up to the user to secure it.
#svn export
#chown jenkins config.xml
svn export --force jobs
chown -R jenkins jobs
echo "Installing Jenkins plugins."
cd plugins
wget --no-check-certificate
chown jenkins copyartifact.hpi
wget --no-check-certificate
chown jenkins emotional-hudson.hpi
wget --no-check-certificate
chown jenkins greenballs.hpi
wget --no-check-certificate
chown jenkins jobConfigHistory.hpi
wget --no-check-certificate
chown jenkins plot.hpi
wget --no-check-certificate
chown jenkins log-parser.hpi
wget --no-check-certificate
chown jenkins scp.hpi
wget --no-check-certificate
chown jenkins WebSVN2.hpi

echo "Restarting Jenkins server, so that it finds and initializes all the new plugins."
/etc/init.d/jenkins restart



1. Builds fail because rnv is not installed. Have to get it from SourceForge:

cd ~
apt-get install libexpat-dev
cd rnv-1.7.10
make install

2. NON-FATAL, but investigate: in Stylesheets:
	Unable to locate tools.jar. Expected to find it in /usr/lib/jvm/java-6-openjdk-amd64/lib/tools.jar


BUILD Build for P5, XSLT 2.0
test -d release/p5 || mkdir -p release/p5/xml/tei/stylesheet/
for i in  bibtex common2 docx dtd docbook epub epub3 fo2 html html5 latex2 nlm odds2 odt profiles/default rdf relaxng rnc slides tbx tite tools txt xhtml2 xsd ; do \
		tar cf - --exclude .svn $i | (cd release/p5/xml/tei/stylesheet; tar xf - ); \
(cd odt;  mkdir TEIP5; saxon -o:TEIP5/teitoodt.xsl -s:teitoodt.xsl expandxsl.xsl ; cp odttotei.xsl teilite.dtd TEIP5; jar cf ../teioo.jar TEIP5 TypeDetection.xcu ; rm -rf TEIP5)

/bin/sh: 1: jar: not found
mkdir -p /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp catalog.xml /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp teioo.jar /var/lib/jenkins/jobs/Stylesheets/workspace/debian-tei-p5-xsl2/debian/tei-p5-xsl2/usr/share/xml/tei/stylesheet
cp: cannot stat `teioo.jar': No such file or directory
make[1]: *** [installp5] Error 1
make[1]: Leaving directory `/var/lib/jenkins/jobs/Stylesheets/workspace'


Permalink 02:26:04 pm, by mholmes, 107 words, 84 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 160

Switching to new machine

This took all day. I first attempted to set the home dir for mholmes on the second disk drive, but when I did that, I ended up with no bash profile. I ended up leaving it at /home/mholmes, but symlinking to specific folders on the second drive instead.

To move VirtualBox vms, I first deleted all snapshots, then moved the disks over (only the disks). Then I created new VMs for the HDs. All working normally, after lots of Windows updates and a bit of tweaking on the Win7 machine (which by default tried to attach the old IDE disk image to a SATA disk controller).

Permalink 09:00:00 am, by Greg, 504 words, 150 views   English (CA)
Categories: Labs; Mins. worked: 0

Build system complete

There is now a complete build system for Ubuntu 12.04 set up. Here's how it works.
A machine called papaya is set up with a mirror of the precise repo (main, restricted, security, extras, universe and multiverse), Google's repos for Earth and Chrome, Oracle's Virtualbox and a groovy icon set (Faenza). It also has a reprepro setup that runs a kind-of local ppa with a few home-brew apps for use in the HCMC labs: hcmc-desktop (a metapackage that installs a bunch of necessary software and sets up stuff like printers and so forth), hcmc-auth (for LDAP logins), hcmc-oxygen (xml editor) and hcmc-style (adjusts the candy-cane look to a greyscale look).

In order to manage the mirrors see the setup documention here. The mirror should automatically update itself every day. To add a new repo to be mirrored run the script called in the admin user's homedir. It's a wizard-kind-of-thing that leads you by the hand through the process.

In order to add a package to the hcmc repo there is a script in the admin homedir called It is extremely basic, adding anything it finds in the admin user's homedir/packages directory to the repo, ignoring anything that is already in the repo. It demands a passphrase (twice) for my gpg key (ask me for it) in order to add a package.

To remove a package from the hcmc repo there is a script called in the admin user's homedir. It takes a package name as an argument (e.g. hcmc-desktop) and also demands my gpg passphrase

In order to install a fresh Ubuntu 12.04 you can either do the vanilla install first, then run the Bob the Builder script, or you can grab the hcmc-mini.iso from and put it on a thumb drive using something like unetbootin, which is in the repos. The hcmc-mini.iso is a custom-built iso which has a set of preseeds built in to it so it sets up everything required in the HCMC labs. The great thing about using it is that it pulls all packages for the install directly from papaya, so there is no need to update the machine after the install. After the install you're left with a completely set up HCMC lab machine that's ready to go.
***** NOTE: the admin user that gets set up by hcmc-mini is preseeded with a LAME password because I have so far been unsuccessful in creating an md5 hash to store in the preseed - although it is *supposed* to work. I'll change the preseed if I can get it to work. In the meantime, chage the admin user's password after the build is finished.

The hcmc-mini.iso image is created by a script in the admin user's homedir called It downloads a stock netboot image from an official Ubuntu source, mangles it to include the necessary preseeds, then repacks it in to a bootable iso image, storing it in /var/www/iso/precise.


Permalink 03:10:48 pm, by Greg, 188 words, 340 views   English (CA)
Categories: Announcements; Mins. worked: 0

Tomcat launcher

I wrote a quick bash script that lets me start or stop a local instance of tomcat with a single click. If you ONLY intend to ever run one tomcat at a time this will work pretty well.

It uses instead of, and sets the CATALINA_PID variable to write a file containing the pid of the launched tomcat.

It first checks to see if there is a pid file at the location set by CATALINA_PID. If there is, the script reads the file and, making the assumption that you want to stop the running tomcat, calls ' stop', waits a few seconds and checks for the pid file again. If the file still exists it runs kill -9 on the pid, hopefully *really* stopping tomcat.

If there is no pid file we assume that tomcat is not running, and run the launch command. In my case I set the java version first, then provide a path for the PID variable, then run ' start'

More details in the code comments.

UPDATED FOR MAC OS: added OS X-specific alerts.

Script attached


Permalink 09:44:40 am, by Greg, 90 words, 224 views   English (CA)
Categories: R & D; Mins. worked: 0

CLI tricks - sudo permission denied

Bash commands using sudo AND redirect can fail on the redirect because sudo permissions are not passed on to the next stage in your command.

For example, the following failed for me with a permission denied error:
sudo echo "something important" >> /etc/apt//mirror.list

It failed because the bit after the final double-quote is a redirect apparently, and not part of the original echo command. The solution is to wrap the whole thing up thusly:
sudo bash -c "echo \"something important\" >> /etc/apt//mirror.list"


Permalink 04:08:53 pm, by Greg, 32 words, 90 views   English (CA)
Categories: Servers; Mins. worked: 90

Repo tweak

Wrote a little script that makes it easier to add a repo to a live apt-mirror and added it to the apt machine admin user's home directory - it's called


Permalink 01:04:35 pm, by Greg, 74 words, 103 views   English (CA)
Categories: R & D; Mins. worked: 0

Covert symlinks to files

Had a directory with millions of symlinks. I needed to move the directory without the dependencies imposed by the symlinks. What I wanted was to 'convert' the symlinks to actual files - that is, replace the symlink with a copy of its target file. Found this, which worked a treat. Here's the actual code:

for file in *;
link=$(readlink "${file}");
if [ "${link}" ]
rm "${file}";
cp -v "${link}" "${file}";


Permalink 10:17:10 am, by Greg, 23 words, 500 views   English (CA)
Categories: Documentation, Announcements; Mins. worked: 0

CLI tricks - recently changed files

Produce a sorted list of recently changed files by running this:
find . -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort


Permalink 11:02:11 am, by mholmes, 71 words, 117 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 90

Building out new computer

My new box has arrived, and I've started building it up with the next LTS version of Ubuntu (not actually due for release till April). I've installed with hcmc admin logon only, added gnome-shell, and started adding applications. I'll keep hacking at this till 12.04 seems pretty stable, and then switch from my old machine by moving HDs to the new one. In the meantime, there's a lot of testing to do.


Permalink 10:41:04 am, by mholmes, 109 words, 188 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 60

Testing of eXist war builds

Working with RVDB in Belgium, I've been helping to investigate problems building and running the eXist WAR distro (trunk) on Windows. It appears to be the case that the WAR distro does not run correctly on Windows unless an additional stax jar file is added (this jar was recently removed from trunk, as it was thought to be unnecessary). We have now clarified and confirmed the symptoms and solution, and RVDB wil make a detailed bug report. We don't run eXist under Windows ourselves, but 70% of eXist downloads came from Windows users over the last few years, so it's important that the upcoming new version work well on Windows.


Permalink 10:18:57 am, by Greg, 294 words, 158 views   English (CA)
Categories: Documentation; Mins. worked: 5

CLI tricks - who's logged in to this machine?

Not as easy as you'd think. There are a couple of easy ways to find out who's logged in to a linux machine: 'users' and 'who'. But I've discovered that when a user has been authenticated via ldap they don't show up when 'users' and 'who' are called. This MAY be a result of my configuration of the machine, but until that is determined, the only I've found of doing this is a bit convoluted. Run this one-liner when logged in to a remote machine:

me=`whoami`;today=`date +"%a %b %d"`;last|sort -r|grep "$today"|grep -v "$me"|grep -v reboot|cut -d " " -f1

me=`whoami` -> we want to able to exclude ourselves from the output.
today=`date +"%a %b %d"` -> we need the current date, without the time
last -> the actual command we need to find users logged in with ldap
sort -r -> last outputs oldest last, this reverses the order
grep $today -> cleans up our output to only show stuff from today
grep -v "$me" -> exclude me from output
grep -v reboot -> exclude the last reboot command
cut -d " " -f1 -> get the first field of the output (this is the login name of the ldap user)

All of that and all you *really* get is the last logged in user. They may or may not actually be logged in right now.
I'll have to look in to my ldap config a bit more closely. There must be a way of including ldap auth'd users in the output from 'users' or 'who'.

NOTE: a simpler, possibly just-as-useful, method is to run this

last | grep 'still logged in' | grep 'tty' | cut -d " " -f 1

which returns the logged user running on a local display.


Permalink 10:24:46 am, by mholmes, 409 words, 2050 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation; Mins. worked: 60

UTF-8 on Tomcat (and proxied through Apache)

This is a collection of bits and pieces we have painfully learned over the years regarding the setup of Tomcat proxied through Apache, enabling the reliable use of UTF-8 encoding in form submissions etc.

To make sure Tomcat starts with UTF-8 instead of the default, I put a script in [tomcat]/bin called

./ dFile.encoding="UTF-8"

If another script is starting Tomcat, that script must ensure that the process is launched with the dFile.encoding="UTF-8" flag on the command line.

Add the correct parameter (URIEncoding="UTF-8") in conf/server.xml:

    <Connector port="8080" protocol="HTTP/1.1" 
               redirectPort="8443" URIEncoding="UTF-8" />

If Tomcat is proxied through Apache using the AJP connector, Tomcat's conf/server.xml file may need to be tweaked to add a URIEncoding="UTF-8" parameter for that:

    <!-- Define an AJP 1.3 Connector on port 8019 -->
    <Connector port="8019" protocol="AJP/1.3" redirectPort="8081" />

changed to:

    <!-- Define an AJP 1.3 Connector on port 8019 -->
    <Connector port="8019" protocol="AJP/1.3" redirectPort="8081" URIEncoding="UTF-8" />

This needs to be added to the Apache configuration:

JkOptions +ForwardURICompatUnparsed
This does not relate specifically to UTF-8 encoding, but it's collected here for completeness:

Tomcat 7 handles default "welcome files" differently from 6. If you look in Tomcat's conf/web.xml, you'll see this:


Obviously, there's no handler for an empty directory root; Tomcat 7, when presented with a directory root, goes to this list to see what to pass to the webapp. Absent a matching item, Tomcat 6 would simply pass the directory root through to the webapp, but Tomcat 7 doesn't; it always chooses the first item and passes that through. If your application isn't expecting it, the result will be an error. Assuming your application (Cocoon or eXist) is set up to handle everything it might get, then the best option is to comment out all the <welcome-file> elements; if this is done, then Tomcat seems to hand off all processing to the webapp.

Incidentally, you have to restart Tomcat between changes to web.xml to cause them to have an effect (according to the web).

Other configuration changes may need to be made in eXist and/or Cocoon (if used), and these are detailed elsewhere on the blog. These changes relate specifically to Tomcat.


Permalink 03:06:46 pm, by mholmes, 28 words, 192 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation, Documentation; Mins. worked: 15

.htaccess: turning on directory indexing

To turn on directory indexing, add this to your .htaccess file:

Options +Indexes 

The additional setting for fancy indexing doesn't seem to work on our servers:

IndexOptions +FancyIndexing 


Permalink 02:25:29 pm, by mholmes, 35 words, 175 views   English (CA)
Categories: Servers, R & D, Activity log; Mins. worked: 45

Firefighting on htaccess files

Authentication through LDAP stopped working for all our PHP sites, so after RE figured out the problem (pointing at the wrong LDAP server in Mustard) we had to make changes to all the .htaccess files.


Permalink 05:36:01 pm, by mholmes, 55 words, 154 views   English (CA)
Categories: Servers, Activity log; Mins. worked: 30

Worked around bug on teiJenkins

I encountered this bug on teiJenkins, and after reading around a bit, followed the advice there to work around it:

sudo mv /etc/init/ureadahead.conf /etc/init/ureadahead.conf.disable
sudo shutdown -r now

It seemed to solve the problem. We should do df -h every now and then to check for a recurrence.


Permalink 11:15:52 am, by Greg, 210 words, 162 views   English (CA)
Categories: Servers; Mins. worked: 0

More permgenspace problems

We've had two permgenspace problems in the last two days on pear. A bit of research turned up a stack overflow answer that suggests the use of -XX:+CMSClassUnloadingEnabled in the java options.

I also found this (old) page which outlines things quite succinctly. To sum up:
-Xms and -Xmx should be set to the same number because "making these settings the same value prevents the JVM from frequently re-sizing the heap as it grows, something that can trigger frequent garbage collections and can degrade performance."
Only adjust permgensize if your other settings still produce permgenspace errors.

Another, more recent page goes in to more detail. It too argues that "it is recommended that the initial and maximum values for heap size be set to the same value. This is often referred to as a fully committed heap and this will instruct the JVM to create a heap that is initially at its maximum size and prevent several full garbage collections from occurring as the heap expands to its maximum size."

When we get the old-db-server-cum-new-tomcat-server ready we'll try this setup out:
Oracle 1.6 JVM, latest Tomcat (right now that's 7.0.25) and use the recommendations from We then plan to stress test the new setup with jMeter (


Permalink 09:27:19 am, by Greg, 50 words, 154 views   English (CA)
Categories: Documentation; Mins. worked: 0

Chard re-purposed

Chard is being re-purposed as a tomcat-legacy/tomcat-bleeding-edge machine called
Sysadmin is removing the IP name chard, so anyone using chard in a connection string will suddenly have failures - but of course no-one is using chard in a connection string because they aren't supposed to...


Permalink 07:51:11 pm, by mholmes, 79 words, 254 views   English (CA)
Categories: R & D, Activity log, Documentation; Mins. worked: 20

Setting up Exchange Calendar in Thunderbird

I've had to figure this out three times now, so I'm blogging it.

  • Install the Lightning add-on for Thunderbird.
  • Install the Provider for Microsoft Exchange add-on.
  • Click on the Calendar view, and right-click under Home. Choose New Calendar.
  • Select "On the network", then Next, and choose Microsoft Exchange.
  • This is the unintuitive bit: in Location, put this:
    Replace "netlink" with your netlink id.
  • When asked to log in, use "UVIC\netlink" as your login id.


Permalink 04:46:54 pm, by mholmes, 46 words, 136 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation; Mins. worked: 20

RSync - reminder to self

Running across this reminded me that I've been meaning to set up ssh keys for rsync to Rutabaga for ages now, and not got around to it, so I've just done it, and built a single backup script for all current projects which doesn't require passwords.


Permalink 10:09:34 am, by Greg, 202 words, 229 views   English (CA)
Categories: Activity log; Mins. worked: 180

Update eXist DB script

Further to Martin's post...

Using the included java admin client can be clunky for frequent use, so I've written a bash script that uploads the most recently modified files to the database. It's been tested on Ubuntu 11.10 and Mac OS X (10.6 and 10.7).

To use it, provide values for the variables at the top of the script and set the script to be executable. It is only designed to work in a very specific context (one local directory, one remote db, etc.) so you'll need a copy for each project. It is also utterly dependent upon accessing a script inside the eXist DB that provides the timestamp of the most recently modified file in the DB. Martin wrote one and posted about it here:

Extra info
In Gnome-Shell I installed an extension called Quick Launch that allows me to add functionality similar to the old Gnome Drawers applet, providing me with a fast, gui-based method of launching the updater from the Gnome panel.

I've added a quicklaunch for the admin client (sometimes you need it) using javaws. The essential command is:


Works a treat.


This blog is the location for all work involving software and hardware maintenance, updates, installs, etc., both routine and urgent, in the server room, the labs and the R&D rooms.



 << Current>>
Jan Feb Mar Apr
May Jun Jul Aug
Sep Oct Nov Dec

XML Feeds