Maintenance

After an update to DSM 4.2 rutabaga no longer allowed rsync backups, failing with:

sh: rsync: not found
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: remote command not found (code 127) at io.c(605) [Receiver=3.0.9]

After much wailing and gnashing of teeth we discovered that non-interactive users do not have /usr/syno/bin in their path (it *is* in their path if they shell in to the NAS, so they can run rsync *from* the NAS when shell'd in).

So, that's an easy fix, says us: add a symlink to /usr/syno/bin/rsync in a logical spot that *is* in a non-interactive path, like /usr/bin.

Problem: admin user cannot su root (error message = su: must be suid to work properly), so cannot create symlink.

Answer: TURN ON TELNET AND LOG IN AS ROOT USING THE WORST POSSIBLE METHOD!!! Then, you make the symlink and turn off telnet - quick!

Tomcat has now been configured so that a slash not followed by a filename is handled by the webapp, not trapped as an error by Tomcat itself. I've also uploaded and tested the new Moses app on Peach, confirming that while the eXist dashboard and eXide fail under Tomcat 6, they work under Tomcat 7.

After a system update, rsync stopped working over ssh on Rutabaga. We eventually discovered that users rsyncing over ssh require the ssh service to be turned on (Control panel/Terminal/SSH service). This was apparently not necessary in previous builds, or perhaps it was turned off by the update for some reason.

teijenkins found itself unable to complete an update because it was out of disk space. This was initially surprising, but I discovered that it was actually the small boot partition that was out of space, because of the accumulation of old kernels. This command:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

cribbed from this page cleaned out the old kernels and solved the problem.

As Martin notes, we met with RE about the near future of several servers.
Long-and-short:
a) The new machine, aka Peach, will replace Pear and Grape as soon as it's configured. RE is setting it up with a single Tomcat install, but we're going to try running multiple instances of it instead of having an installed codebase for each instance like we've had in the past. Once we're ready to test it, we'll get Grape's webapps and Pear's webapps on to it and test it for a week or so. When we're happy that things are working we'll bring down Grape and Pear.
b) when Grape comes down, it will be rebuilt as an additional node in our webcluster - this should happen by Christmas.
c) when Pear comes down it will become a BCP (bulk-copy) machine for Peach and Mango (the DB server), providing us with a rapid-recovery scheme. This build should be done by the end of February.
d) Once the CFI machine is purchased it will be replacing arugula as the NFS/disk box, and arugula will be added to the cluster. Timeline ASAP.
e) Lettuce will wait until we have a chance to put time in to the MoEML map refresh. We expect to begin in the new year. When Lettuce is no longer required for MoEML we'll add it to the cluster.
f) RE will look in to Cress and let us know what he thinks we should do with it.

Got a call from sysadmin this morning about the ise machine filling its fs with ehcache.data. It has been addressed in cocoon.xconf (see here) but we aren't sure how long it's been since last dealing with the problem.
So, sysadmin has noted it in the machine's config log(?) and I'm noting it here that tomcat was restarted this morning in order to remove the ehcache.data file, which had filled the filesysytem (the file was about 4GB).

GN will blog the outcomes. Posting time.

Due to my misunderstanding how the licensing works for fennel's ESXi server, our license lapsed the other day and we were unable to restart a VM.

For future reference, we now have to pay a fee for academic licenses for VMware products. We just got a 3 year subscription for ~$800 after taxes. What this provides is a 3 year lifespan on our "web store". Any approved HCMC user (right now this is only me) can go to the web store and download a 'free' copy of most VMware offerings. The ESXi server license I just downloaded says there is a 1 year term on the license (it looks like it's a full calendar year plus, it expires on Dec. 31/2013), so we'll need to head back to the store next December and get a whole new license - it doesn't get renewed.

Our WebStore URL: http://e5.onthehub.com/WebStore/Welcome.aspx?vsro=8&ws=2f6647eb-e7a0-df11-ad57-0030487d8897

GN and I repurposed Plum to replace the temporarily-suspended teijenkins machine, while licensing issues with VMWare are sorted out. This functioned as a fresh test of the Jenkins build script, which went great with two little hiccups (TEI packages have changed, and zip is no longer installed on Ubuntu Server by default). Those changes have been added to the build script in SVN. The new machine is running (but very slow to build, due to its Arm processor).

The apt server was running low on disk space, so I learned how to extend logical volumes today. This is what I did:

1) Find out how much total space you've got on (for e.g.) disk sdb:
sudo hdparm -I /dev/sdb | grep GB

2) Extend the logical volume by adding 20GB to the volume itself (not the disk)
sudo lvextend -L+20G /dev/volumegroupname/volumename

3) You also need to extend the filesystem, and you can't do it while the volume is mounted:
sudo umount /dev/volumegroupname/volumename

4) You need to run fsck before resizing a filesystem:
sudo e2fsck -f /dev/mapper/vgpool-lvrepo

5) Now you can resize the filesystem:
sudo resize2fs /dev/volumegroupname/volumename

6) And remount:
sudo mount /dev/volumegroupname/volumename /mountlocation

Done.