Tomcat has now been configured so that a slash not followed by a filename is handled by the webapp, not trapped as an error by Tomcat itself. I've also uploaded and tested the new Moses app on Peach, confirming that while the eXist dashboard and eXide fail under Tomcat 6, they work under Tomcat 7.
Category: "Activity log"
After a system update, rsync stopped working over ssh on Rutabaga. We eventually discovered that users rsyncing over ssh require the ssh service to be turned on (Control panel/Terminal/SSH service). This was apparently not necessary in previous builds, or perhaps it was turned off by the update for some reason.
teijenkins found itself unable to complete an update because it was out of disk space. This was initially surprising, but I discovered that it was actually the small boot partition that was out of space, because of the accumulation of old kernels. This command:
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
cribbed from this page cleaned out the old kernels and solved the problem.
As Martin notes, we met with RE about the near future of several servers.
a) The new machine, aka Peach, will replace Pear and Grape as soon as it's configured. RE is setting it up with a single Tomcat install, but we're going to try running multiple instances of it instead of having an installed codebase for each instance like we've had in the past. Once we're ready to test it, we'll get Grape's webapps and Pear's webapps on to it and test it for a week or so. When we're happy that things are working we'll bring down Grape and Pear.
b) when Grape comes down, it will be rebuilt as an additional node in our webcluster - this should happen by Christmas.
c) when Pear comes down it will become a BCP (bulk-copy) machine for Peach and Mango (the DB server), providing us with a rapid-recovery scheme. This build should be done by the end of February.
d) Once the CFI machine is purchased it will be replacing arugula as the NFS/disk box, and arugula will be added to the cluster. Timeline ASAP.
e) Lettuce will wait until we have a chance to put time in to the MoEML map refresh. We expect to begin in the new year. When Lettuce is no longer required for MoEML we'll add it to the cluster.
f) RE will look in to Cress and let us know what he thinks we should do with it.
Got a call from sysadmin this morning about the ise machine filling its fs with ehcache.data. It has been addressed in cocoon.xconf (see here) but we aren't sure how long it's been since last dealing with the problem.
So, sysadmin has noted it in the machine's config log(?) and I'm noting it here that tomcat was restarted this morning in order to remove the ehcache.data file, which had filled the filesysytem (the file was about 4GB).
GN will blog the outcomes. Posting time.
Due to my misunderstanding how the licensing works for fennel's ESXi server, our license lapsed the other day and we were unable to restart a VM.
For future reference, we now have to pay a fee for academic licenses for VMware products. We just got a 3 year subscription for ~$800 after taxes. What this provides is a 3 year lifespan on our "web store". Any approved HCMC user (right now this is only me) can go to the web store and download a 'free' copy of most VMware offerings. The ESXi server license I just downloaded says there is a 1 year term on the license (it looks like it's a full calendar year plus, it expires on Dec. 31/2013), so we'll need to head back to the store next December and get a whole new license - it doesn't get renewed.
Oct 18, 2013: I applied for, and received a new activation code for ESXi server and sent in a request to sysadmin to apply it.
GN and I repurposed Plum to replace the temporarily-suspended teijenkins machine, while licensing issues with VMWare are sorted out. This functioned as a fresh test of the Jenkins build script, which went great with two little hiccups (TEI packages have changed, and zip is no longer installed on Ubuntu Server by default). Those changes have been added to the build script in SVN. The new machine is running (but very slow to build, due to its Arm processor).
If you need to add a new drive to (and extend) the LVM setup:
1) Make sure the drive is free of partitions. Install the drive and power up.
2) Find out what the machine is calling the new drive. This will be something in the sd[a-z] range, and is not predictable - you'll need to check before moving on. For the purposes of this post, we'll say that it's called 'sdc'.
3) Create a physical volume on the new disk:
sudo pvcreate /dev/sdc
--- if you only need to extend an existing volume you can start here ---
4) We need to know what the existing volume-groupe and volume names are:
should return something that starts with 'VG Name vgpool', where 'vgpool' is the string we need.
5) To extend the volume group to include our new device:
sudo vgextend vgpool /dev/sdc1
6) Extend the logical volume by adding 20GB to the volume - note that we can access the volume at /dev/mapper/<volume-group-name>/<volume-name>
sudo lvextend -L+20G /dev/mapper/vgpool-lvrepo
** if you don't know how much disk you have left on any physical device, run this:
sudo hdparm -I /dev/sdc | grep GB
7) You also need to extend the filesystem, and you can't do it while the volume is mounted:
sudo umount /dev/mapper/vgpool-lvrepo
8) You need to run fsck before resizing a filesystem:
sudo e2fsck -f /dev/mapper/vgpool-lvrepo
9) Now you can resize the filesystem:
sudo resize2fs /dev/mapper/vgpool-lvrepo
6) And remount:
sudo mount /dev/mapper/vgpool-lvrepo /mountlocation
This is my documentation of recent changes to the way the TEI Jenkins Continuous Integration Server VMs are configured.
The new VM whose machine name is teijenkins1204 is now running Jenkins on port 8080, and also Apache proxying Jenkins on port 80. JA (sysadmin) has pointed the DNS entry for teijenkins.hcmc.uvic.ca at this machine, so the Jenkins instance is accessible on port 80 and 8080 there. The dual access allows for normal access to Jenkins on port 80 (convenient), but also for a situation in which we might want to put up a temporary warning page on port 80 while solving a problem with Jenkins accessed through 8080, then restore the proxy setup. I followed these instructions to set up proxying.
That machine is also running denyhosts.
The old machine hostname has been changed to teijenkinsdev, and JA has pointed teijenkinsdev.hcmc.uvic.ca at it. The Jenkins service has been disabled on this machine using sysv-rc-conf. When I ran that tool, it showed X characters for the jenkins service on run levels 2, 3, 4 and 5. I removed all of them; to re-enable the service, run it again and replace them. In the meantime, you can run Jenkins manually using
sudo /etc/init.d/jenkins start. The plan is to reprovision that machine with more RAM and disk space, and then build it up with Ubuntu 12.10 to test and tweak the build script; doing this for every Ubuntu release will make it less messy to migrate to the next LTS when it comes out.