Overnight I updated teijenkins.hcmc.uvic.ca from 14.04 to 14.04.01. When it came back up, after re-enabling commented-out third-party repositories in /etc/sources.list.d (tei and jenkins), I discovered that the proxying of Jenkins on port 80 was no longer working. After some investigation, I discovered that the original sites-available/jenkins configuration file was not being recognized by apache -- running sudo a2ensite jenkins resulted in "jenkins site does not exist" -- because now it needs to be called jenkins.conf. Once I'd renamed it I was able to enable it, but it took a reboot to make it work; just restarting apache wasn't enough.
My Jenkins box was showing me an error message about proxying being set up wrongly in its Management page. It turns out recent changes to Jenkins have either triggered this problem, or triggered the error message because of better monitoring (I couldn't actually figure out which). At the same time, the Stylesheets job, which checks out code from GitHub, was failing to trigger, or possibly failing to correctly poll the GitHub repo (again, I couldn't quite figure out which, although I think it was failing to poll), so clearly something was wrong. I consulted this page, and modified the /etc/apache2/sites_available/jenkins file to add "nocanon" to the first line shown below, and to add the second and third lines. After a reboot, Jenkins came up without the error message in the Management page, and the Stylesheets job instantly polled the repo, updated, and started building.
ProxyPass / http://localhost:8080/ nocanon ProxyPassReverse / http://localhost:8080 AllowEncodedSlashes NoDecode
In the next few days I'll begin populating a 14.04 repository on the apt machine.
We were running short of disk space, so I added another old drive and added it to the LVM setup. Most of the 'how-to' is identical to my post on extending a logical volume, but I've rewritten the post to reflect the additional steps required to add a new disk.
The post is here: http://hcmc.uvic.ca/blogs/index.php?blog=11&p=10175
Last night we upgraded Tomcat on peach to the latest version in order to take advantage of some recent GC optimizations. Unfortunately it also included a security fix that broke all of our Cocoon apps. So, bcgenesis, mariage, ach, myndir, etc. were dead.
Sysadmin fixed some of them with an adjustment to the web.xml file inside of each app like this:
Around line 708 in <app-name>/WEB-INF/web.xml, look for
and change webdav/* to something else, like webdavservlet/*
After an update to DSM 4.2 rutabaga no longer allowed rsync backups, failing with:
sh: rsync: not found
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: remote command not found (code 127) at io.c(605) [Receiver=3.0.9]
After much wailing and gnashing of teeth we discovered that non-interactive users do not have /usr/syno/bin in their path (it *is* in their path if they shell in to the NAS, so they can run rsync *from* the NAS when shell'd in).
So, that's an easy fix, says us: add a symlink to /usr/syno/bin/rsync in a logical spot that *is* in a non-interactive path, like /usr/bin.
Problem: admin user cannot su root (error message = su: must be suid to work properly), so cannot create symlink.
Answer: TURN ON TELNET AND LOG IN AS ROOT USING THE WORST POSSIBLE METHOD!!! Then, you make the symlink and turn off telnet - quick!
After a system update, rsync stopped working over ssh on Rutabaga. We eventually discovered that users rsyncing over ssh require the ssh service to be turned on (Control panel/Terminal/SSH service). This was apparently not necessary in previous builds, or perhaps it was turned off by the update for some reason.
teijenkins found itself unable to complete an update because it was out of disk space. This was initially surprising, but I discovered that it was actually the small boot partition that was out of space, because of the accumulation of old kernels. This command:
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
cribbed from this page cleaned out the old kernels and solved the problem.
GN and I repurposed Plum to replace the temporarily-suspended teijenkins machine, while licensing issues with VMWare are sorted out. This functioned as a fresh test of the Jenkins build script, which went great with two little hiccups (TEI packages have changed, and zip is no longer installed on Ubuntu Server by default). Those changes have been added to the build script in SVN. The new machine is running (but very slow to build, due to its Arm processor).
If you need to add a new drive to (and extend) the LVM setup:
1) Make sure the drive is free of partitions. Install the drive and power up.
2) Find out what the machine is calling the new drive. This will be something in the sd[a-z] range, and is not predictable - you'll need to check before moving on. For the purposes of this post, we'll say that it's called 'sdc'.
3) Create a physical volume on the new disk:
sudo pvcreate /dev/sdc
--- if you only need to extend an existing volume you can start here ---
4) We need to know what the existing volume-groupe and volume names are:
should return something that starts with 'VG Name vgpool', where 'vgpool' is the string we need.
5) To extend the volume group to include our new device:
sudo vgextend vgpool /dev/sdc1
6) Extend the logical volume by adding 20GB to the volume - note that we can access the volume at /dev/mapper/<volume-group-name>/<volume-name>
sudo lvextend -L+20G /dev/mapper/vgpool-lvrepo
** if you don't know how much disk you have left on any physical device, run this:
sudo hdparm -I /dev/sdc | grep GB
7) You also need to extend the filesystem, and you can't do it while the volume is mounted:
sudo umount /dev/mapper/vgpool-lvrepo
8) You need to run fsck before resizing a filesystem:
sudo e2fsck -f /dev/mapper/vgpool-lvrepo
9) Now you can resize the filesystem:
sudo resize2fs /dev/mapper/vgpool-lvrepo
6) And remount:
sudo mount /dev/mapper/vgpool-lvrepo /mountlocation
EDIT: These instructions have been superceded. See this.
The Tomcat setup has recently changed with the move of stable apps to Grape, so this is an updated set of instructions. MDH and GN have rights to run the tomcat script; SA probably does too.
- ssh email@example.com
- cd /etc/init.d
- sudo ./tomcat stop
At this point, tomcat will seem to have stopped, but it probably hasn't. Confirm by doing this:
- ps aux | grep tomcat
If you see both tomcat running, you'll have to kill -9 tomcat. Since the process runs as hcmc, and you can't sudo-kill it, do this:
- su hcmc
- kill -9 [the process number]
Do ps aux again to check that the process is dead (you should not see tomcat running now). Assuming that worked, then
- exit (to return to mholmes)
- sudo ./tomcat start
Wait a couple of minutes, then check that tomcat is back up and that projects are working.