Maintenance

After an update to DSM 4.2 rutabaga no longer allowed rsync backups, failing with:

sh: rsync: not found
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: remote command not found (code 127) at io.c(605) [Receiver=3.0.9]

After much wailing and gnashing of teeth we discovered that non-interactive users do not have /usr/syno/bin in their path (it *is* in their path if they shell in to the NAS, so they can run rsync *from* the NAS when shell'd in).

So, that's an easy fix, says us: add a symlink to /usr/syno/bin/rsync in a logical spot that *is* in a non-interactive path, like /usr/bin.

Problem: admin user cannot su root (error message = su: must be suid to work properly), so cannot create symlink.

Answer: TURN ON TELNET AND LOG IN AS ROOT USING THE WORST POSSIBLE METHOD!!! Then, you make the symlink and turn off telnet - quick!

After a system update, rsync stopped working over ssh on Rutabaga. We eventually discovered that users rsyncing over ssh require the ssh service to be turned on (Control panel/Terminal/SSH service). This was apparently not necessary in previous builds, or perhaps it was turned off by the update for some reason.

teijenkins found itself unable to complete an update because it was out of disk space. This was initially surprising, but I discovered that it was actually the small boot partition that was out of space, because of the accumulation of old kernels. This command:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

cribbed from this page cleaned out the old kernels and solved the problem.

GN and I repurposed Plum to replace the temporarily-suspended teijenkins machine, while licensing issues with VMWare are sorted out. This functioned as a fresh test of the Jenkins build script, which went great with two little hiccups (TEI packages have changed, and zip is no longer installed on Ubuntu Server by default). Those changes have been added to the build script in SVN. The new machine is running (but very slow to build, due to its Arm processor).

The apt server was running low on disk space, so I learned how to extend logical volumes today. This is what I did:

1) Find out how much total space you've got on (for e.g.) disk sdb:
sudo hdparm -I /dev/sdb | grep GB

2) Extend the logical volume by adding 20GB to the volume itself (not the disk)
sudo lvextend -L+20G /dev/volumegroupname/volumename

3) You also need to extend the filesystem, and you can't do it while the volume is mounted:
sudo umount /dev/volumegroupname/volumename

4) You need to run fsck before resizing a filesystem:
sudo e2fsck -f /dev/mapper/vgpool-lvrepo

5) Now you can resize the filesystem:
sudo resize2fs /dev/volumegroupname/volumename

6) And remount:
sudo mount /dev/volumegroupname/volumename /mountlocation

Done.

The Tomcat setup has recently changed with the move of stable apps to Grape, so this is an updated set of instructions. MDH and GN have rights to run the tomcat script; SA probably does too.

• ssh mholmes@pear.hcmc.uvic.ca
• cd /etc/init.d
• sudo ./tomcat stop

At this point, tomcat will seem to have stopped, but it probably hasn't. Confirm by doing this:

• ps aux | grep tomcat

If you see both tomcat running, you'll have to kill -9 tomcat. Since the process runs as hcmc, and you can't sudo-kill it, do this:

• su hcmc
• kill -9 [the process number]

Do ps aux again to check that the process is dead (you should not see tomcat running now). Assuming that worked, then

• exit (to return to mholmes)
• sudo ./tomcat start

Wait a couple of minutes, then check that tomcat is back up and that projects are working.

This is my documentation of recent changes to the way the TEI Jenkins Continuous Integration Server VMs are configured.

The new VM whose machine name is teijenkins1204 is now running Jenkins on port 8080, and also Apache proxying Jenkins on port 80. JA (sysadmin) has pointed the DNS entry for teijenkins.hcmc.uvic.ca at this machine, so the Jenkins instance is accessible on port 80 and 8080 there. The dual access allows for normal access to Jenkins on port 80 (convenient), but also for a situation in which we might want to put up a temporary warning page on port 80 while solving a problem with Jenkins accessed through 8080, then restore the proxy setup. I followed these instructions to set up proxying.

That machine is also running denyhosts.

The old machine hostname has been changed to teijenkinsdev, and JA has pointed teijenkinsdev.hcmc.uvic.ca at it. The Jenkins service has been disabled on this machine using sysv-rc-conf. When I ran that tool, it showed X characters for the jenkins service on run levels 2, 3, 4 and 5. I removed all of them; to re-enable the service, run it again and replace them. In the meantime, you can run Jenkins manually using sudo /etc/init.d/jenkins start. The plan is to reprovision that machine with more RAM and disk space, and then build it up with Ubuntu 12.10 to test and tweak the build script; doing this for every Ubuntu release will make it less messy to migrate to the next LTS when it comes out.

This morning I was contacted by two different departments to report that the Agenda application used to schedule classes is not working. On investigation, I discovered that the login page is truncated immediately at the point where the first call to the database is made. After reading through the code, I couldn't see anything wrong, so I suspected a connection problem such as an ACL issue between web.uvic.ca/lang and csmgenr2 (the mysql server). The db is there, functioning, and accessible through PhpMyAdmin.

I raised a ticket with sysadmin, and MC got back to me to say that there doesn't seem to be a connection issue, but the error in the logs was with a PHP include:

2012-08-29T08:43:23-07:00 local@axolotl.comp.uvic.ca user.notice php_cgi: PHP
Warning: main() [<a href='function.include'>function.include</a>]: Failed
opening 'DB.php' for inclusion
(include_path='.:/usr/local/php-4.4.8/include/php:/usr/local/php-4.4.8/lib/php')
in
/home3/80/lang/www/agenda/Application_Files/script_files/php/database_connection.php
on line 11

The error comes when the db connection script tries to include DB.php, which is the PEAR library this project uses for accessing MySql. This sent me to check the PHP settings info in phpinfo.php, and I found a number of oddities:

• The PHP version is 4.4.9,
• the configure command output includes this: './configure' '--prefix=/usr/local/php-4.4.9', and
• the configuration file is here: /usr/local/php-4.4.9/lib/php.ini.
• HOWEVER, the extension_dir is /usr/local/php-4.4.8/lib/php/extension, and
• the include_path is .:/usr/local/php-4.4.8/include/php:/usr/local/php-4.4.8/lib/php.

My current theory is that when PHP was updated from 4.4.8 to 4.4.9, those key settings in the master php.ini file should have been changed, but weren't; and as a result, the include of DB.php is failing. I've reported this back to sysadmin, and I'm waiting for a response. In the meantime, I've tried overriding the include_path setting in a local php.ini, but I can't get that to work; perhaps those settings can't be overridden.

In the meantime, I could no longer bear to look at the logo which proclaimed "Agenda / Organize. Mangage. Simplfy" (sic, seriously), so I've fixed that.

Just blogging something I keep having to look up. This is how to switch your SVN repo from the old tapor URL to hcmc:

svn switch --relocate https://revision.tapor.uvic.ca/svn/[reponame] https://revision.hcmc.uvic.ca/svn/[reponame] 

Sysadmin fixed the ACLs (there were two sets of entries, apparently, which confused the process for a while), and now the machine is up and running with Jenkins on port 8080. I tried moving it to port 80, but was prevented by the Ubuntu rule which won't allow a service not running as root to run on a port below 1024 (and we definitely don't want Jinks running as root). So I may end up running Apache just so I can proxy it to port 80. Seems like overkill, but there doesn't seem to be an alternative solution.

I'm going to run the two machines side by side until I come back from vacation, so we can make sure the new one is stable before we switch to it and bring down the old one.