Maintenance

Due to my misunderstanding how the licensing works for fennel's ESXi server, our license lapsed the other day and we were unable to restart a VM.

For future reference, we now have to pay a fee for academic licenses for VMware products. We just got a 3 year subscription for ~$800 after taxes. What this provides is a 3 year lifespan on our "web store". Any approved HCMC user (right now this is only me) can go to the web store and download a 'free' copy of most VMware offerings. The ESXi server license I just downloaded says there is a 1 year term on the license (it looks like it's a full calendar year plus, it expires on Dec. 31/2013), so we'll need to head back to the store next December and get a whole new license - it doesn't get renewed.

Our WebStore URL: http://e5.onthehub.com/WebStore/Welcome.aspx?vsro=8&ws=2f6647eb-e7a0-df11-ad57-0030487d8897

GN and I repurposed Plum to replace the temporarily-suspended teijenkins machine, while licensing issues with VMWare are sorted out. This functioned as a fresh test of the Jenkins build script, which went great with two little hiccups (TEI packages have changed, and zip is no longer installed on Ubuntu Server by default). Those changes have been added to the build script in SVN. The new machine is running (but very slow to build, due to its Arm processor).

The apt server was running low on disk space, so I learned how to extend logical volumes today. This is what I did:

1) Find out how much total space you've got on (for e.g.) disk sdb:
sudo hdparm -I /dev/sdb | grep GB

2) Extend the logical volume by adding 20GB to the volume itself (not the disk)
sudo lvextend -L+20G /dev/volumegroupname/volumename

3) You also need to extend the filesystem, and you can't do it while the volume is mounted:
sudo umount /dev/volumegroupname/volumename

4) You need to run fsck before resizing a filesystem:
sudo e2fsck -f /dev/mapper/vgpool-lvrepo

5) Now you can resize the filesystem:
sudo resize2fs /dev/volumegroupname/volumename

6) And remount:
sudo mount /dev/volumegroupname/volumename /mountlocation

Done.

The Tomcat setup has recently changed with the move of stable apps to Grape, so this is an updated set of instructions. MDH and GN have rights to run the tomcat script; SA probably does too.

• ssh mholmes@pear.hcmc.uvic.ca
• cd /etc/init.d
• sudo ./tomcat stop

At this point, tomcat will seem to have stopped, but it probably hasn't. Confirm by doing this:

• ps aux | grep tomcat

If you see both tomcat running, you'll have to kill -9 tomcat. Since the process runs as hcmc, and you can't sudo-kill it, do this:

• su hcmc
• kill -9 [the process number]

Do ps aux again to check that the process is dead (you should not see tomcat running now). Assuming that worked, then

• exit (to return to mholmes)
• sudo ./tomcat start

Wait a couple of minutes, then check that tomcat is back up and that projects are working.

This is my documentation of recent changes to the way the TEI Jenkins Continuous Integration Server VMs are configured.

The new VM whose machine name is teijenkins1204 is now running Jenkins on port 8080, and also Apache proxying Jenkins on port 80. JA (sysadmin) has pointed the DNS entry for teijenkins.hcmc.uvic.ca at this machine, so the Jenkins instance is accessible on port 80 and 8080 there. The dual access allows for normal access to Jenkins on port 80 (convenient), but also for a situation in which we might want to put up a temporary warning page on port 80 while solving a problem with Jenkins accessed through 8080, then restore the proxy setup. I followed these instructions to set up proxying.

That machine is also running denyhosts.

The old machine hostname has been changed to teijenkinsdev, and JA has pointed teijenkinsdev.hcmc.uvic.ca at it. The Jenkins service has been disabled on this machine using sysv-rc-conf. When I ran that tool, it showed X characters for the jenkins service on run levels 2, 3, 4 and 5. I removed all of them; to re-enable the service, run it again and replace them. In the meantime, you can run Jenkins manually using sudo /etc/init.d/jenkins start. The plan is to reprovision that machine with more RAM and disk space, and then build it up with Ubuntu 12.10 to test and tweak the build script; doing this for every Ubuntu release will make it less messy to migrate to the next LTS when it comes out.

MC changed the include_path and extension_dir settings -- it appears there was an update in May, at which time these should have been changed but weren't. However, the app still won't work. There are two remaining settings that look wrong to me:

MYSQL_INCLUDE     no value
MYSQL_LIBS     no value 

On a phpinfo.php for PHP5, these show as:

MYSQL_INCLUDE     -I/usr/local/include/mysql
MYSQL_LIBS     -L/usr/local/lib/mysql -lmysqlclient 

I'm waiting to see if a fix to these will do the job. If not, we'll have to start picking carefully through every one of the settings, trying to figure out what might be wrong. The only other thing I can think of is that PEAR might be broken somehow in this new install.

In the long run, we should get Agenda shifted from PHP4 to PHP5. If this update was done in May, and we're the first to report that the paths were broken, then clearly hardly anyone is using it any more.

This morning I was contacted by two different departments to report that the Agenda application used to schedule classes is not working. On investigation, I discovered that the login page is truncated immediately at the point where the first call to the database is made. After reading through the code, I couldn't see anything wrong, so I suspected a connection problem such as an ACL issue between web.uvic.ca/lang and csmgenr2 (the mysql server). The db is there, functioning, and accessible through PhpMyAdmin.

I raised a ticket with sysadmin, and MC got back to me to say that there doesn't seem to be a connection issue, but the error in the logs was with a PHP include:

2012-08-29T08:43:23-07:00 local@axolotl.comp.uvic.ca user.notice php_cgi: PHP
Warning: main() [<a href='function.include'>function.include</a>]: Failed
opening 'DB.php' for inclusion
(include_path='.:/usr/local/php-4.4.8/include/php:/usr/local/php-4.4.8/lib/php')
in
/home3/80/lang/www/agenda/Application_Files/script_files/php/database_connection.php
on line 11

The error comes when the db connection script tries to include DB.php, which is the PEAR library this project uses for accessing MySql. This sent me to check the PHP settings info in phpinfo.php, and I found a number of oddities:

• The PHP version is 4.4.9,
• the configure command output includes this: './configure' '--prefix=/usr/local/php-4.4.9', and
• the configuration file is here: /usr/local/php-4.4.9/lib/php.ini.
• HOWEVER, the extension_dir is /usr/local/php-4.4.8/lib/php/extension, and
• the include_path is .:/usr/local/php-4.4.8/include/php:/usr/local/php-4.4.8/lib/php.

My current theory is that when PHP was updated from 4.4.8 to 4.4.9, those key settings in the master php.ini file should have been changed, but weren't; and as a result, the include of DB.php is failing. I've reported this back to sysadmin, and I'm waiting for a response. In the meantime, I've tried overriding the include_path setting in a local php.ini, but I can't get that to work; perhaps those settings can't be overridden.

In the meantime, I could no longer bear to look at the logo which proclaimed "Agenda / Organize. Mangage. Simplfy" (sic, seriously), so I've fixed that.

Just blogging something I keep having to look up. This is how to switch your SVN repo from the old tapor URL to hcmc:

svn switch --relocate https://revision.tapor.uvic.ca/svn/[reponame] https://revision.hcmc.uvic.ca/svn/[reponame] 

Sysadmin fixed the ACLs (there were two sets of entries, apparently, which confused the process for a while), and now the machine is up and running with Jenkins on port 8080. I tried moving it to port 80, but was prevented by the Ubuntu rule which won't allow a service not running as root to run on a port below 1024 (and we definitely don't want Jinks running as root). So I may end up running Apache just so I can proxy it to port 80. Seems like overkill, but there doesn't seem to be an alternative solution.

I'm going to run the two machines side by side until I come back from vacation, so we can make sure the new one is stable before we switch to it and bring down the old one.

The new VM was set up today, so Greg built it with Ubuntu 12.04, and I ran my Jinks-builder script on it. Everything seemed to go perfectly, and presumably it's building away behind the scenes, but because the ACL prevents any access to it except on port 22, I can't get in to finish the setup yet. We'll get there eventually...