Maintenance

So, I've deployed a replacement for papaya. In the process I've discovered that Google Earth no longer has auto-updates available. This means that if you want to install Google Earth 7 or later, you need to download and install the full package.
I've written an install script in to the hcmc-config app in the build, so it isn't hard to install when needed.

JT had trouble with freezing on a machine yesterday, and doing updates seemed to fix it, so I booted them all up and did all the updates.

I cannot for the life of me find a way to get Jenkins to check out anything from revision.hcmc.uvic.ca (which is only available through https/apache). No configuration of credential settings, ssh keys, user names, passwords or anything else will work. I'm floored, and this stops our build server from being practical.

If we can't get this working, and we can't enable revision.hcmc.uvic.ca svn repos to be anonymous-readable (ticket in with sysadmin right now), then we'll have to take the projects we want to build to SF, Github, or somewhere else we know will work.

vips (available on the Linux command line) can create Zoomify tilesets like this:

vips dzsave clean_C1.png clean_C1 --layout zoomify --suffix .png

As far as we can tell, these are identical to those created by Zoomify (and we got the same number of tile groups folders using vips as we did with Zoomify).

We actually need JPEGs because OpenLayers can't yet handle PNGs, so we've determined that this is what we want for tiling (assuming that we want full-quality images).

 vips dzsave img.jpg img --layout zoomify --suffix .jpg[Q=100]

Linguistics have a rather confusing audio setup in their booth which is basically using an entire mixing desk to supply phantom power to a mic; they also have problems with playback to the in-booth speakers because the're missing a Y-splitter cable to feed mono 1/4-inch jacks to a stereo mini jack to the speakers. Greg and I spent some time working around the problem, and arrived at a setup which works perfectly in Audacity and has nice level settings, but it's not working with the VocaliD system which runs in the Chrome browser; they're going to contact tech support for VocaliD about that. I wish they had a limiter on their mic input too; the head-mounted mic very easily clips.

Note to self, because I'll forget:

git ls-remote --tags https://github.com/TEIC/Stylesheets.git | egrep -o "([0-9]+\.)+[0-9]+" | sort -t . -k 1,1n -k 2,2n -k 3,3n | tail -n1

This will work assuming version numbers are x.y.z.

New monitors came, set up new standing desk, after switching my desk with one from the other room which is more suitable for it. Time to completion about two hours. Reminder to self: I need to find and bring a 14mm hex wrench to adjust the counterweight so that it stays in position more firmly.

We've decided to use PostGreSQL for this project due to its more sophisticated static views ('materialized views') and the efficient updating of materialized views ('refresh concurrently').
Downside? These features don't show up until v9.4, so I installed 9.4 and PHPPGadmin on my VM.

I now have a reasonable first pass at a schema - for now it only addresses the core data sets (soldiers, war diaries, and ancillary tables that help flesh out those main tables).

In order to clean up as much as possible, Martin is going to write some (streaming) XSLT to smooth out some of the inevitable bumps that exist in data sets of this size. To produce a MySQL dump in XML format, you can use PHPmyAdmin or mysqldump on the CLI. Doing it from PHPmyAdmin I ended up with a 908MB file, whereas the command 'mysqldump --xml -u root -p cgwp > /tmp/cgwp.xml' produces a 729MB file. I haven't investigated why this is, but the header of the documents suggest that PHPmyAdmin has its own take on dumping XML.

I built a 14.04 server with Apache, PHP, MariaDB 10, and PHPMyAdmin and pushed the entire sql dump at it (structure and data in the same 200MB file). It gobbled it up without ANY drama. All data looks usable and complete (170,000+ soldiers).
Documentation isn't really necessary, but I followed the MariaDB instructions here, and the rest is basically bog-standard stuff (see here for a step-by-step).

The data I pushed out of MS SQL is kind of cruft-ey. I'm using some sed commands to clean up:
sed -i '/^-- /d' CGWP_data.sql
sed -i '/^LOCK TABLES/d' CGWP_data.sql
sed -i '/^\/\*\!/d' CGWP_data.sql
sed -i '/^UNLOCK TABLES/d' CGWP_data.sql
sed -i '/^[[:space:]]*$/d' CGWP_data.sql
csplit CGWP_data.sql '/^INSERT INTO/'

the csplit doesn't work properly though. I'll look in to it another day.