Maintenance

Due to a bug in the svn client in Jenkins, svn externals cannot be handled, so I had to set the Jenkins jobs to "ignore externals" (the default), and then check for the presence of the external file(s) at the beginning of the build process; if they're missing, ant will do an svn export using exec to put it where we need it. Curses. But working now.

I'm blogging this because the Red Bean book is confusing, various different syntaxes have worked in different svn versions, and it took me a few minutes to figure out what would work.

I wanted to centralize some XQuery modules that will be shared between different projects in Endings (and elsewhere potentially). This is how I ended up doing it (example from Mariage):

[In the Mariage repo, "static" folder:]
[Create a local folder in the repo for all externals:]
svn mkdir externals
cd externals

[Specify a subfolder name which will be external, along with the remote path to the source repo:]
svn propset svn:externals "xquery https://revision.hcmc.uvic.ca/svn/hcmc/xquery" .

[svn up and svn commit]

A number of jobs were not configured to discard old builds, so I had to manually prune them by working on the filesystem. There were a lot of plugin updates to do as well, and a surprising number of plugins are installed; it seems to me that a lot of basic functionality is now handled through plugins rather than bundled into the main app, so the number of plugins has proliferated.

Our Android tablet is sluggish and not really useful, so when I saw a post about the release of this useful tool, I used it to install Cyanogenmod 12.1 on the tablet. Works a treat, and is much faster. Keep this tool handy.

I've rolled back my Oxygen install to 18.0 following this experience. I don't know whether it is an Oxygen bug or not, but I never saw it prior to upgrading to 18.1.

MC reported he'd had to restart tomcat-devel several times, so we started looking through logs. I think I've traced the current Tomcat issue to a weird combination of circumstances.

Way back when, the Map of Early Modern London project used to be a PHP app. It had URLs like this:

mapoflondon.uvic.ca/render_page.php?id=LIME1

We turned it into an eXist app, with proper page URLS like this:

mapoflondon.uvic.ca/LIME1.htm

I wrote some cunning logic in the app to detect requests to the old URLs and redirect them to the new. However, the logic never fired in normal circumstances, because Apache on the front-end saw ".php" and passed the page off to the PHP interpreter.

To deal with this, I and someone from Systems cooked up an Apache rewrite rule in the virtual host, which would turn the old URL into the new. However, it's not very sophisticated; it treats search parameters very crudely, so it turns this:

/render_page.php?id=LIME1&title=Lime%20Street/

into this:

/LIME1&title=Lime%20Street/.htm

which throws up errors when eXist sees it, naturally. Now, it turns out that this particular URL, presumably along with some other similar ones, had been links on the old site way back when, and are now in places such as archive.org, and some bot somewhere has been hitting these URLs, and generating a lot of errors.

I think at this point in time it might make sense to revisit that old rewrite rule and make it much simpler, so that it converts anything which contains the string:

"render_page.php"

into

"404.htm"

and doesn't worry about trying to parse it at all. Wrote to MC to see if he thinks that makes sense.

We started work with a standard checkout of OSM Bright, and we worked against the model of openstreemap-carto-common/style.xml, which we have already successfully used to build tiles. OSM Bright itself cannot be compiled with the carto tool, because it's missing key information.

The sample project is in /home/mholmes/WorkData/maps/osm-bright/hcmc-plain/ on my machine.

First of all, we edited two of the four .mss files, base.mss and palette.mss, renaming them. In this process, we simply commented out a lot of the content, since we're aiming for a land/water/nothing else view. The other two mss files were ignored.

Next, we copied osm-bright.osm2pgsql.mml to hcmc-plain.osm2pgsql.mml, and began editing it. This is the file which is compiled using carto to create the XML styles file which is used by the tile builder. There was a lot to learn:

• The osm-bright version we were starting from was broken because it was missing a style attribute required on the two Datasource objects: "type": "shape",. Without this, carto fails to process the resulting XML. In this example file, it's on the two initial Datasource elements, which are based on shape files which are downloaded (see previous post).
• The files in osm-bright/res and osm-bright/img need to be in the same place relative to the .xml styles file you create, so the simplest thing is to place those folders alongside the .xml file when building the tiles.
• The shape file data needs to be accessible and linked from the .xml file. Since that data is in /usr/share/openstreetmap-carto-common/data on the maptiler server, we simply created a symlink from "data" in the tile-generation location to that folder, and made the link a local one in the mml file ("file": "data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp").
• The default name for the database in the mml file is "osm"; on my maptiler server, it's "gis", so that needed to be changed: "dbname": "gis" in 28 places.
• Most layers in the mml file have "status": "on" by default; we turned most of them off, sometimes adding the status attribute when it was missing.

Then we compiled the mml file using carto:

carto hcmc-plain.osm2pgsql.mml > hcmc-plain.xml

Then we copied that file (along with img and res folders) up to the maptiler server, configured generate_tiles.py, and ran it.

I always have to look this up, so...

Hypothetically, you have a development server that is only accessible on-campus, from a specific subnet. But you want to use/view the app from some off-campus location.

SSH to the rescue:

ssh -nNT -L 8888:firewalled.server.uvic.ca:80 someuser@another.machine.uvic.ca

Then, go your browser and head to http://localhost:8888 and see the firewalled page!

The explanation:
flags -nNT > 'n' basically sends input to /dev/null, 'N' restricts the connection to just tunnelling, and 'T' stops the remote machine giving you a pseudo-TTY.

8888: > is the arbitrary port we'll use for the tunnelling

firewalled.server.uvic.ca > the server we want to access

:80 > the remote port you want to access. In this case it's a web server, so port 80

someuser@another.machine.uvic.ca > this connects a user to another remote machine that you have SSH access to, AND is allowed to access firewalled.server.uvic.ca on port 80.

Ran through a few iterations of the build script and fixed a number of bugs; I've now reached a point where all my previous code to configure Jenkins is based on the assumption that Jenkins starts up functioning normally after install, as it used to, but the behaviour has changed and there's now a randomly-created user name and password that the administrator has to use to log in from the beginning. I will need to figure out how to get around this, but it is a good feature in itself.

This process resulted in UEFI secure boot being turned off, because apparently I'm running some driver or other that won't work with it, but other than that all seems well.