Maintenance

I've rolled back my Oxygen install to 18.0 following this experience. I don't know whether it is an Oxygen bug or not, but I never saw it prior to upgrading to 18.1.

MC reported he'd had to restart tomcat-devel several times, so we started looking through logs. I think I've traced the current Tomcat issue to a weird combination of circumstances.

Way back when, the Map of Early Modern London project used to be a PHP app. It had URLs like this:

mapoflondon.uvic.ca/render_page.php?id=LIME1

We turned it into an eXist app, with proper page URLS like this:

mapoflondon.uvic.ca/LIME1.htm

I wrote some cunning logic in the app to detect requests to the old URLs and redirect them to the new. However, the logic never fired in normal circumstances, because Apache on the front-end saw ".php" and passed the page off to the PHP interpreter.

To deal with this, I and someone from Systems cooked up an Apache rewrite rule in the virtual host, which would turn the old URL into the new. However, it's not very sophisticated; it treats search parameters very crudely, so it turns this:

/render_page.php?id=LIME1&title=Lime%20Street/

into this:

/LIME1&title=Lime%20Street/.htm

which throws up errors when eXist sees it, naturally. Now, it turns out that this particular URL, presumably along with some other similar ones, had been links on the old site way back when, and are now in places such as archive.org, and some bot somewhere has been hitting these URLs, and generating a lot of errors.

I think at this point in time it might make sense to revisit that old rewrite rule and make it much simpler, so that it converts anything which contains the string:

"render_page.php"

into

"404.htm"

and doesn't worry about trying to parse it at all. Wrote to MC to see if he thinks that makes sense.

We started work with a standard checkout of OSM Bright, and we worked against the model of openstreemap-carto-common/style.xml, which we have already successfully used to build tiles. OSM Bright itself cannot be compiled with the carto tool, because it's missing key information.

The sample project is in /home/mholmes/WorkData/maps/osm-bright/hcmc-plain/ on my machine.

First of all, we edited two of the four .mss files, base.mss and palette.mss, renaming them. In this process, we simply commented out a lot of the content, since we're aiming for a land/water/nothing else view. The other two mss files were ignored.

Next, we copied osm-bright.osm2pgsql.mml to hcmc-plain.osm2pgsql.mml, and began editing it. This is the file which is compiled using carto to create the XML styles file which is used by the tile builder. There was a lot to learn:

• The osm-bright version we were starting from was broken because it was missing a style attribute required on the two Datasource objects: "type": "shape",. Without this, carto fails to process the resulting XML. In this example file, it's on the two initial Datasource elements, which are based on shape files which are downloaded (see previous post).
• The files in osm-bright/res and osm-bright/img need to be in the same place relative to the .xml styles file you create, so the simplest thing is to place those folders alongside the .xml file when building the tiles.
• The shape file data needs to be accessible and linked from the .xml file. Since that data is in /usr/share/openstreetmap-carto-common/data on the maptiler server, we simply created a symlink from "data" in the tile-generation location to that folder, and made the link a local one in the mml file ("file": "data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp").
• The default name for the database in the mml file is "osm"; on my maptiler server, it's "gis", so that needed to be changed: "dbname": "gis" in 28 places.
• Most layers in the mml file have "status": "on" by default; we turned most of them off, sometimes adding the status attribute when it was missing.

Then we compiled the mml file using carto:

carto hcmc-plain.osm2pgsql.mml > hcmc-plain.xml

Then we copied that file (along with img and res folders) up to the maptiler server, configured generate_tiles.py, and ran it.

This process resulted in UEFI secure boot being turned off, because apparently I'm running some driver or other that won't work with it, but other than that all seems well.

ST has supplied a bounding box for her Stolo project. Unfortunately it's huge; but I rendered it with zoom levels 10 through 16, and it took only a few hours, generating 47,762 tiles. That may not be enough detail, though. Relevant segment from /home/maptiler/stolo/generate_tiles.py:

    render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")

    minZoom = 10
    maxZoom = 16

    stolo = (-122.508545,48.789676, -121.322021,49.378797)
    render_tiles(stolo, mapfile, tile_dir, minZoom, maxZoom)

I've now set it going again (using screen) to render the next two zoom levels (17 and 18), and we'll see how long that takes.

For the TCCD project, I really would like to get a Jenkins job building things, but the problem is that everything is on GitHub and the repo is enormous due to the huge number of images. Partial checkouts with git are tricky. However, by looking at this documentation, I've figured out and tested a way to script a partial checkout using svn instead.

Say you want only the [proj]/level1/level2/level3 directory:

svn co -N https://github.com/[proj] proj
cd proj
svn up -N trunk
svn up -N trunk/level1
svn up -N trunk/level1/level2
svn up -N trunk/level1/level2/level3

That will get you a sparse checkout down to the contents of the level3 directory.

I have a little workgroup of TEI folks helping with an update/rewrite of the Jenkins server building script from 2012, ready for Ubuntu 16.04. I've now worked through about two thirds of the script, updating, fixing and tweaking, and my test setup is working well. The only gnarly problem so far is an issue with Japanese fonts, and I'm waiting for some LATEX expertise to help me resolve that one.

Set up the machine

1. Boot VM and do a regular install.
2. Update the VM: 'sudo apt-get update && sudo apt-get -y dist-upgrade'
3. Install required packages: 'sudo apt-get -y install apache2 apache2-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto osm2pgsql python-mapnik dh-autoreconf node-carto'

Tweak OS

1. Open postgres config file: 'sudo nano /etc/postgresql/9.5/main/postgresql.conf'
2. Uncomment the line: '#maintenance_work_mem = 64MB'
3. and change memory allocation to 256MB: 'maintenance_work_mem = 256MB'
4. Save and exit
5. Open system control config: 'sudo nano /etc/sysctl.conf'
6. Add this to top of file: 'kernel.shmmax=268435456'
7. Reboot machine and run 'sudo sysctl kernel.shmmax'
8. The result should reflect the change: 268435456

1. 'sudo visudo'
2. add this to end of file: '{your-username} ALL=(ALL) NOPASSWD:ALL'

Prepare database

1. Become postgres user: 'sudo -u postgres -i'
2. Create a postgres user for our OSM tasks: 'createuser osmdbadmin'
3. Invoke postgres cli: 'psl'
4. The installation of mapnik et al. should have created a database called 'gis'. Let's check: '\c gis'
5. You should now have a command prompt that looks like this: 'gis=#'
6. Create the postgis extension: 'CREATE EXTENSION postgis;'
7. Change ownership of geometry table: 'ALTER TABLE geometry_columns OWNER TO osmdbadmin;'
8. Change ownership of spatial ref table: 'ALTER TABLE spatial_ref_sys OWNER TO osmdbadmin;'
9. Back out of psql: '\q'

Add data to database

1. Create a directory for our download: 'sudo mkdir -p /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia'
2. Download the data: 'sudo wget --directory-prefix=/usr/share/openstreetmap-carto/data/north-america/canada/british-columbia https://s3.amazonaws.com/metro-extracts.mapzen.com/victoria_canada.osm.pbf'
3. Become postgres user to import data: 'sudo -u postgres -i'
4. Import using osm2pgsql: 'osm2pgsql --slim -d gis -C 16000 --number-processes 3 /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia/victoria_canada.osm.pbf'

Set up mod_tile/renderd

1. Check out the code to your homedir: 'cd ~ && git clone git://github.com/openstreetmap/mod_tile.git'
2. Start building it: 'cd mod_tile && ./autogen.sh'
3. Configure: './configure'
4. Make and install: 'make && sudo make install'
5. Install mod_tile itself: 'sudo make install-mod_tile && sudo ldconfig'
6. There are apache setup instructions here if you want to set that end of things up.

Configure renderd

1. Open conf file for editing: 'sudo nano /usr/local/etc/renderd.conf'
2. Uncomment this line at the top of the file: 'socketname=/var/run/renderd/renderd.sock'
3. In the [default] section, change the XML line to look like this: 'XML=/usr/share/openstreetmap-carto/style.xml'

Set up manual tile generator

1. This script must be run by a user that has access to the db. I created an OS user that matches the db admin name: 'sudo adduser osmdbadmin'
2. Switch to osmdbadmin user: 'su osmdbadmin'
3. Download the script: 'cd ~ && wget https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py'
4. Edit for opening: 'nano ~/generate_tiles.py'
5. Make it executable: 'chmod +x ~/generate_tiles.py'
6. Change stylesheet pointer (around line 198) to: '/usr/share/openstreetmap-carto/style.xml'
7. Remove everything at the end that references 'World' and German cities (lines starting with 'bbox' and 'render_tiles').
8. Add a stanza that looks like this:
   # University of Victoria
   bbox = (-123.4065,48.4150, -123.2145,48.5051)
   render_tiles(bbox, mapfile, tile_dir, 14, 18 , "UVic")
   
9. Save and close the file.
10. You should now be able to run the generator: 'cd ~ && ./generate_tiles.py'
11. Read notes on errors, below.

Notes on generate_tiles.py errors

#!/bin/bash
sudo sed -i 's|<Font face-name="Arundina Sans Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Droid Sans Fallback Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="gargi Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Normal"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige NormalItalic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Tibetan Machine Uni Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml

sudo sed -i 's|minzoom|minimum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|maxzoom|maximum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml

The import of the North America data into my VM took a total of 25 hours, but seems to have completed successfully. Yay. Next step, tomorrow, trying to build some actual tiles.

Back in 2012, I developed a script for automatically building a TEI Jenkins server, and it was used to create the Jenkins server which is the current TEI build server. That server has now been substantially updated, and we have a TEI working group to write a new script based on Ubuntu 16.04, and to include the more complex Jenkins jobs that we now run.

This stuff is being developed in TEIC/Jenkins on GitHub, and I've updated some of my original scripts which help to automate the generation and configuration of test VMs for running the script. I now have a generic PreJenkins machine, as before, and I can build and run a clone from it. I'm up to the point where everything looks good and the builder script is ready to start installing stuff. At this point, I'll wait for confirmation from the rest of the WG on a couple of issues I've raised (which java we should use, and whether we should install the TEI packages).