Maintenance

Note to self, very handy thing:

avahi-browse -vatr

ST has supplied a bounding box for her Stolo project. Unfortunately it's huge; but I rendered it with zoom levels 10 through 16, and it took only a few hours, generating 47,762 tiles. That may not be enough detail, though. Relevant segment from /home/maptiler/stolo/generate_tiles.py:

    render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")

    minZoom = 10
    maxZoom = 16

    stolo = (-122.508545,48.789676, -121.322021,49.378797)
    render_tiles(stolo, mapfile, tile_dir, minZoom, maxZoom)

I've now set it going again (using screen) to render the next two zoom levels (17 and 18), and we'll see how long that takes.

For the TCCD project, I really would like to get a Jenkins job building things, but the problem is that everything is on GitHub and the repo is enormous due to the huge number of images. Partial checkouts with git are tricky. However, by looking at this documentation, I've figured out and tested a way to script a partial checkout using svn instead.

Say you want only the [proj]/level1/level2/level3 directory:

svn co -N https://github.com/[proj] proj
cd proj
svn up -N trunk
svn up -N trunk/level1
svn up -N trunk/level1/level2
svn up -N trunk/level1/level2/level3

That will get you a sparse checkout down to the contents of the level3 directory.

I have a little workgroup of TEI folks helping with an update/rewrite of the Jenkins server building script from 2012, ready for Ubuntu 16.04. I've now worked through about two thirds of the script, updating, fixing and tweaking, and my test setup is working well. The only gnarly problem so far is an issue with Japanese fonts, and I'm waiting for some LATEX expertise to help me resolve that one.

Set up the machine

1. Boot VM and do a regular install.
2. Update the VM: 'sudo apt-get update && sudo apt-get -y dist-upgrade'
3. Install required packages: 'sudo apt-get -y install apache2 apache2-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto osm2pgsql python-mapnik dh-autoreconf node-carto'

Tweak OS

1. Open postgres config file: 'sudo nano /etc/postgresql/9.5/main/postgresql.conf'
2. Uncomment the line: '#maintenance_work_mem = 64MB'
3. and change memory allocation to 256MB: 'maintenance_work_mem = 256MB'
4. Save and exit
5. Open system control config: 'sudo nano /etc/sysctl.conf'
6. Add this to top of file: 'kernel.shmmax=268435456'
7. Reboot machine and run 'sudo sysctl kernel.shmmax'
8. The result should reflect the change: 268435456

1. 'sudo visudo'
2. add this to end of file: '{your-username} ALL=(ALL) NOPASSWD:ALL'

Prepare database

1. Become postgres user: 'sudo -u postgres -i'
2. Create a postgres user for our OSM tasks: 'createuser osmdbadmin'
3. Invoke postgres cli: 'psl'
4. The installation of mapnik et al. should have created a database called 'gis'. Let's check: '\c gis'
5. You should now have a command prompt that looks like this: 'gis=#'
6. Create the postgis extension: 'CREATE EXTENSION postgis;'
7. Change ownership of geometry table: 'ALTER TABLE geometry_columns OWNER TO osmdbadmin;'
8. Change ownership of spatial ref table: 'ALTER TABLE spatial_ref_sys OWNER TO osmdbadmin;'
9. Back out of psql: '\q'

Add data to database

1. Create a directory for our download: 'sudo mkdir -p /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia'
2. Download the data: 'sudo wget --directory-prefix=/usr/share/openstreetmap-carto/data/north-america/canada/british-columbia https://s3.amazonaws.com/metro-extracts.mapzen.com/victoria_canada.osm.pbf'
3. Become postgres user to import data: 'sudo -u postgres -i'
4. Import using osm2pgsql: 'osm2pgsql --slim -d gis -C 16000 --number-processes 3 /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia/victoria_canada.osm.pbf'

Set up mod_tile/renderd

1. Check out the code to your homedir: 'cd ~ && git clone git://github.com/openstreetmap/mod_tile.git'
2. Start building it: 'cd mod_tile && ./autogen.sh'
3. Configure: './configure'
4. Make and install: 'make && sudo make install'
5. Install mod_tile itself: 'sudo make install-mod_tile && sudo ldconfig'
6. There are apache setup instructions here if you want to set that end of things up.

Configure renderd

1. Open conf file for editing: 'sudo nano /usr/local/etc/renderd.conf'
2. Uncomment this line at the top of the file: 'socketname=/var/run/renderd/renderd.sock'
3. In the [default] section, change the XML line to look like this: 'XML=/usr/share/openstreetmap-carto/style.xml'

Set up manual tile generator

1. This script must be run by a user that has access to the db. I created an OS user that matches the db admin name: 'sudo adduser osmdbadmin'
2. Switch to osmdbadmin user: 'su osmdbadmin'
3. Download the script: 'cd ~ && wget https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py'
4. Edit for opening: 'nano ~/generate_tiles.py'
5. Make it executable: 'chmod +x ~/generate_tiles.py'
6. Change stylesheet pointer (around line 198) to: '/usr/share/openstreetmap-carto/style.xml'
7. Remove everything at the end that references 'World' and German cities (lines starting with 'bbox' and 'render_tiles').
8. Add a stanza that looks like this:
   # University of Victoria
   bbox = (-123.4065,48.4150, -123.2145,48.5051)
   render_tiles(bbox, mapfile, tile_dir, 14, 18 , "UVic")
   
9. Save and close the file.
10. You should now be able to run the generator: 'cd ~ && ./generate_tiles.py'
11. Read notes on errors, below.

Notes on generate_tiles.py errors

#!/bin/bash
sudo sed -i 's|<Font face-name="Arundina Sans Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Droid Sans Fallback Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="gargi Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Normal"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige NormalItalic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Tibetan Machine Uni Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml

sudo sed -i 's|minzoom|minimum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|maxzoom|maximum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml

The import of the North America data into my VM took a total of 25 hours, but seems to have completed successfully. Yay. Next step, tomorrow, trying to build some actual tiles.

Back in 2012, I developed a script for automatically building a TEI Jenkins server, and it was used to create the Jenkins server which is the current TEI build server. That server has now been substantially updated, and we have a TEI working group to write a new script based on Ubuntu 16.04, and to include the more complex Jenkins jobs that we now run.

This stuff is being developed in TEIC/Jenkins on GitHub, and I've updated some of my original scripts which help to automate the generation and configuration of test VMs for running the script. I now have a generic PreJenkins machine, as before, and I can build and run a clone from it. I'm up to the point where everything looks good and the builder script is ready to start installing stuff. At this point, I'll wait for confirmation from the rest of the WG on a couple of issues I've raised (which java we should use, and whether we should install the TEI packages).

Note: this post covers several days, and I expect to keep updating it as we learn more.

I'm attempting to replicate, with documentation, Greg's building of an OSM tile-rendering machine, starting from instructions here:

https://switch2osm.org/serving-tiles/manually-building-a-tile-server-14-04/

but using 16.04 beta 2. I first built a VM from Ubuntu 16.04 server beta2, with 16GB RAM and 200GB of disk space, adding only the install of ssh-server at install time, and selecting the non-LVM full-disk option. These are deviations that I had to deal with:

In the first set of software installs:

libtiff4-dev needs to be changed to libtiff5-dev

Then I did followed the instructions for installing and configuring postgres although the package version was different (do apt-cache search to find the current one). When I got to osm2pgsql, I just did

sudo apt-get install osm2pgsql

and then for mapnik:

sudo apt-get install libmapnik-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto python-mapnik python3-mapnik

At some point, the node-carto package got installed; that's a good thing, but I don't know how it happened.

Next, I had chosen to use openstreetmap-carto instead of OSMBright, so I didn't need to do that bit.

The next stumbling block was that when you edit the renderd.conf file, the paths proposed are not correct, because things are in different places due to installing from the repos rather than building locally. You want this:

socketname=/var/run/renderd/renderd.sock
plugins_dir=/usr/lib/mapnik/3.0/input
font_dir=/usr/share/fonts/truetype/ttf-dejavu
XML=/usr/share/openstreetmap-carto/style.xml
HOST=localhost

When you come to tuning postgres, only three of the four settings can be found. Don't add the missing setting, just set the others:

shared_buffers = 128MB
#checkpoint_segments = 20    ##DO NOT ADD THIS: IT PREVENTS POSTGRES FROM STARTING.
maintenance_work_mem = 256MB
autovacuum = off

I got the north-america-latest.osm.pbf, and you have to remember to chown its containing folder and itself to the user you created for tile-building. Then you su to that user and run the import command. That took 25 hours.

Starting renderd worked, and I was able to view specific tiles through Apache, after getting the paths of those tiles from GN; there's no default interface that allows you to browse tiles (there used to be, apparently).

The next stage is getting renderd to start automatically; the original instructions predate systemd, so that's potentially problematic. We've shelved that for the moment.

The next stage is to download the generate_tiles.py file from GitHub:

As the maptiler user, in the maptiler user's home directory:

curl https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py > generate_tiles.py
chmod u+x generate_tiles.py

Then you need to edit generate_tiles.py. You can comment out all the German cities at the end, but use one of them as a model for your own bounding box and render instruction. Create the bounding box (bottom-left, top-right) using this:

http://hcmc.uvic.ca/people/greg/mapnik/bbox.html

To avoid rendering the whole world in too much detail, change this line:

render_tiles(bbox, mapfile, tile_dir, 0, 5, "World")

to this:

render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")

You'll also need to change the path to the stylesheet:

mapfile = "/usr/share/openstreetmap-carto/style.xml"

Now when you try to run the script, you'll get this error:

RuntimeError: Failed to find font face 'Arundina Sans Italic' in FontSet 'fontset-0' in FontSet at line 17 of '/usr/share/openstreetmap-carto/style.xml'

So fonts required by the stylesheet are missing. These are my attempts to resolve that problem:

The Arundina fonts package is already installed by default, so that's a puzzle. I tried this:

sudo apt-get install latex-fonts-sipa-arundina

That put some fonts there, but they have the wrong names by default. So in this file:

/usr/share/openstreetmap-carto/style.xml

replace "Arundina Sans Italic" with "Arundina Italic".

Next is a missing Tamil font, Paranar:

sudo apt-get install fonts-taml-tscu

and so on for a range of other fonts:

sudo apt-get install fonts-knda fonts-droid-fallback fonts-noto fonts-sil-padauk fonts-indic fonts-tibetan-machine

However, there are still some fonts which are not available, but are references in the style.xml file. For the remainder, where I was able to find a good analogue or close option, I substituted it; otherwise, I commented out the line in the file which referenced it.

For finding out which font names python knows about, this helps:

from mapnik import *
for face in FontEngine.face_names():
  print face

Out of curiosity, I just tested WordPress 4.4.2 with the SQLite plugin.

Including download time, it took less than 5 minutes to get it working. Instructions are simple and easy to follow.

For fun, I also installed phpLiteAdmin, which allows phpMyAdmin-like access to your DB. Installing/configuring this added about 1 minute to my total investment of time.

The advantage is that everything you need to back up is in one folder. There is no daemon for SQLite, so there *should* be a reduction in memory usage too.

Links:
WordPress: https://wordpress.org/download/
WP SQlite plugin: https://wordpress.org/plugins/sqlite-integration/installation/
phpLiteAdmin: https://www.phpliteadmin.org/download/

Based on these instructions and the bit on pre-rendering tiles in this set of instructions, along with and the set of already-available docker recipes, it seems like we may be able to get one or more docker instances running that would build specified tilesets for us (pre-rendering them).