Build a Mapnik Tileserver - no longer academic

08/04/16

Permalink 02:09:52 pm, by Greg, 1217 words, 564 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation, Documentation; Mins. worked: 540

Build a Mapnik Tileserver - no longer academic

Having successfully built a 14.04 tileserver using the instructions here, I've decided to do a new build, this time using Ubuntu Server16.04, Mapnik 3, and Carto instead of 14.04, Mapnik 2, and OSMBright. This post is intended to be a full copy-paste how-to. Using it, I successfully built the (> 1 million) tiles necessary for this.

Set up the machine

I'm using VirtualBox on linux. The VM is configured with 16GB RAM, 200GB virtual HDD, and bridged networking. I'm installing Ubuntu Server 16.04 (currently in beta, so I get it from here.
  1. Boot VM and do a regular install.
  2. Update the VM: 'sudo apt-get update && sudo apt-get -y dist-upgrade'
  3. Install required packages: 'sudo apt-get -y install apache2 apache2-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto osm2pgsql python-mapnik dh-autoreconf node-carto'

Tweak OS

As this is going to be doing some pretty heavy lifting, you'll want to tweak the way the system handles DB work. We'll tweak Postgres and the kernel to do this.
  1. Open postgres config file: 'sudo nano /etc/postgresql/9.5/main/postgresql.conf'
  2. Uncomment the line: '#maintenance_work_mem = 64MB'
  3. and change memory allocation to 256MB: 'maintenance_work_mem = 256MB'
  4. Save and exit
  5. Open system control config: 'sudo nano /etc/sysctl.conf'
  6. Add this to top of file: 'kernel.shmmax=268435456'
  7. Reboot machine and run 'sudo sysctl kernel.shmmax'
  8. The result should reflect the change: 268435456
NOTE: On a private machine like this I like to set up passwordless sudo for my main admin user (obviously not recommended for a 'real' machine):
  1. 'sudo visudo'
  2. add this to end of file: '{your-username} ALL=(ALL) NOPASSWD:ALL'

Prepare database

  1. Become postgres user: 'sudo -u postgres -i'
  2. Create a postgres user for our OSM tasks: 'createuser osmdbadmin'
  3. Invoke postgres cli: 'psl'
  4. The installation of mapnik et al. should have created a database called 'gis'. Let's check: '\c gis'
  5. You should now have a command prompt that looks like this: 'gis=#'
  6. Create the postgis extension: 'CREATE EXTENSION postgis;'
  7. Change ownership of geometry table: 'ALTER TABLE geometry_columns OWNER TO osmdbadmin;'
  8. Change ownership of spatial ref table: 'ALTER TABLE spatial_ref_sys OWNER TO osmdbadmin;'
  9. Back out of psql: '\q'

Add data to database

We need some data to put in the database. We'll use the metropolitan Victoria area from mapzen:
  1. Create a directory for our download: 'sudo mkdir -p /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia'
  2. Download the data: 'sudo wget --directory-prefix=/usr/share/openstreetmap-carto/data/north-america/canada/british-columbia https://s3.amazonaws.com/metro-extracts.mapzen.com/victoria_canada.osm.pbf'
  3. Become postgres user to import data: 'sudo -u postgres -i'
  4. Import using osm2pgsql: 'osm2pgsql --slim -d gis -C 16000 --number-processes 3 /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia/victoria_canada.osm.pbf'
The VM is now set up, and the data is in the database. In order to actually create the tiles we need a renderer. We're using mod_tile/renderd, which is an apache module. It allows apache to respond to tile requests by first checking if the tile exists. If it does, apache serves it. If it doesn't, renderd renders it and hands it to apache for serving. The tile is then available from the filesystem.

Set up mod_tile/renderd

  1. Check out the code to your homedir: 'cd ~ && git clone git://github.com/openstreetmap/mod_tile.git'
  2. Start building it: 'cd mod_tile && ./autogen.sh'
  3. Configure: './configure'
  4. Make and install: 'make && sudo make install'
  5. Install mod_tile itself: 'sudo make install-mod_tile && sudo ldconfig'
  6. There are apache setup instructions here if you want to set that end of things up.

Configure renderd

Renderd is a daemon that mod_tile uses for the rendering of new tiles. If you'll be using the full functionality of this stack you will need to follow the instructions here.
  1. Open conf file for editing: 'sudo nano /usr/local/etc/renderd.conf'
  2. Uncomment this line at the top of the file: 'socketname=/var/run/renderd/renderd.sock'
  3. In the [default] section, change the XML line to look like this: 'XML=/usr/share/openstreetmap-carto/style.xml'

Set up manual tile generator

This allows you to explicitly construct a complete tile set without using apache/mod_tile. It uses a python script called 'generate_tiles.py'.
  1. This script must be run by a user that has access to the db. I created an OS user that matches the db admin name: 'sudo adduser osmdbadmin'
  2. Switch to osmdbadmin user: 'su osmdbadmin'
  3. Download the script: 'cd ~ && wget https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py'
  4. Edit for opening: 'nano ~/generate_tiles.py'
  5. Make it executable: 'chmod +x ~/generate_tiles.py'
  6. Change stylesheet pointer (around line 198) to: '/usr/share/openstreetmap-carto/style.xml'
  7. Remove everything at the end that references 'World' and German cities (lines starting with 'bbox' and 'render_tiles').
  8. Add a stanza that looks like this:
    # University of Victoria
    bbox = (-123.4065,48.4150, -123.2145,48.5051)
    render_tiles(bbox, mapfile, tile_dir, 14, 18 , "UVic")
  9. Save and close the file.
  10. You should now be able to run the generator: 'cd ~ && ./generate_tiles.py'
  11. Read notes on errors, below.

Notes on generate_tiles.py errors

The stock stylesheet references a bunch of fonts that are not installed by default. I have heavy-handedly removed these references with a script, but Martin has made an effort to find and install the right fonts, or reasonable alternates. Take a look at his post for progress. Other than nuking the missing fonts, my script also address an annoying warning that you get about deprecated code. Here's the script:
#!/bin/bash
sudo sed -i 's|<Font face-name="Arundina Sans Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Droid Sans Fallback Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="gargi Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Normal"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige NormalItalic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Tibetan Machine Uni Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml

sudo sed -i 's|minzoom|minimum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|maxzoom|maximum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml

Pingbacks:

No Pingbacks for this post yet...

Maintenance

This blog is the location for all work involving software and hardware maintenance, updates, installs, etc., both routine and urgent, in the server room, the labs and the R&D rooms.

Reports

XML Feeds