Maintenance

I've rolled back my Oxygen install to 18.0 following this experience. I don't know whether it is an Oxygen bug or not, but I never saw it prior to upgrading to 18.1.

We started work with a standard checkout of OSM Bright, and we worked against the model of openstreemap-carto-common/style.xml, which we have already successfully used to build tiles. OSM Bright itself cannot be compiled with the carto tool, because it's missing key information.

The sample project is in /home/mholmes/WorkData/maps/osm-bright/hcmc-plain/ on my machine.

First of all, we edited two of the four .mss files, base.mss and palette.mss, renaming them. In this process, we simply commented out a lot of the content, since we're aiming for a land/water/nothing else view. The other two mss files were ignored.

Next, we copied osm-bright.osm2pgsql.mml to hcmc-plain.osm2pgsql.mml, and began editing it. This is the file which is compiled using carto to create the XML styles file which is used by the tile builder. There was a lot to learn:

• The osm-bright version we were starting from was broken because it was missing a style attribute required on the two Datasource objects: "type": "shape",. Without this, carto fails to process the resulting XML. In this example file, it's on the two initial Datasource elements, which are based on shape files which are downloaded (see previous post).
• The files in osm-bright/res and osm-bright/img need to be in the same place relative to the .xml styles file you create, so the simplest thing is to place those folders alongside the .xml file when building the tiles.
• The shape file data needs to be accessible and linked from the .xml file. Since that data is in /usr/share/openstreetmap-carto-common/data on the maptiler server, we simply created a symlink from "data" in the tile-generation location to that folder, and made the link a local one in the mml file ("file": "data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp").
• The default name for the database in the mml file is "osm"; on my maptiler server, it's "gis", so that needed to be changed: "dbname": "gis" in 28 places.
• Most layers in the mml file have "status": "on" by default; we turned most of them off, sometimes adding the status attribute when it was missing.

Then we compiled the mml file using carto:

carto hcmc-plain.osm2pgsql.mml > hcmc-plain.xml

Then we copied that file (along with img and res folders) up to the maptiler server, configured generate_tiles.py, and ran it.

I always have to look this up, so...

Hypothetically, you have a development server that is only accessible on-campus, from a specific subnet. But you want to use/view the app from some off-campus location.

SSH to the rescue:

ssh -nNT -L 8888:firewalled.server.uvic.ca:80 someuser@another.machine.uvic.ca

Then, go your browser and head to http://localhost:8888 and see the firewalled page!

The explanation:
flags -nNT > 'n' basically sends input to /dev/null, 'N' restricts the connection to just tunnelling, and 'T' stops the remote machine giving you a pseudo-TTY.

8888: > is the arbitrary port we'll use for the tunnelling

firewalled.server.uvic.ca > the server we want to access

:80 > the remote port you want to access. In this case it's a web server, so port 80

someuser@another.machine.uvic.ca > this connects a user to another remote machine that you have SSH access to, AND is allowed to access firewalled.server.uvic.ca on port 80.

This process resulted in UEFI secure boot being turned off, because apparently I'm running some driver or other that won't work with it, but other than that all seems well.

ST has supplied a bounding box for her Stolo project. Unfortunately it's huge; but I rendered it with zoom levels 10 through 16, and it took only a few hours, generating 47,762 tiles. That may not be enough detail, though. Relevant segment from /home/maptiler/stolo/generate_tiles.py:

    render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")

    minZoom = 10
    maxZoom = 16

    stolo = (-122.508545,48.789676, -121.322021,49.378797)
    render_tiles(stolo, mapfile, tile_dir, minZoom, maxZoom)

I've now set it going again (using screen) to render the next two zoom levels (17 and 18), and we'll see how long that takes.

For the TCCD project, I really would like to get a Jenkins job building things, but the problem is that everything is on GitHub and the repo is enormous due to the huge number of images. Partial checkouts with git are tricky. However, by looking at this documentation, I've figured out and tested a way to script a partial checkout using svn instead.

Say you want only the [proj]/level1/level2/level3 directory:

svn co -N https://github.com/[proj] proj
cd proj
svn up -N trunk
svn up -N trunk/level1
svn up -N trunk/level1/level2
svn up -N trunk/level1/level2/level3

That will get you a sparse checkout down to the contents of the level3 directory.

I have a little workgroup of TEI folks helping with an update/rewrite of the Jenkins server building script from 2012, ready for Ubuntu 16.04. I've now worked through about two thirds of the script, updating, fixing and tweaking, and my test setup is working well. The only gnarly problem so far is an issue with Japanese fonts, and I'm waiting for some LATEX expertise to help me resolve that one.

Set up the machine

1. Boot VM and do a regular install.
2. Update the VM: 'sudo apt-get update && sudo apt-get -y dist-upgrade'
3. Install required packages: 'sudo apt-get -y install apache2 apache2-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto osm2pgsql python-mapnik dh-autoreconf node-carto'

Tweak OS

1. Open postgres config file: 'sudo nano /etc/postgresql/9.5/main/postgresql.conf'
2. Uncomment the line: '#maintenance_work_mem = 64MB'
3. and change memory allocation to 256MB: 'maintenance_work_mem = 256MB'
4. Save and exit
5. Open system control config: 'sudo nano /etc/sysctl.conf'
6. Add this to top of file: 'kernel.shmmax=268435456'
7. Reboot machine and run 'sudo sysctl kernel.shmmax'
8. The result should reflect the change: 268435456

1. 'sudo visudo'
2. add this to end of file: '{your-username} ALL=(ALL) NOPASSWD:ALL'

Prepare database

1. Become postgres user: 'sudo -u postgres -i'
2. Create a postgres user for our OSM tasks: 'createuser osmdbadmin'
3. Invoke postgres cli: 'psl'
4. The installation of mapnik et al. should have created a database called 'gis'. Let's check: '\c gis'
5. You should now have a command prompt that looks like this: 'gis=#'
6. Create the postgis extension: 'CREATE EXTENSION postgis;'
7. Change ownership of geometry table: 'ALTER TABLE geometry_columns OWNER TO osmdbadmin;'
8. Change ownership of spatial ref table: 'ALTER TABLE spatial_ref_sys OWNER TO osmdbadmin;'
9. Back out of psql: '\q'

Add data to database

1. Create a directory for our download: 'sudo mkdir -p /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia'
2. Download the data: 'sudo wget --directory-prefix=/usr/share/openstreetmap-carto/data/north-america/canada/british-columbia https://s3.amazonaws.com/metro-extracts.mapzen.com/victoria_canada.osm.pbf'
3. Become postgres user to import data: 'sudo -u postgres -i'
4. Import using osm2pgsql: 'osm2pgsql --slim -d gis -C 16000 --number-processes 3 /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia/victoria_canada.osm.pbf'

Set up mod_tile/renderd

1. Check out the code to your homedir: 'cd ~ && git clone git://github.com/openstreetmap/mod_tile.git'
2. Start building it: 'cd mod_tile && ./autogen.sh'
3. Configure: './configure'
4. Make and install: 'make && sudo make install'
5. Install mod_tile itself: 'sudo make install-mod_tile && sudo ldconfig'
6. There are apache setup instructions here if you want to set that end of things up.

Configure renderd

1. Open conf file for editing: 'sudo nano /usr/local/etc/renderd.conf'
2. Uncomment this line at the top of the file: 'socketname=/var/run/renderd/renderd.sock'
3. In the [default] section, change the XML line to look like this: 'XML=/usr/share/openstreetmap-carto/style.xml'

Set up manual tile generator

1. This script must be run by a user that has access to the db. I created an OS user that matches the db admin name: 'sudo adduser osmdbadmin'
2. Switch to osmdbadmin user: 'su osmdbadmin'
3. Download the script: 'cd ~ && wget https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py'
4. Edit for opening: 'nano ~/generate_tiles.py'
5. Make it executable: 'chmod +x ~/generate_tiles.py'
6. Change stylesheet pointer (around line 198) to: '/usr/share/openstreetmap-carto/style.xml'
7. Remove everything at the end that references 'World' and German cities (lines starting with 'bbox' and 'render_tiles').
8. Add a stanza that looks like this:
   # University of Victoria
   bbox = (-123.4065,48.4150, -123.2145,48.5051)
   render_tiles(bbox, mapfile, tile_dir, 14, 18 , "UVic")
   
9. Save and close the file.
10. You should now be able to run the generator: 'cd ~ && ./generate_tiles.py'
11. Read notes on errors, below.

Notes on generate_tiles.py errors

#!/bin/bash
sudo sed -i 's|<Font face-name="Arundina Sans Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Droid Sans Fallback Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="gargi Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Normal"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige NormalItalic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Tibetan Machine Uni Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml

sudo sed -i 's|minzoom|minimum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|maxzoom|maximum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml

Back in 2012, I developed a script for automatically building a TEI Jenkins server, and it was used to create the Jenkins server which is the current TEI build server. That server has now been substantially updated, and we have a TEI working group to write a new script based on Ubuntu 16.04, and to include the more complex Jenkins jobs that we now run.

This stuff is being developed in TEIC/Jenkins on GitHub, and I've updated some of my original scripts which help to automate the generation and configuration of test VMs for running the script. I now have a generic PreJenkins machine, as before, and I can build and run a clone from it. I'm up to the point where everything looks good and the builder script is ready to start installing stuff. At this point, I'll wait for confirmation from the rest of the WG on a couple of issues I've raised (which java we should use, and whether we should install the TEI packages).

Note: this post covers several days, and I expect to keep updating it as we learn more.

I'm attempting to replicate, with documentation, Greg's building of an OSM tile-rendering machine, starting from instructions here:

https://switch2osm.org/serving-tiles/manually-building-a-tile-server-14-04/

but using 16.04 beta 2. I first built a VM from Ubuntu 16.04 server beta2, with 16GB RAM and 200GB of disk space, adding only the install of ssh-server at install time, and selecting the non-LVM full-disk option. These are deviations that I had to deal with:

In the first set of software installs:

libtiff4-dev needs to be changed to libtiff5-dev

Then I did followed the instructions for installing and configuring postgres although the package version was different (do apt-cache search to find the current one). When I got to osm2pgsql, I just did

sudo apt-get install osm2pgsql

and then for mapnik:

sudo apt-get install libmapnik-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto python-mapnik python3-mapnik

At some point, the node-carto package got installed; that's a good thing, but I don't know how it happened.

Next, I had chosen to use openstreetmap-carto instead of OSMBright, so I didn't need to do that bit.

The next stumbling block was that when you edit the renderd.conf file, the paths proposed are not correct, because things are in different places due to installing from the repos rather than building locally. You want this:

socketname=/var/run/renderd/renderd.sock
plugins_dir=/usr/lib/mapnik/3.0/input
font_dir=/usr/share/fonts/truetype/ttf-dejavu
XML=/usr/share/openstreetmap-carto/style.xml
HOST=localhost

When you come to tuning postgres, only three of the four settings can be found. Don't add the missing setting, just set the others:

shared_buffers = 128MB
#checkpoint_segments = 20    ##DO NOT ADD THIS: IT PREVENTS POSTGRES FROM STARTING.
maintenance_work_mem = 256MB
autovacuum = off

I got the north-america-latest.osm.pbf, and you have to remember to chown its containing folder and itself to the user you created for tile-building. Then you su to that user and run the import command. That took 25 hours.

Starting renderd worked, and I was able to view specific tiles through Apache, after getting the paths of those tiles from GN; there's no default interface that allows you to browse tiles (there used to be, apparently).

The next stage is getting renderd to start automatically; the original instructions predate systemd, so that's potentially problematic. We've shelved that for the moment.

The next stage is to download the generate_tiles.py file from GitHub:

As the maptiler user, in the maptiler user's home directory:

curl https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py > generate_tiles.py
chmod u+x generate_tiles.py

Then you need to edit generate_tiles.py. You can comment out all the German cities at the end, but use one of them as a model for your own bounding box and render instruction. Create the bounding box (bottom-left, top-right) using this:

http://hcmc.uvic.ca/people/greg/mapnik/bbox.html

To avoid rendering the whole world in too much detail, change this line:

render_tiles(bbox, mapfile, tile_dir, 0, 5, "World")

to this:

render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")

You'll also need to change the path to the stylesheet:

mapfile = "/usr/share/openstreetmap-carto/style.xml"

Now when you try to run the script, you'll get this error:

RuntimeError: Failed to find font face 'Arundina Sans Italic' in FontSet 'fontset-0' in FontSet at line 17 of '/usr/share/openstreetmap-carto/style.xml'

So fonts required by the stylesheet are missing. These are my attempts to resolve that problem:

The Arundina fonts package is already installed by default, so that's a puzzle. I tried this:

sudo apt-get install latex-fonts-sipa-arundina

That put some fonts there, but they have the wrong names by default. So in this file:

/usr/share/openstreetmap-carto/style.xml

replace "Arundina Sans Italic" with "Arundina Italic".

Next is a missing Tamil font, Paranar:

sudo apt-get install fonts-taml-tscu

and so on for a range of other fonts:

sudo apt-get install fonts-knda fonts-droid-fallback fonts-noto fonts-sil-padauk fonts-indic fonts-tibetan-machine

However, there are still some fonts which are not available, but are references in the style.xml file. For the remainder, where I was able to find a good analogue or close option, I substituted it; otherwise, I commented out the line in the file which referenced it.

For finding out which font names python knows about, this helps:

from mapnik import *
for face in FontEngine.face_names():
  print face