Note to self, very handy thing:
avahi-browse -vatr
Note to self, very handy thing:
avahi-browse -vatr
ST has supplied a bounding box for her Stolo project. Unfortunately it's huge; but I rendered it with zoom levels 10 through 16, and it took only a few hours, generating 47,762 tiles. That may not be enough detail, though. Relevant segment from /home/maptiler/stolo/generate_tiles.py
:
render_tiles(bbox, mapfile, tile_dir, 0, 2, "World") minZoom = 10 maxZoom = 16 stolo = (-122.508545,48.789676, -121.322021,49.378797) render_tiles(stolo, mapfile, tile_dir, minZoom, maxZoom)
I've now set it going again (using screen) to render the next two zoom levels (17 and 18), and we'll see how long that takes.
For the TCCD project, I really would like to get a Jenkins job building things, but the problem is that everything is on GitHub and the repo is enormous due to the huge number of images. Partial checkouts with git are tricky. However, by looking at this documentation, I've figured out and tested a way to script a partial checkout using svn instead.
Say you want only the [proj]/level1/level2/level3 directory:
svn co -N https://github.com/[proj] proj cd proj svn up -N trunk svn up -N trunk/level1 svn up -N trunk/level1/level2 svn up -N trunk/level1/level2/level3
That will get you a sparse checkout down to the contents of the level3 directory.
I have a little workgroup of TEI folks helping with an update/rewrite of the Jenkins server building script from 2012, ready for Ubuntu 16.04. I've now worked through about two thirds of the script, updating, fixing and tweaking, and my test setup is working well. The only gnarly problem so far is an issue with Japanese fonts, and I'm waiting for some LATEX expertise to help me resolve that one.
Set up the machine
I'm using VirtualBox on linux. The VM is configured with 16GB RAM, 200GB virtual HDD, and bridged networking. I'm installing Ubuntu Server 16.04 (currently in beta, so I get it from here.Tweak OS
As this is going to be doing some pretty heavy lifting, you'll want to tweak the way the system handles DB work. We'll tweak Postgres and the kernel to do this.Prepare database
Add data to database
We need some data to put in the database. We'll use the metropolitan Victoria area from mapzen:Set up mod_tile/renderd
Configure renderd
Renderd is a daemon that mod_tile uses for the rendering of new tiles. If you'll be using the full functionality of this stack you will need to follow the instructions here.Set up manual tile generator
This allows you to explicitly construct a complete tile set without using apache/mod_tile. It uses a python script called 'generate_tiles.py'.Notes on generate_tiles.py errors
The stock stylesheet references a bunch of fonts that are not installed by default. I have heavy-handedly removed these references with a script, but Martin has made an effort to find and install the right fonts, or reasonable alternates. Take a look at his post for progress. Other than nuking the missing fonts, my script also address an annoying warning that you get about deprecated code. Here's the script:#!/bin/bash sudo sed -i 's|<Font face-name="Arundina Sans Bold"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Arundina Sans Italic"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Arundina Sans Regular"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Droid Sans Fallback Regular"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="gargi Medium"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Mallige Bold"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Mallige Normal"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Mallige NormalItalic"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Mukti Narrow Bold"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Mukti Narrow Regular"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Tibetan Machine Uni Regular"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="TSCu_Paranar Bold"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="TSCu_Paranar Italic"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="TSCu_Paranar Regular"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="Unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|<Font face-name="unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|minzoom|minimum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml sudo sed -i 's|maxzoom|maximum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml
The import of the North America data into my VM took a total of 25 hours, but seems to have completed successfully. Yay. Next step, tomorrow, trying to build some actual tiles.
Back in 2012, I developed a script for automatically building a TEI Jenkins server, and it was used to create the Jenkins server which is the current TEI build server. That server has now been substantially updated, and we have a TEI working group to write a new script based on Ubuntu 16.04, and to include the more complex Jenkins jobs that we now run.
This stuff is being developed in TEIC/Jenkins on GitHub, and I've updated some of my original scripts which help to automate the generation and configuration of test VMs for running the script. I now have a generic PreJenkins machine, as before, and I can build and run a clone from it. I'm up to the point where everything looks good and the builder script is ready to start installing stuff. At this point, I'll wait for confirmation from the rest of the WG on a couple of issues I've raised (which java we should use, and whether we should install the TEI packages).
Note: this post covers several days, and I expect to keep updating it as we learn more.
I'm attempting to replicate, with documentation, Greg's building of an OSM tile-rendering machine, starting from instructions here:
https://switch2osm.org/serving-tiles/manually-building-a-tile-server-14-04/
but using 16.04 beta 2. I first built a VM from Ubuntu 16.04 server beta2, with 16GB RAM and 200GB of disk space, adding only the install of ssh-server at install time, and selecting the non-LVM full-disk option. These are deviations that I had to deal with:
In the first set of software installs:
libtiff4-dev needs to be changed to libtiff5-dev
Then I did followed the instructions for installing and configuring postgres although the package version was different (do apt-cache search to find the current one). When I got to osm2pgsql, I just did
sudo apt-get install osm2pgsql
and then for mapnik:
sudo apt-get install libmapnik-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto python-mapnik python3-mapnik
At some point, the node-carto package got installed; that's a good thing, but I don't know how it happened.
Next, I had chosen to use openstreetmap-carto instead of OSMBright, so I didn't need to do that bit.
The next stumbling block was that when you edit the renderd.conf file, the paths proposed are not correct, because things are in different places due to installing from the repos rather than building locally. You want this:
socketname=/var/run/renderd/renderd.sock plugins_dir=/usr/lib/mapnik/3.0/input font_dir=/usr/share/fonts/truetype/ttf-dejavu XML=/usr/share/openstreetmap-carto/style.xml HOST=localhost
When you come to tuning postgres, only three of the four settings can be found. Don't add the missing setting, just set the others:
shared_buffers = 128MB #checkpoint_segments = 20 ##DO NOT ADD THIS: IT PREVENTS POSTGRES FROM STARTING. maintenance_work_mem = 256MB autovacuum = off
I got the north-america-latest.osm.pbf, and you have to remember to chown its containing folder and itself to the user you created for tile-building. Then you su to that user and run the import command. That took 25 hours.
Starting renderd worked, and I was able to view specific tiles through Apache, after getting the paths of those tiles from GN; there's no default interface that allows you to browse tiles (there used to be, apparently).
The next stage is getting renderd to start automatically; the original instructions predate systemd, so that's potentially problematic. We've shelved that for the moment.
The next stage is to download the generate_tiles.py file from GitHub:
As the maptiler user, in the maptiler user's home directory:
curl https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py > generate_tiles.py chmod u+x generate_tiles.py
Then you need to edit generate_tiles.py. You can comment out all the German cities at the end, but use one of them as a model for your own bounding box and render instruction. Create the bounding box (bottom-left, top-right) using this:
http://hcmc.uvic.ca/people/greg/mapnik/bbox.html
To avoid rendering the whole world in too much detail, change this line:
render_tiles(bbox, mapfile, tile_dir, 0, 5, "World")
to this:
render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")
You'll also need to change the path to the stylesheet:
mapfile = "/usr/share/openstreetmap-carto/style.xml"
Now when you try to run the script, you'll get this error:
RuntimeError: Failed to find font face 'Arundina Sans Italic' in FontSet 'fontset-0' in FontSet at line 17 of '/usr/share/openstreetmap-carto/style.xml'
So fonts required by the stylesheet are missing. These are my attempts to resolve that problem:
The Arundina fonts package is already installed by default, so that's a puzzle. I tried this:
sudo apt-get install latex-fonts-sipa-arundina
That put some fonts there, but they have the wrong names by default. So in this file:
/usr/share/openstreetmap-carto/style.xml
replace "Arundina Sans Italic" with "Arundina Italic".
Next is a missing Tamil font, Paranar:
sudo apt-get install fonts-taml-tscu
and so on for a range of other fonts:
sudo apt-get install fonts-knda fonts-droid-fallback fonts-noto fonts-sil-padauk fonts-indic fonts-tibetan-machine
However, there are still some fonts which are not available, but are references in the style.xml file. For the remainder, where I was able to find a good analogue or close option, I substituted it; otherwise, I commented out the line in the file which referenced it.
For finding out which font names python knows about, this helps:
from mapnik import * for face in FontEngine.face_names(): print face
Out of curiosity, I just tested WordPress 4.4.2 with the SQLite plugin.
Including download time, it took less than 5 minutes to get it working. Instructions are simple and easy to follow.
For fun, I also installed phpLiteAdmin, which allows phpMyAdmin-like access to your DB. Installing/configuring this added about 1 minute to my total investment of time.
The advantage is that everything you need to back up is in one folder. There is no daemon for SQLite, so there *should* be a reduction in memory usage too.
Links:
WordPress: https://wordpress.org/download/
WP SQlite plugin: https://wordpress.org/plugins/sqlite-integration/installation/
phpLiteAdmin: https://www.phpliteadmin.org/download/
Based on these instructions and the bit on pre-rendering tiles in this set of instructions, along with and the set of already-available docker recipes, it seems like we may be able to get one or more docker instances running that would build specified tilesets for us (pre-rendering them).