Maintenance

I've been gradually setting permissions throughout the folder hierarchy as they turn out to be necessary, and I think we've now reached a situation where WP is working, and the remaining problems are due to paths set up to point locally in the original install. BS is working on that now.

I always have to look this up, so...

Hypothetically, you have a development server that is only accessible on-campus, from a specific subnet. But you want to use/view the app from some off-campus location.

SSH to the rescue:

ssh -nNT -L 8888:firewalled.server.uvic.ca:80 someuser@another.machine.uvic.ca

Then, go your browser and head to http://localhost:8888 and see the firewalled page!

The explanation:
flags -nNT > 'n' basically sends input to /dev/null, 'N' restricts the connection to just tunnelling, and 'T' stops the remote machine giving you a pseudo-TTY.

8888: > is the arbitrary port we'll use for the tunnelling

firewalled.server.uvic.ca > the server we want to access

:80 > the remote port you want to access. In this case it's a web server, so port 80

someuser@another.machine.uvic.ca > this connects a user to another remote machine that you have SSH access to, AND is allowed to access firewalled.server.uvic.ca on port 80.

Ran through a few iterations of the build script and fixed a number of bugs; I've now reached a point where all my previous code to configure Jenkins is based on the assumption that Jenkins starts up functioning normally after install, as it used to, but the behaviour has changed and there's now a randomly-created user name and password that the administrator has to use to log in from the beginning. I will need to figure out how to get around this, but it is a good feature in itself.

This process resulted in UEFI secure boot being turned off, because apparently I'm running some driver or other that won't work with it, but other than that all seems well.

Note to self, very handy thing:

avahi-browse -vatr

ST has supplied a bounding box for her Stolo project. Unfortunately it's huge; but I rendered it with zoom levels 10 through 16, and it took only a few hours, generating 47,762 tiles. That may not be enough detail, though. Relevant segment from /home/maptiler/stolo/generate_tiles.py:

    render_tiles(bbox, mapfile, tile_dir, 0, 2, "World")

    minZoom = 10
    maxZoom = 16

    stolo = (-122.508545,48.789676, -121.322021,49.378797)
    render_tiles(stolo, mapfile, tile_dir, minZoom, maxZoom)

I've now set it going again (using screen) to render the next two zoom levels (17 and 18), and we'll see how long that takes.

For the TCCD project, I really would like to get a Jenkins job building things, but the problem is that everything is on GitHub and the repo is enormous due to the huge number of images. Partial checkouts with git are tricky. However, by looking at this documentation, I've figured out and tested a way to script a partial checkout using svn instead.

Say you want only the [proj]/level1/level2/level3 directory:

svn co -N https://github.com/[proj] proj
cd proj
svn up -N trunk
svn up -N trunk/level1
svn up -N trunk/level1/level2
svn up -N trunk/level1/level2/level3

That will get you a sparse checkout down to the contents of the level3 directory.

I have a little workgroup of TEI folks helping with an update/rewrite of the Jenkins server building script from 2012, ready for Ubuntu 16.04. I've now worked through about two thirds of the script, updating, fixing and tweaking, and my test setup is working well. The only gnarly problem so far is an issue with Japanese fonts, and I'm waiting for some LATEX expertise to help me resolve that one.

Set up the machine

1. Boot VM and do a regular install.
2. Update the VM: 'sudo apt-get update && sudo apt-get -y dist-upgrade'
3. Install required packages: 'sudo apt-get -y install apache2 apache2-dev libmapnik3.0 mapnik-utils mapnik-vector-tile openstreetmap-carto osm2pgsql python-mapnik dh-autoreconf node-carto'

Tweak OS

1. Open postgres config file: 'sudo nano /etc/postgresql/9.5/main/postgresql.conf'
2. Uncomment the line: '#maintenance_work_mem = 64MB'
3. and change memory allocation to 256MB: 'maintenance_work_mem = 256MB'
4. Save and exit
5. Open system control config: 'sudo nano /etc/sysctl.conf'
6. Add this to top of file: 'kernel.shmmax=268435456'
7. Reboot machine and run 'sudo sysctl kernel.shmmax'
8. The result should reflect the change: 268435456

1. 'sudo visudo'
2. add this to end of file: '{your-username} ALL=(ALL) NOPASSWD:ALL'

Prepare database

1. Become postgres user: 'sudo -u postgres -i'
2. Create a postgres user for our OSM tasks: 'createuser osmdbadmin'
3. Invoke postgres cli: 'psl'
4. The installation of mapnik et al. should have created a database called 'gis'. Let's check: '\c gis'
5. You should now have a command prompt that looks like this: 'gis=#'
6. Create the postgis extension: 'CREATE EXTENSION postgis;'
7. Change ownership of geometry table: 'ALTER TABLE geometry_columns OWNER TO osmdbadmin;'
8. Change ownership of spatial ref table: 'ALTER TABLE spatial_ref_sys OWNER TO osmdbadmin;'
9. Back out of psql: '\q'

Add data to database

1. Create a directory for our download: 'sudo mkdir -p /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia'
2. Download the data: 'sudo wget --directory-prefix=/usr/share/openstreetmap-carto/data/north-america/canada/british-columbia https://s3.amazonaws.com/metro-extracts.mapzen.com/victoria_canada.osm.pbf'
3. Become postgres user to import data: 'sudo -u postgres -i'
4. Import using osm2pgsql: 'osm2pgsql --slim -d gis -C 16000 --number-processes 3 /usr/share/openstreetmap-carto/data/north-america/canada/british-columbia/victoria_canada.osm.pbf'

Set up mod_tile/renderd

1. Check out the code to your homedir: 'cd ~ && git clone git://github.com/openstreetmap/mod_tile.git'
2. Start building it: 'cd mod_tile && ./autogen.sh'
3. Configure: './configure'
4. Make and install: 'make && sudo make install'
5. Install mod_tile itself: 'sudo make install-mod_tile && sudo ldconfig'
6. There are apache setup instructions here if you want to set that end of things up.

Configure renderd

1. Open conf file for editing: 'sudo nano /usr/local/etc/renderd.conf'
2. Uncomment this line at the top of the file: 'socketname=/var/run/renderd/renderd.sock'
3. In the [default] section, change the XML line to look like this: 'XML=/usr/share/openstreetmap-carto/style.xml'

Set up manual tile generator

1. This script must be run by a user that has access to the db. I created an OS user that matches the db admin name: 'sudo adduser osmdbadmin'
2. Switch to osmdbadmin user: 'su osmdbadmin'
3. Download the script: 'cd ~ && wget https://raw.githubusercontent.com/openstreetmap/mapnik-stylesheets/master/generate_tiles.py'
4. Edit for opening: 'nano ~/generate_tiles.py'
5. Make it executable: 'chmod +x ~/generate_tiles.py'
6. Change stylesheet pointer (around line 198) to: '/usr/share/openstreetmap-carto/style.xml'
7. Remove everything at the end that references 'World' and German cities (lines starting with 'bbox' and 'render_tiles').
8. Add a stanza that looks like this:
   # University of Victoria
   bbox = (-123.4065,48.4150, -123.2145,48.5051)
   render_tiles(bbox, mapfile, tile_dir, 14, 18 , "UVic")
   
9. Save and close the file.
10. You should now be able to run the generator: 'cd ~ && ./generate_tiles.py'
11. Read notes on errors, below.

Notes on generate_tiles.py errors

#!/bin/bash
sudo sed -i 's|<Font face-name="Arundina Sans Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Arundina Sans Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Droid Sans Fallback Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="gargi Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige Normal"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mallige NormalItalic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Mukti Narrow Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Tibetan Machine Uni Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Bold"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Italic"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="TSCu_Paranar Regular"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="Unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|<Font face-name="unifont Medium"/>||g' /usr/share/openstreetmap-carto/style.xml

sudo sed -i 's|minzoom|minimum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml
sudo sed -i 's|maxzoom|maximum-scale-denominator|g' /usr/share/openstreetmap-carto/style.xml

The import of the North America data into my VM took a total of 25 hours, but seems to have completed successfully. Yay. Next step, tomorrow, trying to build some actual tiles.