Category: Documentation

22/06/17

Permalink 04:58:27 pm, by mholmes, 333 words, 26 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation, Documentation; Mins. worked: 90

Building a vector tile server

Now that Open Layers fully supports vector layers, we're looking at the practicality of running a vector tile server for our projects. Starting from this docker example, I created a script which I can run on a standard Debian Stretch install to create a working tile server:

#!/bin/bash

#This is to be run on a standard Debian Stretch install.

#Install core stuff
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install apt-transport-https curl unzip build-essential python libcairo2-dev libprotobuf-dev xvfb 

#Temporarily use a specific source for the exact nodejs version we need.
echo "deb https://deb.nodesource.com/node_4.x jessie main" >> /etc/apt/sources.list.d/nodejs.list
echo "deb-src https://deb.nodesource.com/node_4.x jessie main" >> /etc/apt/sources.list.d/nodejs.list

#Install it.
apt-get -qq update && DEBIAN_FRONTEND=noninteractive apt-get -y --allow-unauthenticated install nodejs 

#Now remove the source.
rm /etc/apt/sources.list.d/nodejs.list
apt-get clean

#Added these in order to get the npm install to run properly. 
#The problem was building canvas.
apt-get -y install libjpeg62-turbo-dev libpango1.0-dev libgif-dev g++

#Create directory for tileserver application.
mkdir -p /usr/src/app

#Get the Klokantech code for the server.
cd /usr/src/app 
curl -L https://github.com/klokantech/tileserver-gl/archive/master.zip -o master.zip
unzip master.zip 
cp -r tileserver-gl-master/* ./
rm -rf tileserver-gl-master

#Install the node stuff
npm install --production

#Set environment variable
echo NODE_ENV=\"production\" >> /etc/environment

#Create the folder for the mbtiles files (you'll need to supply these later).
mkdir /data

#In case other servers are installed and running, stop them.
systemctl stop apache2 mysql
systemctl disable apache2 mysql

echo "Now put your mbtiles files into the /data folder, and run /usr/src/app/run.sh."
#Start the tileserver on port 80.
#run.sh

This could form the basis for a VM-based tileserver for our projects, including the Confederation Debates; running a server for all of Canada is quite practical due to the efficiency of the vector format.

01/05/17

Permalink 09:05:19 am, by mholmes, 77 words, 20 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 30

Upgrading teiJenkins java

The upgrade for Jenkins on teiJenkins was being kept back, and it turned out this was because Ubuntu 14.04 has Java 7 by default. I added a PPA for Java 8, updated the alternatives (sudo update-alternatives --config java) to point to the new one, and was then able to install Java 8. Following that, the Jenkins update went ahead. I elected to keep my existing config for Jenkins rather than overwrite. It needed a reboot for Apache to find Jenkins again.

Permalink 08:33:12 am, by mholmes, 23 words, 20 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 20

Extended partition on jenkins.hcmc.uvic.ca

RE provided new space to double the available drive space; followed my own instructions here to extend the partition. No problems at all.

20/04/17

Permalink 05:02:07 pm, by mholmes, 132 words, 23 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 90

How to deploy a new XAR on Jettys

Today I blew up a couple of the apps and had to restart them, through doing this the wrong way. When you have a new XAR to deploy:

  1. Use Chrom*, not FF.
  2. Connect over the internal URL on :8080.
  3. Upload the new package.
  4. If it goes wrong and you see an error message, the chances are the db is now set to read-only.
  5. If that happens, try shutting down the db from the web interface. If that works, restart it from /etc/init.d/jetty. If it fails, you may need to kill all the relevant processes on Peach before restarting.

With these big XARs, we may need to consider testing an alternative process where we uninstall the old XAR and then put the new one in the autodeploy folder before restarting eXist.

20/03/17

Permalink 05:21:32 pm, by mholmes, 25 words, 76 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 30

eXist deployment: tested development branch

Tested a build of the dev branch with my script and deployment stuff locally; all good, and the bug with the java client is fixed.

20/01/17

Permalink 02:46:13 pm, by mholmes, 79 words, 56 views   English (CA)
Categories: Servers, R & D, Activity log, Documentation; Mins. worked: 45

Side-effects of the new site launch

URLs containing "editor" are all being redirected to the HCMC site, even though only the www/editor URLs were supposed to be; that borks our adaptiveDB projects, so I've written to pts to see if we can get it fixed. MS's two projects Bilibin and St Pete were also screwed up, being in a "projects" folder, but in that case, I've just moved the projects up one level to the hcmc/www, and we'll cope with the changed URLs.

10/11/16

Permalink 09:35:31 am, by mholmes, 175 words, 55 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 60

Configuring eXist/Jetty for single app deployment

As we move towards deploying standalone eXist/Jetty applications for our projects, we're figuring out how best to configure them. One issue is that we're probably going to want to point the subdomain (graves.uvic.ca or whatever) at the /apps/graves/ subfolder, but we're still going to need access to some of the default eXist applications such as eXide and the dashboard. This can be accomplished by adding the following line to the webapp/WEB-INF/controller-config.xml file:

<root pattern="/apps/graves/apps" path="xmldb:exist:///db/apps/"/>

Add it as the first entry before similar entries. The effect of this is to leave all the existing graves app functionality handled by the apps/graves/controller.xql, but hand anything accessed on /apps/graves/apps to the appropriate app controller. My testing with eXist 3 RC1 confirms that this works; it should mean that on going live, the dashboard (for instance) would be accessible on graves.uvic.ca/apps/dashboard (and we can access it over TLS for better security when logging in).

22/09/16

Permalink 03:46:44 pm, by mholmes, 296 words, 166 views   English (CA)
Categories: Servers, Activity log, Activity log, Documentation; Mins. worked: 60

Tomcat being tied up/brought down

MC reported he'd had to restart tomcat-devel several times, so we started looking through logs. I think I've traced the current Tomcat issue to a weird combination of circumstances.

Way back when, the Map of Early Modern London project used to be a PHP app. It had URLs like this:

mapoflondon.uvic.ca/render_page.php?id=LIME1

We turned it into an eXist app, with proper page URLS like this:

mapoflondon.uvic.ca/LIME1.htm

I wrote some cunning logic in the app to detect requests to the old URLs and redirect them to the new. However, the logic never fired in normal circumstances, because Apache on the front-end saw ".php" and passed the page off to the PHP interpreter.

To deal with this, I and someone from Systems cooked up an Apache rewrite rule in the virtual host, which would turn the old URL into the new. However, it's not very sophisticated; it treats search parameters very crudely, so it turns this:

/render_page.php?id=LIME1&title=Lime%20Street/

into this:

/LIME1&title=Lime%20Street/.htm

which throws up errors when eXist sees it, naturally. Now, it turns out that this particular URL, presumably along with some other similar ones, had been links on the old site way back when, and are now in places such as archive.org, and some bot somewhere has been hitting these URLs, and generating a lot of errors.

I think at this point in time it might make sense to revisit that old rewrite rule and make it much simpler, so that it converts anything which contains the string:

"render_page.php"

into

"404.htm"

and doesn't worry about trying to parse it at all. Wrote to MC to see if he thinks that makes sense.

25/07/16

Permalink 03:15:54 pm, by mholmes, 396 words, 130 views   English (CA)
Categories: Servers, R & D, Activity log, Activity log, Documentation, Documentation; Mins. worked: 120

Getting a customized tileset built: hcmc-plain

We started work with a standard checkout of OSM Bright, and we worked against the model of openstreemap-carto-common/style.xml, which we have already successfully used to build tiles. OSM Bright itself cannot be compiled with the carto tool, because it's missing key information.

The sample project is in /home/mholmes/WorkData/maps/osm-bright/hcmc-plain/ on my machine.

First of all, we edited two of the four .mss files, base.mss and palette.mss, renaming them. In this process, we simply commented out a lot of the content, since we're aiming for a land/water/nothing else view. The other two mss files were ignored.

Next, we copied osm-bright.osm2pgsql.mml to hcmc-plain.osm2pgsql.mml, and began editing it. This is the file which is compiled using carto to create the XML styles file which is used by the tile builder. There was a lot to learn:

  • The osm-bright version we were starting from was broken because it was missing a style attribute required on the two Datasource objects: "type": "shape",. Without this, carto fails to process the resulting XML. In this example file, it's on the two initial Datasource elements, which are based on shape files which are downloaded (see previous post).
  • The files in osm-bright/res and osm-bright/img need to be in the same place relative to the .xml styles file you create, so the simplest thing is to place those folders alongside the .xml file when building the tiles.
  • The shape file data needs to be accessible and linked from the .xml file. Since that data is in /usr/share/openstreetmap-carto-common/data on the maptiler server, we simply created a symlink from "data" in the tile-generation location to that folder, and made the link a local one in the mml file ("file": "data/simplified-land-polygons-complete-3857/simplified_land_polygons.shp").
  • The default name for the database in the mml file is "osm"; on my maptiler server, it's "gis", so that needed to be changed: "dbname": "gis" in 28 places.
  • Most layers in the mml file have "status": "on" by default; we turned most of them off, sometimes adding the status attribute when it was missing.

Then we compiled the mml file using carto:

carto hcmc-plain.osm2pgsql.mml > hcmc-plain.xml

Then we copied that file (along with img and res folders) up to the maptiler server, configured generate_tiles.py, and ran it.

27/06/16

Permalink 04:42:21 pm, by mholmes, 197 words, 228 views   English (CA)
Categories: Servers, Activity log, Documentation; Mins. worked: 240

Customizing OSM Bright styles for Mapnik to render

We've determined that it's not practical to use the MapBox Classic editing tool to create styles and then generate tiles from them, because the resulting project.xml file does not contain any of the required SQL for pulling data from the PostgreSQL database. Instead, this is what we need to do:

  • Clone the OSM Bright repo from https://github.com/mapbox/osm-bright.git.
  • Customize the mms files to make the changes we want. There's going to be a certain amount of trial and error here, unfortunately, so a quick-and-dirty small-scale rendering job would be a good testbed.
  • Download the land polygon files from OSM, unzip them and run shapeindex on them as instructed in the readme file in OSM Bright.
  • PROBABLY NOT NEEDED: Create configure.py from configure.py.sample, providing the dbname (which is "gis"); I don't think any other info is needed. Also configure the project name.
  • Run carto osm-bright.osm2pgsql.mml > osm-bright.xml in the osm-bright subfolder. This should create the XML for Mapnik, configured with the correct PostgreSQL select statements.
  • Build some tiles with that on the tile server, using generate_tiles.py.

This is gnarly and untested. Will edit as appropriate.

:: Next Page >>

Maintenance

This blog is the location for all work involving software and hardware maintenance, updates, installs, etc., both routine and urgent, in the server room, the labs and the R&D rooms.

Reports

XML Feeds