Landscapes of Injustice

Spent the morning learning the best ways to create nice app-like interfaces for shell scripts on the three platforms.

Windows:

Used "Bat To Exe Converter" to create an exe: http://www.f2ko.de/programs.php?lang=en&pid=b2e

Mac:

Used Platypus. Found a number of things I had to work around to get it going properly. Tested on a couple of other Macs.

Linux:

Settled on a solution where the first time the script is run (or whenever it's run without a folder parameter), a .desktop file is generated, pointing to the path of the script (which can't be relative in a .desktop file). This seems to work OK. Parameters are passed to the script automatically.

This is now ready for testing on any platform.

JS-R requested a simple method of turning the hi-res scans or photos that the field workers will be creating into manageable PDFs. This is doable with ImageMagick and GhostScript, so I've written and tested a simple script that does the job on Linux, Mac and Windows. The Windows version will be needed first, so that has more robustitude at the moment; the other versions need some extra quotes etc. inserted to handle spaces in pathnames more reliably. More error-trapping could be put in, and I'll hammer at it a bit more over the next little while, but the Windows version is ready for field testing. It's in the Landscapes svn repo along with some test data.

After some research and testing I ended up settling on this command for building the PDF:

convert *.jpg -interlace Plane -gaussian-blur 0.05 -compress jpeg -quality 85 ${FNAME}

This seems to give a good combination of size and quality with page-images. May tweak it some more at some point.

Ultimately I'd like to make this a drag-drop sort of widget, so you drag a folder onto it, but that'll have to be done in very different ways for each platform. On Windows, oddly enough, it JUST WORKS -- you drop a folder on the batch file and it runs with the dropped folder as a param. GN suggests using Platypus to create a .app for Mac.

It was interesting/frustrating to get back to writing Windows batch files again after writing Bash files. You stumble over the simplest things.

We will be meeting with the Land Title Office staff, so we met today to prepare a list of questions relating to issues around granularity of properties, co-ownership, and various other thorny problems arising out of the data we already have.

We also talked about the handling of documentary data for the project as a whole, and our clusters in particular. The resulting plan is to get an unlimited account on Zotero for the first year, and see how that suits our needs. We will create a protocol which re-purposes some of the existing Zotero fields we don't need for things we do require that aren't covered (since Zotero doesn't allow custom fields). We will also put in place XML processing for XML output dumps of the Zotero db so that we can run validation and consistency checks on the data, and ensure errors are corrected. Finally, JS-R's existing document database will be exported as XML and turned into a format that Zotero can ingest; this will involve some detailed processing of citation information to create filepath-like string values representing the exact location of a document (archive/collection/box/file etc.). Using this system we will be able to reconstruct a tree-like view of the documents in their archives down the road.

I've got the whole tree we worked out on the whiteboard converted to an ODD file, with some changes to keep element names unique and rationalize some things; I have new data types of various kinds, and I have a test file which is working well. The process of looking at the real data has raised a number of very complicated issues -- it turns out that partial lots can be included in titles, and titles can combine lots and partial lots from disparate blocks; e.g. title id 203 from the existing database contains 8 lots and one partial lot, from four distinct blocks. Combine this with a variety of semi-documented ownership splits and you have such a potential source of inaccuracy and confusion that a very clear policy on handling this sort of thing must be established in advance.

One outstanding issue for my ODD: should I combine all my customizations into their own distinct module? I think probably yes, but our documentation says nothing about this. When I've done with this ODD file, I'll use it as an example for the "Getting started with P5 ODDs" document I'm supposed to be working on, and I think it should also be built into TEIP5-Test as a validation test (today it uncovered a problem with handling of multiple constraints in a constraintSpec which took some sorting out across the two Jinks boxes).

The ODD file is coming along slowly. I hit a bug in the current ODD schema, and another in the Jenkins build, and fixed the latter to get a stable schema I could validate my ODD file against.

Very productive meeting with RRR and DBM on expectations and plans for the GIS cluster. The basic outcome is that we need to start by defining the exact boundaries of the areas we're focusing on (JSR should do that); then they will contact the UBC library to find out exactly which fire insurance maps are available and how accessible they are (digitized or not?), and enumerate other resources. The basic plan is to start with a modern streetmap as a base layer, then create polygons which are linked to a database of properties and a separate database of addresses; these would most likely be set up by us, and populated (at least initially) from the Properties db info we have, then supplemented with info from the city directories. The objective is to have a polygon for every incarnation of every property during the period in question, linked to any addresses and property descriptions it may have had. Properties are then linked to titles and owners, etc., as covered by our other developing plans (the XML schema among other things).

Very useful meeting with members of the Oral History cluster. In the follow-up, we've learned about their media digitization standards, the XML format they're working with for metadata and transcription, and data storage and replication plans. We'll want to set up data replication processes both to provide extra backup for them and to ensure easy access to data locally.

I've started mapping out XML structures for the properties and all their contents, and sent some queries to JS-R. Looks like a TEI extension with a lot of specialist elements and attributes is what we need, and I've started an ODD file.

It seems that video conferencing will be done through BlueJeans, so SA and I tested this out; there's a simple deb for Linux, and it works fine in Firefox, with around a half-second delay between our desks. UVic supports it.

Met with JS-R and SA to discuss the hardware and software infrastructure to support each of the research clusters. Subsequent discussion with GN. Points arising:

• We will create a VM for the Landscapes project, running MariaDB, PHP, and ultimately Tomcat 8 with eXist. It will mount around 4TB of disc space.
• The Adaptive DB code will be updated to handle binary doc uploads.
• We will arrange with tech-savvy nodes for them to replicate our data probably through rsync cron jobs.
• Sysadmin will administer the VM for the usual cost.
• We will consult with individual research clusters asap to get a basic idea of their needs and skill levels with regard to data collection and entry.
• We will put together a presentation for the meeting in May (20th-22nd or thereabouts) detailing our plans so far, and all the possible options, so they can all see their own data collection and storage in the context of what others have asked for or may need. This will allow us to revisit preliminary decisions.
• Basic data collection systems will be in place for testing by around the middle of June, and data collection should start by the beginning of July.
• The land title work headed by JS-R will use a simplified version of the existing Properties db for data collection, but querying will be done on a generated XML version which is optimized for the sorts of queries we want to do, and which will be based around the property rather than the title; this will have massive data duplication to speed queries, so that we can avoid lookups and join-type operations in the interests of speed.