Landscapes of Injustice

Long meeting with the GIS cluster this morning. Arising out of it, for me:

• I'll add a new spreadsheet build task to the ant build.
• It will build a spreadsheet for every year between (say) 1930 and 1950.
• The spreadsheet will have a row for every property.
• One column will be the property description, in a format to be specified by the GIS folks.
• One column will be the title code of the last title applying to the property prior to the year concerned (i.e. the first going back in time from yyyy-01-01).
• One column will be the date of the title.
• Subsequent columns will be booleans for ethnicity of the owners and institutional status of owners.
• A final column, not expected to be used, will be the consideration.

I need to have this working within about ten days.

In addition to fixing a number of minor cosmetic bugs in the spreadsheet rendering as I came across them, I've added the following features today after long discussion with JSR and AM (all morning, in fact):

1. Where the previous title is not nominal, the value for identity of properties with the new title is now output as part of the previous non-nominal title data; it wasn't included before.
2. A new column calculates the difference between the original purchase price and the new purchase price, in 2014 dollars.
3. A second new column calculates that differential as a percentage of the original price, showing the percentage change in value.
4. A final experimental column provides a formula-based readout of the key data involved in a form which makes for easy checking in the database. This works OK in LibreOffice Calc, but I don't know whether it will work in Excel, when it's coming as part of a TSV file.

I've checked a random selection of about ten rows so far, and it looks like all the data is correct, but I want to do much more checking.

The Landscapes build process on Jenkins suddenly started to fail because it couldn't reliably pull down the landscapes_live_latest.xml file from where Peach dumps it every night in my home1t/mholmes/www folder. This is apparently due to a global issue with LDAP, which is having problems resolving permissions for all of our paths with tildes in them.

As a temporary measure, and a test to see if it's robust, I've changed the cron task script on Peach so that it now stores the file in www/public, which is accessible on my /people/martin url, and two tests have now succeeded using that URL. I've also increased the number of retries to 10 (attribute on the ant <get> task), which may have helped. We'll watch this over the next little while and see how it works.

I've fixed the bug that was causing the failure to retrieve previous title information for a subset of the rows. It was a logic error, entirely my fault.

I've added the feature that you requested, whereby if there is no need for a non-nom previous title because the actual preceding title in the chain is non-nominal, the data for the actual preceding title is duplicated in the non-nom title columns.

In the process of investigating, we found a minor problem with one of the records I was using for testing: one of the properties linked in one of the titles was not the one that should have been linked. Actually it was one of the old properties from an early incarnation of the database. Hopefully this was a one-off error, but it might be worth thinking about whether we could possibly write a diagnostic that would find possible other instances of this same problem.

The spreadsheet will be rebuilding itself now and will be ready in a few minutes. The fix for the data error will take until tomorrow to filter through the system into the spreadsheet, because the Jenkins build just gets access to a fresh copy of the database dump once a day (just after midnight).

I've also added a lot of explanatory comments to the file, and I'll try to figure out a way to output those as a piece of documentation.

Description of the problem:

Here's one case which suggests we might be missing some previous titles. Take a look at this series of title ids in the database:

524,635,749,842,3358

Concentrating for the moment on Lot H (D:196 P: 02640 B:00037 S:000 L:00H), which is property id 553, we see a sequence of titles:

842 (1912)
749 (1922)
635 (1939)
524 (1941)

Now, the preceding title for 635 should be 1922, but that has no consideration, so we would expect it to go back to the 1912 title. But in the spreadsheet, no previous title (non-nominal or otherwise) shows up. Why is this? The only reason I can imagine is that the 1912 title has no date. It has a transfer date, but no actual date. That takes it out of the sequence, and it's possible that's the problem, although in theory my code should be using the transfer date instead in this case.

When there's no title date, we are taking the transfer date instead in the spreadsheet we output. My code is also supposed to be using the transfer date in preference to an empty title date when constructing the sequence chains, but it seems to either be failing to retrieve this one, or getting it in the wrong place in the chain.

The same is the case for the other property on 635, lot G. So I think something is wrong here.

Approach to a solution: add debugging xsl:message output which is triggered only when the current title id is 635.

Have conducted an email correspondence with SK of NNM and then travelled to Burnaby on Friday for in-person meeting and to pick up various materials (hard drive, Mizuta docs), and to draft year 3 plans. We didn't understand we were also to do the budget numbers, so I drafted some of those and have sent it to SK for her to complete parts I couldn't do.

The NNM defines "digitizing" as creating image of document or artefact, creating record with metadata for document/artefact and reference to image. The project will likely require in addition a plain-text transcription and then an XML document including marked-up plain text, metadata and reference to image.

Our priorities for content are materials dealing with dispossession, secondarily materials dealing with movement of people (e.g. internment, removal to Japan):
- Hidaka (citizenship defense committee) fond but not items NNM digitized [secondary]
- Mizuta (immigrants to Steveston) maps and lists [secondary]
- Kagetsu (lumber, real estate baron) [primary]
- Morishita (family business) requires Meiji era Japanese [primary]
- Sato fond (Japanese Language School) requires Meiji era Japanese [primary]
- Maikawa (department stores) appears to be largely images [primary]
- Kimura (fishing-related businesses) requires Meiji era Japanese [possibly secondary]
- various small collections (Ishikawa 1 list, Buyemon 33 letters, Nishihata correspondence, Redress re Maikawa, Ishikawa letters, Saito letters) [primary]

Have done initial estimate of dividing labour amongst these tasks (2 grads: 1 in Bby, 1 in Vic; 3 undergrads: 2 in Bby, 1 in Vic) based on research from last year and submitted to SK for review.

Skype meeting notes 2016-02-12

M (SFU), consulting with WA and TW will work on guidelines for us based on SFU's excellent example here:

http://www.sfu.ca/archives2/PDFs/GDL039.pdf

When those are done, WA and TW will test them out by re-editing the existing records currently in AtoM to conform with the Guidelines.

Following that, MH will work with WA and TW to create a spreadsheet template with lots of examples, designed to reduce the degree of duplicated effort involved in manual entry into AtoM (the same metadata required at multiple levels, for example).

Testing of the spreadsheet will be undertaken with trial data and a sacrificial test repository.

During the testing, MH will develop a method of validating a spreadsheet to confirm that it's complete and correct before it's imported into AtoM. (For CCAP, we did this with Python, but Python's CSV parser is a bit flaky so we'll probably use an approach involving XML and proper validation using a schema and Schematron).

All of this needs to be in place and tested by the middle of April, so we're ready to teach it to new RAs at the end of the month.

WA and TW reported problems creating and editing records under the new Thomas Fisher repository over the weekend. Yesterday and today I did some testing, and it appears to be the case that if you don't properly configure the repository (providing a good identifier and name), and/or you don't provide a General Material Designation and/or an identifier for an archival description, you can't save the latter successfully. This has prompted a discussion on the need for us now to formalize our Guidelines and instructions for entering data, which we will do in the next month or so.

After yesterday's detailed discussion of how to handle transaction chains with nominal transactions in the spreadsheet generation, I started work on implementing it. The spreadsheet XSLT had grown by accretion and become horribly unmaintainable, so I've refactored and modularized it, added a function for detecting non-nominals, and bugfixed the results; I'm now ready to add the previous-non-nominal functionality, which should be fairly straightforward now.

SA and I worked on fleshing out the proposal, and wrote to DG (AtoM) with a list of detailed questions about development plans and prospects. We got a reply back at the end of the day. Much more discussion will arise out of that, best kept off the blog for the moment.