Landscapes of Injustice

Lots of work recently on getting the Migration build and schema progressing. Currently, I have a process (in ~/production/utilities/migration) that migrates all of the TEI data from the trunk/ directory, re-organizing and normalizing where possible; this includes:

• The personography is now split into alphabetical files (personography_a, et cetera); kept whole, the personography takes forever to validate on the best of machines, so this will allow better working conditions. Plus, in whatever interface we eventually create, the index of people will necessarily need to be split into alphabetical files for browsing etc.
• The RADish files are now broken into smaller bits, with special processing-instructions in the fonds list that explain the lower-level items that nested within a fonds; this will make working in these files simpler, too.

I've also started work on a new schema. After discussion with GL and SA, I think the best approach will be to create a harmonized schema that's written from scratch but embeds all of the previous work done by MH, SA, and GL on the various cluster schemas.

The approach I've taken is to create a new schema (production/data/schema/LOI.odd) that will now serve as the master ODD file. I've included all the modules we need and am slowly working my way through extending the schema where necessary. There's a few departures from the TEI schema: item can be a root element (see above), list can thus be empty (insofar as it is populated just with processing instructions), address can be mixed content, and that seg/@subtype can be data.enumerated rather than a single token. The last two were additions in the loiDirectories.odd, which I'll need to talk with MH about. Address having mixed content is everywhere in the directories, so I think that's a good rule to retain, but enumerated subtypes only appear a handful of times, so there might be a better solution here.

There's lots left to do here and many decision to be made, but the work is progressing steadily; I've also made a spreadsheet for SA and GL that attempts to account for and organize the various datasets so we can strategize on how to bring everything into coordination.

Met with SA and GL to familiarize myself with the various datasets and to plan out steps going forward. I am going to go through the current data and appraise the current processing, et cetera and come up with a plan for the next steps, which will include reviving the Infrastructure Google Doc (authored by me, SA, and GL) and making suggestions about both the technical infrastructure and the conceptual infrastructure for the data.

The goal here will be to get everything harmonized, documented, and constrained so that the data will all be amenable to conversion to HTML in future.  

Based on further experiments by Lindy, Erin and Lindsey, we've now got the following situation:
Archives:
done so far : ~2000 warning messages
still to do : ~13000 warning messages
going at : ~100 warning messages / day
Therefore, about 130 person-days (13000 warnings / 100 warnings per day) or 900 person-hours to go as of July 22.

Case Files
done so far : 11 reels
still to do : ~120 reels
going at : 5-6 minutes per case file or about 2.5 person-days per reel (to do person, family, gender, citizenship, where sent to, if exiled to Japan, birthday, occupation, employer, home address)
Therefore about 300 person-days (120 reels x 2.5 person-days / reel) or 2000 person-hours to go as of July 22.

Lindsey has 14 days to go so should get through about 1400 warnings.
Lindy and Erin each have 24 days to go so should get through about 10 reels each, 20 reels total.
As of Aug 31
total warnings done = 3500; warnings to go = 11,500 or about 120 person days or 17 person-weeks.
total Case File reels done = 30; Case File reels to go = 100 or about 250 person-days or 35 person-weeks.

Met the rest of the Landscapes team today. Chatted with GL and SA about various plans going forward, which might mean some repository reorganization and the possibility of ODD chaining. Lots of work to do, but I think it will all be worthwhile for future linked data possibilities.

(Posting hours from yesterday). Met with GL to catch up and to discuss future plans. Plotted out some actions and then sent out a few questions regarding the TINA_NNM. 

Time: 90 min

This post details a method used to fix the following problem:

XML files where downloaded through a web browser from revision.hcmc.uvic.ca. They were XML in UTF-8, but Apache didn't know that so it presented them as 8859-1, and the browser happily believed it. All Japanese characters inside the files therefore got borked.

Work proceeded without this being noticed, resulting in a pair of -ography files in the RADish part of the repo which had lots of useful work invested in them, but whose Japanese characters were all completely broken.

Googling around found the Python tool FTFY, which works a treat; it can disembork borken Unicode with remarkable effectiveness. The only situations where it failed were cases of a single isolated character which itself was an archaic or obsolete form.

So the question was how to present the isolated blocks of broken text to FTFY, get them fixed, and then re-integrate them into the file. This is how:

First I cloned the FTFY repo, and ran python3 setup.py install to get a command-line tool (this works better than running stuff in the python interpreter). Then I wrote two XSLT files:

RADish/xsl/fix_chars_1_extract_text_nodes.xsl processes the original file, and finds all text nodes which are ancestor::*[@xml:lang='ja'] or parent::g. It replaces each text node with a temporary <distinct> element with a unique xml:id. It also creates a separate text file which consists of a list of each of those ids, followed by a colon, followed by the borken texten. Then:

ftfy -o outputfile.txt inputfile.txt

fixes almost all of the text inside that text file.

RADish/xsl/fix_chars_2_reinsert_text_nodes.xsl then reads the external file and builds a hashmap from it, then processes the temporary <distinct> elements back into text nodes containing the unborken Japanese.

NH is now in the process of manually fixing the few dozen remaining issues, after we devised some XPath to discover them and fixed a hundred or so together to get the hang of it.

180 minutes.

Created a new spreadsheet for MO at JSR's request, based on titles sold by the Custodian and their preceding titles. May need to do some additional map work, to integrate multiple existing JSON files to create a single one for each row. 180 minutes.