HCMC Blogs

This post details a method used to fix the following problem:

XML files where downloaded through a web browser from revision.hcmc.uvic.ca. They were XML in UTF-8, but Apache didn't know that so it presented them as 8859-1, and the browser happily believed it. All Japanese characters inside the files therefore got borked.

Work proceeded without this being noticed, resulting in a pair of -ography files in the RADish part of the repo which had lots of useful work invested in them, but whose Japanese characters were all completely broken.

Googling around found the Python tool FTFY, which works a treat; it can disembork borken Unicode with remarkable effectiveness. The only situations where it failed were cases of a single isolated character which itself was an archaic or obsolete form.

So the question was how to present the isolated blocks of broken text to FTFY, get them fixed, and then re-integrate them into the file. This is how:

First I cloned the FTFY repo, and ran python3 setup.py install to get a command-line tool (this works better than running stuff in the python interpreter). Then I wrote two XSLT files:

RADish/xsl/fix_chars_1_extract_text_nodes.xsl processes the original file, and finds all text nodes which are ancestor::*[@xml:lang='ja'] or parent::g. It replaces each text node with a temporary <distinct> element with a unique xml:id. It also creates a separate text file which consists of a list of each of those ids, followed by a colon, followed by the borken texten. Then:

ftfy -o outputfile.txt inputfile.txt

fixes almost all of the text inside that text file.

RADish/xsl/fix_chars_2_reinsert_text_nodes.xsl then reads the external file and builds a hashmap from it, then processes the temporary <distinct> elements back into text nodes containing the unborken Japanese.

NH is now in the process of manually fixing the few dozen remaining issues, after we devised some XPath to discover them and fixed a hundred or so together to get the hang of it.

180 minutes.

Late duty, and overwhelmed with work. No time to record what I did today.

At SA's request, I've added a count of tags to each category in the interview rendering; the word-count is done in JavaScript in the HTML because it's easier that way, but the tag-count is more easily done in the XSLT so that's what I've done. I also added a report on unused tags to the index page, because I think we should get rid of any that we don't use as soon as possible.

The encoding team raised the issue of needing to break various elements around page-breaks so that the fw elements don't inherit styling from their ancestor paragraph (or similar). I've written some usefully-repurposable XSLT in /db/data/utilities/break_elements_at_pbs.xsl, and debugged/tested it with TEH's help, to recursively split elements up through the hierarchy and knit them together with @next and @previous.

This was conceptually quite tricky, although the solution is relatively straightforward once written. Recording details here because one day I'll need to reorganize or restructure a document hierarchy in a similar way, and I don't want to have to reinvent this particular wheel. 120 minutes.

Added two new fields, being a VIAF id field which turns into a link to the VIAF record, and a hashtag field for the persons table. The latter required a rewrite of the diagnostics, and some testing. 120 minutes.

Late duty.

I'm through the first pass at creating a little website for the category analysis, and I went to the trouble of getting it to work in Windows as well (which took at least half of the time spent). The only wrinkle apart from Windows filepath-wrangling was that I had to deal with the fact that for base categories, the same block of text may be retrieved multiple times from the same interview because it's tagged with multiple sub-categories, so I had to filter out those dupes.

180 minutes.

Keeping lots of balls in the air simultaneously.

Our weekly meeting analysed and discussed some interesting edge case linked poems. I have new TODOs from the meeting. 150 minutes.

This was a long and complicated one; the Schematron is doing a lot to catch errors, though. 120 minutes.