Landscapes of Injustice

JL asked for a "dropbox" space for research assistants to upload raw audio files and images of documents.
on home1h/loi/oralhistory, I created a folder called files_in and made the following changes to the default permissions as reported by getfacl oralhistory/files_in/ :
1 - the group has no permissions (setfacl -m g:loi:0 oralhistory/files_in/)
2 - the other has no permissions (setfacl -m o:0 oralhistory/files_in/)
3 - the default:other has rwx (setfacl -m d:other:rwx oralhistory/files_in/)
4 - specific netlinks have rwx (setfacl -m u:USERNAME:rwx oralhistory/files_in/)

The three specified users that have access are: jlabove, rsalas, kylafitz

These changes are also documented in the file permissions.txt in the oralhistory folder.

JL also expressed concern about who has access to the interviews folder, so I've asked for more details and if necessary will make changes and document in permissions.txt and the blog.

Generalized the diagnostics.xsl file so that it can be applied to both the Nihonmachi and Maple Ridge dbs, and added the second process into the ant build; also added a check for titles which are not linked to properties.

Rewrote the diagnostics file so that it processes the existing directories files rather than those created in a temp directory when the algorithm was being developed; build the diagnostics into the main build process and directed its output to the products directory which is preserved. Reported on the results and the work required to complete this task to JSR and MT, who will presumably be doing it.

Then, per discussions with JSR, changed the Other East Asian ethnicity to Other Asian, and ported this change to the original Land Titles db. Abstracted the ethnicity assignment code into a separate module so it can be applied to any source, and then created a new library which uses it to process the Land Titles (Powell St) db, XML version, and creates a diagnostic output that shows where the current algorithm would differ from the original assignments which were made by humans. Analyzed these results and reported on the work which will be involved in revisiting the 600+ names for which this process disagrees with the original one.

Polished off a bunch of the obvious and easy-to-check ones. Could do with a new diagnostic that runs on the current fileset now.

Spent the day on the following:

• Finalizing the method for integrating the names dictionary into the ethnicity assignment process.
• Enhancing the Streets pages so that they now incorporate signals for multiple ethnicities.
• Writing an XSLT transformation that processes the streets tables to create a set of TSV files for the GIS folks to use.
• Integrating all this into the build process, and debugging it.

I think it's all working now, but there are unanswered questions about the handling of multiple block/lot values for a single "address".

Well, the results of applying the huge enamdict Japanese name dictionary are of questionable utility.

Even after excluding all the detectably foreign names (any that are written with katakana at all) along with placenames, products etc., it turns out that there is a relatively large number of rare Japanese names which actually collide with western names -- examples are:

Surnames:
DEE      (出江)
DOUGAN   (道願)
HANNA    (伴奈)
HANSEN   (飯泉)
HENRI    (片理)

Forenames:
ANNE     (晏音)
BEN      (勉)
MINA     (茉奈)

and more. The result is that the number of Japanese Provisional climbs from the zero I'd got it down to back up to 107.

I'm going to think about this over the weekend and decide whether the perceived improvement in trustworthy evidence going into the process justifies the extra work to disentangle these new candidate Japanese names (very few of which, I think, are actually Japanese).

I started the initial notes for a blog post documenting our ethnicity assignment process for Japanese names, and in the process of some background research I identified a resource I had not known about before, the Japanese Multilingual Named Entity Dictionary. This is a fabulous resource that comes in (in XML) at 150MB, and includes not only personal but also placenames and similar entity names. I've downloaded it and written a processor which produces a much-reduced version which will suit our purposes for personal name identification; the result is just over 30MB, so it should be usable in a Saxon process as a lookup. It will provide one more piece of strong evidence that a form is known to be a Japanese name (last or first, male, female or gender-neutral), and will provide the additional potential for finding kanji representations for romanized forms in an automated fashion.

Meanwhile, work on the blog post is stalled while I integrate this resource into the process.

A couple of queries still outstanding, but basically we're done.

Added the Steveston and Haney data into the ethnicity assignment system, which generated a lot more Japanese provisionals, but I powered through them, leaving only nine. Did one pass through the OEAProvs too, marking some that were obviously not, and fixed a pile of transcription errors in the Haney and Steveston directories.

Worked most of the day on the remaining JProvs, getting it down to a gnarly remainder of 12 names. JSR and RRR asked for a new spreadsheet of info on Custodian sales, which I've done one version of, but it's not clear whether they want it organized by title or by property yet, so there'll probably have to be more work done there.