HCMC Blogs

Updated the minutes, highlighted the actions, and moved them into the main Google doc with the previous minutes.

Following discussions yesterday, I've made a minor change to the retrieval code that runs when you click on "Other entries containing this morpheme" on the website. It now retrieves only entries with the morpheme in their <hyph>, not (as before) anywhere at all.

I've made a similar change to a number of XPaths in the PDF generation code, and a quick check of the resulting PDF shows that it doesn't seem to have had any adverse effects.

That means that if the team wishes to tag <m>s in <cit>/<phr> contexts (e.g. for particles), that can be done without any side-effects on the website or on the PDF generation. I could add a feature to the website which triggers only for entries with this feature:

<f name="baseType">
  <symbol value="particle"/>
</f>

which would "Retrieve all example sentences containing this particle", but I think that should only be done if we're consistently tagging the particles in cits. Of course, if that can be done through search-and-replace (in the cit/phr XPath context) for all particles, then there's no reason not to do it since it won't cost us any time.

Reported as follows to JS-R and SF:

I've examined all the Japanese results, and I can confirm:

• All of the 213 identified as definitely Japanese are Japanese.
• Of the 219 identified as possibly Japanese, only a proportion (less than half) actually are. They are obvious in the list, and can be assigned to Japanese manually. The reason for this is that their names match the Japanese regular expression template for name forms (as do many non-Japanese names), but they don't happen to appear in the Asian Surnames list. Maikawa is a good example of this. There are blocks of these (15 Maikawas, for instance) where a name with a strong presence in Powell Street is missing from the Asian Surnames list.
• Of the some-probability-Chinese list, I'm no expert on Chinese names but if our assumption is correct that there are few-to-zero Koreans or Vietnamese to worry about at the time and place in question, then I would suggest they're pretty much all Chinese. The very few instances of surnames which could be non-Chinese (e.g. Gee, Lee) are solved by obviously Chinese forenames.

So this is what I think we can do, with confidence:

1. Run the code to assign all the definite 213 Japanese to Japanese.
2. Pull up all the possibly-Japanese, sort by surname, check the Japanese ones and set them to Japanese manually in blocks.

Re the Chinese assignments, I would be less confident because here we're depending on our assumptions about possible other nationalities, but the results look pretty solid to me; we could certainly assign them all to Chinese_Provisional, but to be honest I think we might as well make them Chinese.

Finally, we should then retrieve all instances with no nationality assignment and go through them manually. I don't know how many that would be, but it'll be mainly people of European origin, so they can probably all be classified as Other in blocks, with a few exceptions. There will be some Chinese whose names don't actually appear in the Asian Surnames list (e.g. Lim and Chong), but I don't think there will be any Japanese, unless they're the result of misspelling in the original documents or the data entry.

At MS's request, integrated corrected versions of the English stories generated using OCR, used with permission from Northwestern University Press, into the site; also added a link to a Russian version of the first story on another site.

Early morning Skype which took up two hours (including 15 minutes prep) then some follow-up which took another half hour.

Using probabilities calculated from the Asian_Surnames database, along with some regex stuff we've developed, I've written an XSLT file which parses the XML dump from the database looking for owners with no assigned ethnicity, and then creates lists of those most probably Japanese, less probably Japanese, possibly Chinese, and probably not either; in each of the first three cases, it also generates SQL insert statements to assign the appropriate ethnicity. Results sent to SF.