Nxaʔamxcín (Moses) Dictionary Blog

17Aug16: JT has created a new diagnostics page (http://jenkins.hcmc.uvic.ca/job/moses/lastSuccessfulBuild/artifact/trunk/utilities/diagnostics.html) which looks for the following errors in complete, edited, and light-edited files:

-entries with glosses in cits

-entries with no gloss at all (and no name, persName, placeName, orgName or label)

-entries with no def

-entries containing more than one xr (These need to be concatenated into multiple refs within one xr.)

-pron:segs with parentheses in them (These need to be analyzed by a human, and made into either two seg type=n's, or two separate form elements, as necessary.)

-use of n-CTL t-TR Ø-OBJ n-SUBJ or n-CTL t-TR Ø-OBJ s-SUBJ on a word-final -n or -s if preceded by other transitive morphology (n-CTL, t-TR, stu, xit, ɬ-DIR, ɬ-EP,min, nun, tuɬ.)

26Sep16: MDH has added the following additional diagnostics:

-entries with the same string gloss-tagged more than once

-placeName entries that "duplicate" regular entries in <form> (e.g. entry for "deadfall"), so SMK and ECH can review them and make sure they are handled consistently.

-refs in xrs containing non-phonemic characters ... that is, anything BUT these characters

ʔ a á à ạ c ʼ ə h ḥ ʷ i í ì ị k l ḷ ˀ ɬ ƛ m n p q r s ṣ t u ú ù ụ w x y ʕ combining acute accent, combining grave accent, combining dot below, whitespace.

We also need to remember to deal with glosses which end in the English inflections -ing, -ed, -en or -s, as well as ablaut forms like "blow/blew". We can check these with the Find function once all the files are edited, and as we are proofing the print dictionary.

MDH had also asked about diagnostics for what makes an entry "complete", to improve the statistics report. The answer is simply:

-no Not Yet Edited Comment, AND either
-a completed hyph with no m:UNASSIGNED, OR
-a root or stem <fs>

For the time being (26Sep16), MDH has just moved the counts of total entries and entries with no Not Yet Edited Comment into the Statistics section of the new diagnostics page.

If EJD needs to edit the diagnostics further in the future, the file is: trunk/utilities/diagnostics.xsl

Once SMK and EJD have completed a first pass of editing all the files, and resolved all the problems identified by the diagnostics, MDH can make them part of the schematron to make sure we don't introduce new mistakes subsequently.

A little table of contents for our Diagnostics page at:

http://jenkins.hcmc.uvic.ca/job/moses/lastSuccessfulBuild/artifact/trunk/utilities/

1) Broken Cross References - to be checked every time someone finishes editing an alphabetical file. The broken xrs in the affix files will be dealt with later by a separate process. (Commenting out affix cits that duplicate completed entries elsewhere.)

2) Duplicate IDs - to be checked every time someone finishes editing an alphabetical file.

3) Feature Structure Problems - to be checked every time someone finishes editing an alphabetical file.

4) Glosses_All: List of all glosses in the entire database, with the entries they appear in. May be useful for final-final proofread, but doesn't need urgent action.

5) Glosses_Complete: These were all reviewed and gloss-tagging standardized by Marianne Huijsmans in August 2015. No further action needed.

6) Glosses_Edited: List of all glosses in edited and light-edited files, with the entries they appear in. This list could be reviewed and any non-standard glosses fixed, per Notes on Definitions and Gloss-Tagging, but this is not a priority item,

MDH: Found over 3,000. Seems like a lot...

SMK: We revised the process later. The correct total is only 735! And dropping ...

The three compound test cases listed here have been found correctly in the latest build. So it appears the new rules for finding compounds are working.

I spent a long time this afternoon puzzling over the root-based index in the latest build, trying to answer Martin's question from Feb. 2: Is the revised compound-finding rule working correctly?

I think it is, although a lot of our test cases can't be tested fully until other files are completed and added to the build.

The current compound-finding rule looks for:

"all entries with the root or stem in question whose hyphs ALSO contain another <m corresp=m:""> pointing to an entry with a root or stem feature structure"

The new rule is actually finding fewer compounds right now, because it requires the entries for both roots (and/or stems) to be in files included in the build.

Our test case "wəswisxnascʼəlcʼəl", with stem "wisxn" and root "cʼəl" is currently NOT getting found as a compound, because c-glot.xml is unedited, so the root entry for "cʼəl" is not in the build. "wəswisxnascʼəlcʼəl" doesn't get caught by the compound-finder, so it passes through to the next rule and gets caught as a reduplication, and appears under Reduplications under the stem "wisxn". This is wrong, but it's temporary. When c-glot is edited and added to the build, "wəswisxnascʼəlcʼəl" will sort correctly as a compound under both "wisxn" and cʼəl"

Meanwhile, many compounds with root √x̣əƛʼ and other roots which are in the build are being found correctly and organized under both roots.

Another good test case to look at when EJD finishes s.xml and we add it to the build is "siʔsiʔtax̣x̣ƛʼcinʔ". This is not currently being found as a compound, but it should be once s.xml is added to the build.

A couple of the other compounds with √x̣əƛʼ were not found correctly due to the underdot under the x̣ floating under the schwa - likely copy-paste errors made when completing hyphs. I have now fixed these, so they should also be found correctly in the next build.
For the record, they are:

"swahamaɬx̣x̣ƛʼcintn" Ribbon Cliff
"sqəlˀtmx̣Wax̣ƛʼcin" gelding

For further test cases, see SMK's notebook, 22Mar16.

After a couple of false starts due to inadvertently nested morphemes, we've now been able to build a version of the dictionary with over 5,000 entries and 600 pages. SK says we're actually more than half-way through the editing, when other incomplete files are taken into consideration.

SK pointed out that we were harvesting compounds which were not compounds, because we were conflating multiple forms in an entry when doing the test for more than one root or stem. I've now fixed that, and it looks like the current crop of compounds is right. I also worked around a mysterious bug that was eliminating spaces in the rendering of hyphs in the root-based index; I don't know why lone spaces between components were getting lost, but using an en space instead of a #x20 solved it for some reason.

SK suggested a change to the generation of the root-based index, which we implemented, but in order to test it I finally had to bite the bullet and go through the process of getting the dictionary build set up (config was lost during the hard-drive failure last year, and the subsequent transition to a new machine). I discovered a number of things:

• My XEP config had been lost because it was in ~/apps, which was not backed up. I've now reconstructed it; the config file xep.xml was partially reconstructed already based on the ScanCan server setup, but was missing the Aboriginal Sans fonts. I've now added those fonts to the font folder in the xep app folder and to xep.xml, and this stuff is in my backup script.
• I have three Java collations used in the form of jar files for sorting purposes. The source code for two of them is in svn, but the third, MosesOrthographyCollation, was missing both its jar file and source code, although the project framework was there. Not sure how that happened, but probably because it was developed in a different location on the dead drive and didn't get svn'ed or backed up for some reason. I had a built copy of the jar file in the eXist project, so I'm able to use that, but I'll actually need to reconstruct the source code (not complicated) so it can be tweaked if necessary. There's a hard-coded list of the orth symbols in order showing in the built dictionary PDF, so that won't be hard.
• The dictionary now builds OK, and the change to compound discovery, although it should not have retrieved any new compounds per SK for the moment (the particular cases in point are in a file that's not yet being used in the build), it does seem to have found some; I'm waiting for SK to look at those and determine whether they are actually correct, or whether our new criteria are too broad somehow.

The feature structure checker was wrongly flagging lex-suff entries with a single <m> element which mapped to two other morphemes through @corresp as requiring a feature structure. Reported by SMK, fixed today.

Added a couple more tests for problem feature structures based on request from SMK.