Nxaʔamxcín (Moses) Dictionary Blog

For the record, this is what I did to test the Autophonemicizer on 21Dec12:

My test file was l-affric.xml.

I checked the first few already-edited entries, and no problems had been introduced by the autophonemcizer.

Then I carefully checked the transformation on lines 1811 to 2596, and cut-and pasted the correctly transformed pron:segs and hyphs into the main file. (I already did the quotes in this file on 13Dec12.) There were a couple of problems due to typos in the inputs, but nothing that we could have predicted or improved.

Then I pasted the rest of the autophonemicized file into the main l-affric file, so it is done.

Martin ran the autophonemicizer on all files with status="unedited" and status="rescued" on 24Dec12. Besides l-affric, the ones with status="editing" and status="additions_needed" are lexical suffix and affix files. In these, the prons are all already edited, the hyphs have been commented out, and the quotes will eventually be commented out too, so there is nothing to be gained from autophonemicizing these files.

On SK's instructions, I've run the autophonemicizer on all files with @status="unedited" and "rescued". Everything validates, and I've committed to SVN and uploaded to the db.

The autophonemicizer now handles prons, hyphs and quotes, and seems to be working well. One final tweak might be to re-generate the @xml:id of the entry based on the phonemicized hyph as well, but we may want to leave that until later, when we can just more reliably use the first seg in the pron.

SK will test, and if she's happy, she'll send me a list of files this should be run on, and I'll run it on Monday.

We had a common situation in unprocessed files in which the same transcription would appear in multiple segs in the same pron, with different bibls. I've written XSLT to detect and collapse these instances, and after some testing and tweaking we ran it on the whole collection today. This puts us in a better position to do phonemicization of the segs, because we don't have to allow for duplicates during that process.

Phonemicization has also been extended to deal with R, and we've realized that we can add a phonemic seg based on the existing hyph (text content), so the code that already works for phr should be usable. I have to finish that bit, and also rewrite the phr stuff so that it does some collapsing of its own at the end of the process.

Once in a while Moses loses its marbles when ingesting a large number of documents. A couple of suggestions:

• Delete everything in the main collection before attempting to upload.
• Upload in small batches.

When it does fall over, this is what seems to be necessary:

• Stop Tomcat on Pear (as yourself, cd /etc/init.d; sudo ./tomcat stop).
• Delete the moses webapp completely.
• Upload your local working copy (from ~/apps/apache-tomcat..).
• Overwrite the web.xml file from the local copy with the one from the SVN tree.
• Make sure everything is owned by hcmc.
• As hcmc, chmod -R g+w *
• Restart Tomcat (as yourself, cd /etc/init.d; sudo ./tomcat start).
• Confirm all projects are back up.

I remembered something today that will make it much easier to autophonemicize pron:segs.

In Lexware, all the entries were hyphed. MDH took the same input to generate both the pron:segs (removing the morpheme delimiters) and the hyphs (adding the m-tags).

So we need use the contents of the hyphs, minus their m-tags, to generate the pron:<seg type="p" subtype="i">s. Then the morpheme delimiters will be included, and we can use the same autophonemicizer transformation that we used for the <phr>s. Then we simply remove + - = √ from the pron:segs again. Then we can collapse duplicate pron:segs, the same as we're doing for <phr>s.

So we don't have to worry about writing complicated rules using a long list of prefixes to find the "root-initial" context in pron:segs without morpheme delimiters.

We will also autophonemicize the contents of the hyphs, by removing the m-tags, autophonemicizing, and then adding the m-tags back in based on the morpheme delimiters.

Autophonemicizer2.doc contains detailed instructions for all this.

MDH has also implemented a transformation to collapse identical segs within the same pron. This happens before the autophonemicization.

Visions of Autohyphenators dance in my head! Maybe next year ...

We've decided to extend the existing XSLT rather than create a new file and duplicate code, so <segs> in <pron>s will now be handled by the same transformation that handles <phr>s in <quote>s. The bones are in place, and the existing code is still working well on our test set with the empty branching in place to support <pron>s. The test set has been extended and we've already got a long regex to match the range of prefixes which will serve as a surrogate for the more convenient morpheme delimiters.

One final step: the collapsing of duplicate <phr> elements, which is currently handled by an external post-transformation, needs to be brought back into this single file, so that all the changes required are done at the same time.

Here's a summary of what MDH needs to do next to improve the Autophonemicizer for <phr>s, and implement an Autophonemicizer for pron:segs:

1) Add new step 5c from Autophonemicizer2.doc to the autophonemicizer for <phr>s.

2) Add new step 7 from Autophonemicizer2.doc, to compare generated <phr type="p" subtype="i">s to attested <phr type="n">s and collapse identical ones. This will be a subsequent step to the whole autophonemicization process.

3) Adapt the autophonemicizer for pron:segs

Got our abstract finished and submitted. The deadline has been extended, but it's good to get it out of the way.