Nxaʔamxcín (Moses) Dictionary Blog

A quick "layman's language" recap of the algorithm MDH has already encoded ...

-Group words first by their root or stem, as in the previous approach; list the plain root first.

-Group compounds and sort them as in the previous approach; print these last, as they are most complex words with a given root or stem.

-Assign each morpheme a numerical value (list to be finalized).

-Generate sort keys for words as follows:

-First, look at the numbers of the prefix and suffix nearest the root. Add the lower of these numbers to the sort key.

-Then ignore the affix whose number has been added to the sort key, and repeat the process of comparing and adding numbers, building the sort key until no morphemes are left.

I've implemented the basic features of the new algorithm, and sent some results to SMK to see if they make sense. Remaining to be done: sorting of compounds (which can be pulled in from the previous approach), the handling of repeated instances of the root, and the implementation of indent level (which is currently based purely on the length of the key).

SMK has come up with a rather brilliant plan to prioritize the morphemes closer to the root. She'll write up a document to articulate the decisions we made today, and I'll code it in the form of a recursive XSLT function which will generate a sort key based on the numerical values assigned to the morphemes. The length of the key can, we think, also be used to create an indent level, with the output correction that nothing is indented more than one level more than its predecessor.

This never ends. We are now arriving at a canonical sort order for actual morphemes within morpheme groups, so that the secondary sort key can use zero-padded numbers instead of morpheme ids (which obviously don't sort alphabetically in any ideal way). We have also added a final item to the end of the secondary sort key, which consists of the text of the hyph; this will be used to differentiate two items with identical morphemic structure. Finally, I've added many more latin glyphs to the MosesPhonemicSort collation (which is now our default "sort for all occasions" library) so that it can handle e.g. upper-case letters in morpheme ids, and also so that it ignores morpheme delimiter characters, thereby improving the final-stage part of the key mentioned above.

The XSLT function for generating a secondary sort key had been created in an accretive fashion, and ended up being very difficult to change because altering the sequence of the fifteen (so far) steps required manual adjustment of letter codes for all of them. I've now rewritten it in an optimized manner (optimized in the sense that it's probably slightly slower, but trivial to reorder or expand with new steps, as may happen next week).

I should do this sort of thing more often.

Implemented everything laid out in yesterday's post, and some additional tweaks; we're now able to use document order of affixes as an additional component of the sorting mechanism. We also met and discussed the dictionary organization and layout.

SMK and I worked hard on the secondary sorting today, and we have a working system for the lexical suffix group, which is our core test. The basic idea works like this:

• Each particular group has the first part of its sort key created by its own key function, which is customized depending on how the group should be sorted (so in the case of Lexical Suffixes, the first part of the key is a concatenation of the lexical suffix ids in the order they appear in the item, with underscores between them).
• The custom function then calls a generic getSecondarySortKey function, which creates the remainder of the sort key. This works by generating a series of sort key components, working through the print order of the groups. For each major group, it creates a key based on an underscore followed by a letter prefix (a, b, c etc.), then another underscore, then the xml:id of the morpheme which is in that group. The secondary key function has a parameter enabling it to ignore the primary key component, which has already been handled by the custom function.
• The result is a key on which we can sort each major group. This has been implemented so far for everything up to lexical suffixes -- in other words, it generates keys for each of the major groups which are harvested AFTER lexical suffixes (anything harvested before will not appear in the lexical suffix group anyway, because it will be in a preceding group), and each of those groups is handled in their print order.
• The underscore has been added to the MosesPhonemicCollation class at the beginning of the sort order, so it can be used to massage the sort order when necessary.
• TODO #1: complete the secondary sort key function so that it handles everything still missing (Compounds, Primary Affixes and Lexical Prefixes).
• TODO #2: Add the final three sort categories (Gi, Gii, Giii) at the end of the sequence.
• TODO #3: Write or fix up the custom sort functions for all of the other major groups.

Today I ended up writing some XQuery that can write very thorny XSLT templates for sorting subgroups of subentries in the hierarchical index. I now have a system which can do the initial harvesting, sort most of the subgroups on their identifying morphemes (three groups are not yet sorted because it's not clear how to sort them), and spit out results in a human-readable form for checking. It's not really complicated, just incredibly detailed and picky work that takes ages to do, and even longer to debug and check.

Grouping of subentries is now complete, and SMK is checking the results. Meanwhile, I've written my first subgroup sorting template, heavily recursive, and she'll check the results of that next time she's in. It looks like I have a working approach to the subgroup sorting, although it's pretty complicated.

I've started the process of generating the root-based hierarchy of entries, a phenomenally complicated process, but with SMK's excellent documentation I'm making good progress. I'm still completely at a loss as to how I'll sort entries within the subgroups, but we'll come to that when we've managed to generate all the subgroups accurately.