Nxaʔamxcín (Moses) Dictionary Blog

I would dearly love to be able to do the right thing with regard to XHTML mime types, but mixing AJAX and XHTML 1.1 seems to be a really hairy mess. IE8 and below, of course, don't want application/xhtml+xml at all, while IE9 is supposed to accept it, but it seems in practice it only actually wants it for XHTML5, not 1.1. Firefox is also fussy; once it's getting application/xhtml+xml, it doesn't want to handle AJAX in the same way. I did spend some time on a sitemap selector that checks the accept header of the browser, and then serves the correct mime type, but unfortunately the JavaScript also has to function slightly differently depending on the mime type, which complicates everything. I ended up rolling back to a previous version of the site. It will work in IE9 in compatibility mode, but that's pretty annoying.

I'm wondering if the solution might be to rewrite the site using XHTML5, but serve it as text/html (which is OK as long as it has the right doctype -- it will happen to be valid XHTML5, but I'll be sending it as equally-valid HTML5). That would probably make the AJAX operation much easier. We don't really care about old versions of IE, but I would like to have the site work on IE9 if it's not going to cost too much time and effort.

Solved the problem outlined in the previous post by using generate-id(). The retrieval of entries for a particular gloss takes place only once now; after that, the same <div> is shown or hidden.

I've done a lot of work on the English wordlist, and discovered that doing grouping in XSLT with Saxon seems to be much faster than doing it in XQuery in eXist, so I'm returning a simpler list of items from eXist and doing all the heavy lifting with <xsl:for-each-group> in XSLT. The English glosses now come down in distinct values, accompanied by a list of Moses headwords that contain the gloss. If you click on the gloss, you now get a distinct list of the Moses headwords, each with its distinct gloss, and then you can click on the headword to retrieve the complete entry. I still haven't figured out how to hide/show the content block under the gloss, because it doesn't come back with a distinct id; the gloss key which is used to look up the entries can't function as an id (it sometimes begins with a number, and has unacceptable characters in it such as asterisks). I'll need to figure out a strategy for this.

Note to self: here's a possibility: The XSLT could use generate-id() to create a unique id for the list item for each gloss, and the gloss could pass that id to GetEntriesForGloss(), which could then include a modified version of it in the div that comes back from the AJAX call. Then it would be easy to do hide/show, and avoid retrieving the same thing multiple times.

I've started work on a more sophisticated way of generating the English wordlist, which would show a single headword for each distinct gloss, followed by an embedded list of each of the entries which contain that gloss, and their brief definitions. My first shot at generating the XML for this is taking forever on my local machine -- it's a bit recursive, and there are thousands of entries -- so I think I'll need to reconsider. The first thing to do is to add range indexes for all the key tags; I should have done that originally. Once that's done, I can time the production of the full list, and perhaps break it down by letters of the alphabet so that it's only trying to render one letter at a time. This might take a while to do, indeed.

As planned, the views of Moses entries ("finished" and "all") now look like the wordlist view (term + glosses), and the Moses-English wordlist has been removed from the menu.

Productive meeting this morning, giving rise to the following ideas:

• The "entries" and "wordlist" views should in fact be merged, such that the entries are presented initially in the form that the wordlist now takes (with the headword followed by all its glosses), expanding to a full entry when clicked on.
• Duplicate glosses will be removed; there's no reason for the same word or phrase to be wrapped in a <gloss> tag more than once in the same entry. I'll see about generating a list of duplicates using XQuery to help in detecting them.
• In the "entries" view (Moses), affixes should not display any glosses, because where <gloss> tags are contained inside them, they're actually glossing a word containing the affix rather than the affix itself. Instead, the feature structure information explaining the function of the affix should be displayed, in a manner which makes it clearly distinct from regular glosses, and the affix head form itself should be distinguished visually so that it's clear that it's not a regular word.
• The English-Moses wordlist is to be replaced by a serious attempt to produce something along the lines of a regular English-Moses dictionary, generated automatically from the DB. This will most possibly be as unsatisfactory as previous projects which used Lexware to generate the same sort of dictionary, but we may be able to do a more sophisticated job than that. Ultimately, when the initial phase of the work on the core code is complete, we may add some markup to <dicteg>s which enables us to generate a more complete set of English headwords by harvesting English words and Moses equivalents from <dicteg>s. (The example here is "flannel", inside the current entry for "cloth"; the Moses equivalent is a phrase containing "cloth", so it doesn't have its own entry in Moses, but in English it probably should.)
• DONE: The Moses-to-English basic view should not use words from gloss tags; instead, it should use the content of the <def>/<seg> elements as the "gloss" of the word.
• DONE: Unattested glosses should never be displayed at all in the Moses-English view. They're only to be used when constructing the English-Moses view.

There are about 13.5 thousand entries.

I wrote a test unit to figure out what was happening with the RuleBasedCollator, and I've fixed it. These are the issues I came up against:

• The class was generating a ParseException, but Cocoon or Saxon were silently swallowing the exception, and defaulting to a normal alpha sort. Until I ran the test unit, I didn't know the exception was being triggered.
• The exception was caused by the fact that I had included both decomposed and composed versions of (for instance) a with a dot below. It turns out that the RuleBasedCollator class actually does normalization itself before doing comparisons, so there was no need for this, and in fact it was throwing the ParseException because it looked as though I was asserting that a character was equal to itself (or unequal -- the exception wasn't very clear on this).
• There were a couple of typos in Unicode codepoints for raised W, which were causing the precise sort I was using for testing to fail anyway.
• There was a stray combining-dot-below, virtually invisible and hard to select, in the rule definition, making it unparseable.

The collator now seems to be working fine. The code is below:

package ca.uvic.hcmc.moses;

/**
 *
 * @author mholmes
 */

import java.text.ParseException;
import java.text.RuleBasedCollator;

public class MosesCollation  extends RuleBasedCollator{

  public MosesCollation() throws ParseException
  {
    super(mosesRules);
  }

/*
 * Commented-out statements below are replaced by simpler ones, because the
 * RuleBasedCollator automatically does Unicode normalization before it does
 * its comparisons; including "parallel" versions of characters was triggering
 * a ParseException.
 */
  private static String glottal =    new String("\u0294");
  private static String a =          new String("a,a\u0301,a\u0300,\u00e1,\u00e0");
  //private static String aDot =       new String("\u1ea1,a\u0323,\u1ea1\u0301,a\u0323\u0300,\u00e1\u0323,\u00e0\u0323");
  private static String aDot =       new String("\u1ea1,\u1ea1\u0301,\u1ea1\u0300,\u00e1\u0323,\u00e0\u0323");
  private static String cDot =       new String("c\u0323");
  private static String cApos =      new String("c\u02bc");
  private static String schwa =      new String("\u0259,\u0259\u0301,\u0259\u0300");
  private static String schwaDot =   new String("\u0259\u0323,\u0259\u0323\u0301,\u0259\u0323\u0300");
  //private static String hDot =       new String("\u1e25,h\u0323");
  private static String hDot =       new String("\u1e25");
  //private static String hDotW =      new String("\u1e25\u02b7,h\u0323\u02b7");
  private static String hDotW =      new String("\u1e25\u02b7");
  private static String i =          new String("i,i\u0301,i\u0300,\u00ed,\u00ec");
  private static String iDot =       new String("\u1ecb,\u1ecb\u0301,\u1ecb\u0300,i\u0323\u0301,i\u0323\u0300,\u00ed\u0323,\u00ec\u0323");
  private static String kApos =      new String("k\u02bc");
  private static String kW =         new String("k\u02b7");
  private static String kAposW =     new String("k\u02bc\u02b7");
  //private static String lDot =       new String("\u1e37,l\u0323");
  private static String lDot =       new String("\u1e37");
  private static String lGlot =      new String("l\u02c0");
  //private static String lDotGlot =   new String("\u1e37\u02c0,l\u0323\u02c0");
  private static String lDotGlot =   new String("\u1e37\u02c0");
  private static String lBelt =      new String("\u026c");
  private static String barLamApos = new String("\u019b\u02bc");
  private static String mGlot =      new String("m\u02c0");
  private static String nGlot =      new String("n\u02c0");
  private static String pApos =      new String("p\u02bc");
  private static String qApos =      new String("q\u02bc");
  private static String qW =         new String("q\u02b7");
  private static String qAposW =     new String("q\u02bc\u02b7");
  private static String rGlot =      new String("r\u02c0");
  //private static String sDot =       new String("\u1e63,s\u0323");
  private static String sDot =       new String("\u1e63");
  private static String tApos =      new String("t\u02bc");
  private static String u =          new String("u,u\u0301,u\u0300,\u00fa,\u00f9");
  private static String uDot =       new String("\u1ee5,\u1ee5\u0301,\u1ee5\u0300,u\u0323\u0301,u\u0323\u0300,\u00fa\u0323,\u00f9\u0323");
  private static String wGlot =      new String("w\u02c0");
  private static String xW =         new String("x\u02b7");
  private static String xDot =       new String("x\u0323");
  private static String xDotW =      new String("x\u0323\u02b7");
  private static String yGlot =      new String("y\u02c0");
  private static String phar =       new String("\u0295");
  private static String pharGlot =   new String("\u0295\u02c0");
  private static String pharW =      new String("\u0295\u02b7");
  private static String pharGlotW =  new String("\u0295\u02c0\u02b7");

  private static String mosesRules =
    ("< " + glottal + " < " + a + " < " + aDot + " < c " +
     " < " + cDot + " < " + cApos + " < " + schwa + " < " + schwaDot +
     " < h < " + hDot + " < " + hDotW + " < i " +
     " < " + iDot + " < k < " + kApos + " < " + kW +
     " < " + kAposW + " < l < " + lDot + " < " + lGlot +
     " < " + lDotGlot + " < " + lBelt + " < " + barLamApos + " < m " +
     " < " + mGlot + " < n < " + nGlot + " < p " +
     " < " + pApos + " < q < " + qApos + " < " + qW +
     " < " + qAposW + " < r < " + rGlot + " < s " +
     " < " + sDot + " < t < " + tApos + " < " + u + 
     " < " + uDot + " < w < " + wGlot + " < x " +
     " < " + xW + " < " + xDot + " < " + xDotW + " < y " +
     " < " + yGlot + " < " + phar + " < " + pharGlot + " < " + pharW +
     " < " + pharGlotW);

}

Created simple wordlist output formats, so we can get a handle on how best to deal with <gloss> tags. They're drawing from the complete set of entries, edited or not. They show a couple of things that need fixing, most of which we know about, but they also show that there's a tendency to include the same gloss multiple times in an entry. For instance, in an edited entry we have this:

 ṣə̣́nṣə̣nt: tame, gentle, quiet, tame, tame, gentle

The entry shows why: there are multiple <gloss> tags containing the same words sprinkled through the <entry>.

It's possible for me to write the code so that it ignores these duplicate entries, but there are a couple of problems with that: first, SMK and ECH would be doing lots of extra tagging that we're ignoring, making the entries more complicated, and second, the generation of wordlists will take much longer because it'll have to check every <gloss> to see if it's a duplicate. So I think a good policy would be to make sure we only tag a particular word or phrase once as a gloss in any given entry.

I've rewritten the RuleBased Collator -- source code is below. However, when invoked in the Moses web application, it appears not to be working at all. The next stage will be writing a test package to test it within the Java environment to see if it's the Java that's broken, or something in the XSLT or Saxon which is failing to use it properly. Source code:

/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */

package ca.uvic.hcmc.moses;

/**
 *
 * @author mholmes
 */

import java.text.ParseException;
import java.text.RuleBasedCollator;

public class MosesCollation  extends RuleBasedCollator{

  public MosesCollation() throws ParseException
  {
    super(mosesRules);
  }

  private static String glottal =    new String("\u0294");
  private static String a =          new String("a,a\u0301,a\u0300,\u00e1,\u00e0");
  private static String aDot =       new String("\u1ea1,a\u0323,\u1ea1\u0301,a\u0323\u0300,\u00e1\u0323,\u00e0\u0323");
  private static String cDot =       new String("c\u0323");
  private static String cApos =      new String("c\u02bc");
  private static String schwa =      new String("\u0259,\u0259\u0301,\u0259\u0300");
  private static String schwaDot =   new String("\u0259\u0323,\u0259\u0323\u0301,\u0259\u0323\u0300");
  private static String hDot =       new String("\u1e25,h\u0323");
  private static String hDotW =      new String("\u1e25\u02b7,h\u0323\u02b7");
  private static String i =          new String("i,i\u0301,i\u0300,\u00ed,\u00ec");
  private static String iDot =       new String("\u1ecb,\u1ecb\u0301,\u1ecb\u0300,i\u0323\u0301,i\u0323\u0300,\u00ed\u0323,\u00ec\u0323");
  private static String kApos =      new String("k\u02bc");
  private static String kW =         new String("k\u02b7");
  private static String kAposW =     new String("k\u02bc\u027b");
  private static String lDot =       new String("\u1e37,l\u0323");
  private static String lGlot =      new String("l\u02c0");
  private static String lDotGlot =   new String("\u1e37\u02c0,l\u0323\u02c0");
  private static String lBelt =      new String("\026c");
  private static String barLamApos = new String("\u019b\u02bc");
  private static String mGlot =      new String("m\u02c0");
  private static String nGlot =      new String("n\u02c0");
  private static String pApos =      new String("p\u02bc");
  private static String qApos =      new String("q\u02bc");
  private static String qW =         new String("q\u02b7");
  private static String qAposW =     new String("q\u02bc\u027b");
  private static String rGlot =      new String("r\u02c0");
  private static String sDot =       new String("\u1e63,s\u0323");
  private static String tApos =      new String("t\u02bc");
  private static String u =          new String("u,u\u0301,u\u0300,\u00fa,\u00f9");
  private static String uDot =       new String("\u1ee5,\u1ee5\u0301,\u1ee5\u0300,u\u0323\u0301,u\u0323\u0300,\u00fa\u0323,\u00f9\u0323");
  private static String wGlot =      new String("w\u02c0");
  private static String xW =         new String("x\u02b7");
  private static String xDot =       new String("x\u0323");
  private static String xDotW =      new String("x\u0323\u02b7");
  private static String yGlot =      new String("y\u02c0");
  private static String phar =       new String("\u0295");
  private static String pharGlot =   new String("\u0295\u02c0");
  private static String pharW =      new String("\u0295\u02b7");
  private static String pharGlotW =  new String("\u0295\u02c0\u02b7");

  private static String mosesRules =
    ("< " + glottal + " < " + a + "a < " + aDot + " < c " +
     " < " + cDot + "̣  < " + cApos + " < " + schwa + " < " + schwaDot +
     " < h < " + hDot + " < " + hDotW + " < i " +
     " < " + iDot + " < k < " + kApos + " < " + kW +
     " < " + kAposW + " < l < " + lDot + " < " + lGlot +
     " < " + lDotGlot + " < " + lBelt + " < " + barLamApos + " < m " +
     " < " + mGlot + " < n < " + nGlot + " < p " +
     " < " + pApos + " < q < " + qApos + " < " + qW +
     " < " + qAposW + " < r < " + rGlot + " < s " +
     " < " + sDot + " < t < " + tApos + " < " + u + 
     " < " + uDot + " < w < " + wGlot + " < x " +
     " < " + xW + " < " + xDot + " < " + xDotW + " < y " +
     " < " + yGlot + " < " + phar + " < " + pharGlot + " < " + pharW +
     " < " + pharGlotW);

}