Nxaʔamxcín (Moses) Dictionary Blog

Attempted to carry out instruction #3 in SMK's blog post "Global changes to speed up editing", but I can't find a single example of a situation which matches the case. After getting no results on sample files with XSLT, I ran the following XQuery on the whole db to try to find some cases to work with:

xquery version "1.0";

declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace util="http://exist-db.org/xquery/util";
declare namespace hcmc="http://hcmc.uvic.ca/ns";

declare function hcmc:cleanUpNode($inNode as node()) as xs:string{
	translate(normalize-space($inNode), '*', '')
};

for $d in //def[following-sibling::cit][count(seg) = 1]
let $seg := $d/seg
where $d/following-sibling::cit/quote/phr[hcmc:cleanUpNode(.) = hcmc:cleanUpNode($seg)]
return $d/ancestor::entry

along with some variants of it, I couldn't find a single hit, so I've given up on this one.

Tasks 1 and 2 in Sarah's "Global changes to speed up editing" are now complete. The second was a bit complicated because identifying the context for it was predicated on the existence of a hyph, but the hyph had been commented out in the preceding task (#1). However, a very messy regular expression match against the commented-out hyph did the job, after some tweaking.

NOTE: This includes major changes throughout the files, so, for the record, these changes all took place in one revision, rev 1346.

Per SMK's request on the blog.

ECH and SMK finished their editing work yesterday, so this morning I did some final tidying up (lots of minor font and formatting issues), added some extra detail to the references, and created the submission package (three different versions of the file, one with figures all moved to the end, etc.). I think we're finally done! Posted it on a server for ECH to submit, because it's too big to attach to an email.

List of tasks for August:

• Create a print output consisting of only the name entries (entries with <f name="name">/<binary value="true">) using the existing dictionary XSL:FO code.
• Work on the automatic morpheme linker routines outlined in SK's document entitled Autohyphenator. Start with simple ones and work towards the more complicated ones.
• Carry out all the tasks in SK's previous post.

I have some changes to make arising out of this, but what remains is not major.

We have identified the following repetitive steps that the editors are doing. We're hoping these steps can be done globally, programmatically. Please make the following global changes in all files with status="unedited", and Sarah and Caitlin's "editing" files, c.xml and x-uvul.xml.

1. DONE, 9Aug13: Comment out hyphs in monomorphemic entries.

2. DONE, 9Aug13. REVISED with bug fixes, 9Sep13:

For inferred root entries, change structures like this:

<entry xml:id="qəm">

<form>
<pron>
<seg type="p">qə́m</seg>
<bibl><!--[No source]--></bibl>
</pron>
<!--Original hyph element before phonemicization:-->
<!--<hyph xmlns="http://www.tei-c.org/ns/1.0">
√<m corresp="m:UNASSIGNED">qəm</m>-</hyph>-->
<!--<hyph>√<m corresp="m:UNASSIGNED">qəm</m>-</hyph>-->
</form>

<sense> </sense>

<!--[Not yet edited.]-->
</entry>

to this:

<entry xml:id="qəm">

<form>
<pron>
<seg type="p" subtype="i">qəm</seg>
<bibl corresp="psn:ECH">ECH</bibl>
</pron>
</form>

<sense>
<def>
<seg><note type="editorial" resp="psn:ECH">Meaning unclear.</note></seg>
<note type="referToElders" resp="psn:SMK">Is qəm a word by itself? If so, what
does it mean? Click 'other entries containing this morpheme' for examples of
words with the root qəm.</note>
</def>
</sense>

<fs>
<f name="baseType">
<symbol value="root"/>
</f>
</fs>

<note type="editorial" resp="psn:ECH">Root entry added based on attested complex forms.</note>

<!-- [Editor: please double check the following and update this entry accordingly: Is this a root or a stem? Update <symbol value=> Was it inferred by ECH or MDK? Update <bibl> Has it been added based on several attested complex forms, or just one? Update <note> above.]-->

<!--[Not yet edited.]-->
</entry>

This involves the following steps:

-remove stress mark from contents of pron:seg

-add subtype="i" within the pron:seg tag

-replace <bibl><!--[No source]--></bibl> with <bibl corresp="psn:ECH">ECH</bibl>

-replace empty <sense> </sense> with the following. Can you embed the contents of the pron:seg into the contents of the <note>??

<sense>
<def>
<seg><note type="editorial" resp="psn:ECH">Meaning unclear.</note></seg>
<note type="referToElders" resp="psn:SMK">Is contents of pron:seg a word by itself? If so, what does it mean? Click 'other entries containing this morpheme' for examples of words with the root contents of pron:seg</note>
</def>
</sense>

<fs>
<f name="baseType">
<symbol value="root"/>
</f>
</fs>

<note type="editorial" resp="psn:ECH">Root entry added based on attested complex forms.</note>

<!-- [Editor: please double check the following and update this entry accordingly: Is this a root or a stem? Update <symbol value=> Was it inferred by ECH or MDK? Update <bibl> Has it been added based on several attested complex forms, or just one? Update <note> above.]-->

3. MDH attempted, 13Aug13, but no matching contexts can be found in the data:

Within each <sense>, remove defs which duplicate the <seg> and following <bibl> in a <quote>. (These were mistakenly copied twice in the Lexware to XML conversion.)

For example, change this:

<sense>
<def>
<seg>I <gloss>hear</gloss>d you</seg>
<bibl corresp="psn:W">W8.138</bibl>
</def>
<def>
<seg>I did <gloss>not</gloss> <gloss>hear</gloss> you very well</seg>
<bibl corresp="psn:W">W8.139</bibl>
</def>
<cit>
<quote>
<phr type="p" subtype="i">lút x̣ə́st √cq=ánaʔ-mn-c-n</phr>
<bibl corresp="psn:ECH">ECH</bibl>
<phr type="n">lút x̣ə́st √cq=ánaʔ-mən-č-ən</phr>
<bibl corresp="psn:W">W8.139</bibl>
<seg>I did <gloss>not</gloss> <gloss>hear</gloss> you very well</seg>
<bibl corresp="psn:W">W8.139</bibl>
</quote>
</cit>
</sense>

to this:

<sense>
<def>
<seg>I <gloss>hear</gloss>d you</seg>
<bibl corresp="psn:W">W8.138</bibl>
</def>
<cit>
<quote>
<phr type="p" subtype="i">lút x̣ə́st √cq=ánaʔ-mn-c-n</phr>
<bibl corresp="psn:ECH">ECH</bibl>
<phr type="n">lút x̣ə́st √cq=ánaʔ-mən-č-ən</phr>
<bibl corresp="psn:W">W8.139</bibl>
<seg>I did <gloss>not</gloss> <gloss>hear</gloss> you very well</seg>
<bibl corresp="psn:W">W8.139</bibl>
</quote>
</cit>
</sense>

Searching for duplicates such as this one this could be more difficult in cases where programmatic gloss tagging has been done differently in the defs than in the cits. Duplicate defs to be deleted should contain:

-a seg identical to a quote:seg in a following cit, ignoring <gloss> tags and the *character, and
-a sister bibl completely identical to a quote:bibl in a following cit.

4. DONE, 13Aug13: When two or more defs in the same <sense> are identical, collapse them together, but concatenate their <bibl>s. For example, change this ...

<sense>
<def>
<seg>Nez Perce</seg>
<bibl corresp="psn:EP">EP2.84.8</bibl>
</def>

<def>
<seg>Nez Perce</seg>
<bibl corresp="psn:JM">JM2.141.2</bibl>
</def>
</sense>

to this:

<sense>
<def>
<seg>Nez Perce</seg>
<bibl corresp="psn:EP psn:JM">EP2.84.8; JM2.141.2</bibl>
</def>
</sense>

5. DONE, 7Aug13. Please add <name type="flora"> and <name type="fauna"> to the schema, so they will appear in the dropdown list of values.

Tasks arising out of today's meeting:

• Add MLW to the ODD/schema (personography item).
• Add handling for an <entry> with no <form> element (xml:id="unknown") [SMK adds: we will also want to suppress this "unknown" entry from the print dictionary.]
• Build a PDF dictionary with only the affix files and lex-suf (for reference by team).
• In lex-suf, and any affix files which contain matches for the XPath below, do an identity transform which in this context:

  seg[@subtype='i'][following-sibling::bibl[1][@corresp='psn:MDK']]
  

  removes the @subtype value.
• Remove traffic lights from names page (if straightforward).
• Add light-edited (yellow) and light-proofed (green) to the document status values in ODD and schema, then add traffic-light handling.
• Add all light-edited and better docs to the XInclude file and validate to catch any duplicate ids.

All ready for next week now.

I am posting this exchange about inferred glosses so that I don't have to think it through all over again in the future!

SMK wrote:

Regarding the search engine, I blogged on 12/12/12:

"ECH's goal for the search engine in the web database is that, if a user searches for "fat", s/he will get results including fat, fatten, fattening, fatty. Our current settings, and our policies for adding inferred glosses, seem to be accomplishing this nicely. An entry which has "fatty" in its def is found by a search for "fat", because it also has an inferred gloss "fat". Searching for "fat*" also returns defs including fat, fatten, fattening, fatty ... but also fatal, fathom, father."

However, we also noticed the converse on 16/04/13:

When I searched for the inflected form “fired”, I also I got all the entries with “fire”.

BUT when I search for “fatty” or “fatten”, I don’t get all the entries with “fat”. What is the difference here?

MDH replied:

I think you're just discovering that a stemming analyzer is not an educated human. It doesn't understand semantics; it just knows how to strip off (some) inflectional endings and index the resulting stems, and then how to stem the search input and search the stemmed index with it. You will never find an automated search engine that gives you perfect results.

Right now, the search is paying no attention to whether things are in gloss tags or not; as I understand it, the purpose of the gloss tags is to construct and English-Nxa’amxcin list, not to aid in searching.

The situation with "fatty" is definitely a bit odd; it appears that if you search for that word, you it doesn't get stemmed prior to the search, whereas if you search for "fired" it does. Perhaps the stemmer avoids stemming -tty inputs because there are many which shouldn't be stemmed? ("batty", "natty", "patty", for instance.)

SMK continued:

OK, so when I search for fatten, fattened, or fattening, I get the same 5 hits – 3 for “fattening”, one for “fattened”, and one for “fatten” – i.e. everything with the stem “fatten”. It doesn't go all the way down to the root “fat”, and that's fine.

When I search for “fatty”, all I get is the one entry for “fatty”, as you explained above. That's fine too.

We had been adding inferred glosses for the uninflected English stems and roots of attested glosses, e.g.

<def>
<seg>I am <gloss>fattening</gloss> it up</seg><bibl corresp="psn:W">W10.138</bibl>
<seg><gloss subtype="i">fatten</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
<seg><gloss subtype="i">fat</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
</def>

Here, <gloss subtype="i">fatten</gloss> adds nothing to the search capabilities, because the stemmer can find “fatten” within “fattening”.

But does this entry with “fattening” get found when I search for “fat” because of the stemmer, or because of the <gloss subtype="i">fat</gloss>? It must be because of the inferred gloss, because the stemmer only stems as far as “fatten”.

In the case of “fatty”, where we know the stemmer doesn't operate on it, it still gets found when I search for “fat” because of the <gloss subtype="i">fat.

(“fattening” and “fatty” do NOT get found when I search for “fat” just because they contain the string f-a-t, because “fatal” and “father” are NOT found by a search for “fat”. To find anything with the string f-a-t, I would need to search for “fat*”.)

So the inferred glosses do play a role in improving the search. That said, I don't think we should be going out of our way to add inferred glosses for this reason.