Nxaʔamxcín (Moses) Dictionary Blog

All ready for next week now.

I am posting this exchange about inferred glosses so that I don't have to think it through all over again in the future!

SMK wrote:

Regarding the search engine, I blogged on 12/12/12:

"ECH's goal for the search engine in the web database is that, if a user searches for "fat", s/he will get results including fat, fatten, fattening, fatty. Our current settings, and our policies for adding inferred glosses, seem to be accomplishing this nicely. An entry which has "fatty" in its def is found by a search for "fat", because it also has an inferred gloss "fat". Searching for "fat*" also returns defs including fat, fatten, fattening, fatty ... but also fatal, fathom, father."

However, we also noticed the converse on 16/04/13:

When I searched for the inflected form “fired”, I also I got all the entries with “fire”.

BUT when I search for “fatty” or “fatten”, I don’t get all the entries with “fat”. What is the difference here?

MDH replied:

I think you're just discovering that a stemming analyzer is not an educated human. It doesn't understand semantics; it just knows how to strip off (some) inflectional endings and index the resulting stems, and then how to stem the search input and search the stemmed index with it. You will never find an automated search engine that gives you perfect results.

Right now, the search is paying no attention to whether things are in gloss tags or not; as I understand it, the purpose of the gloss tags is to construct and English-Nxa’amxcin list, not to aid in searching.

The situation with "fatty" is definitely a bit odd; it appears that if you search for that word, you it doesn't get stemmed prior to the search, whereas if you search for "fired" it does. Perhaps the stemmer avoids stemming -tty inputs because there are many which shouldn't be stemmed? ("batty", "natty", "patty", for instance.)

SMK continued:

OK, so when I search for fatten, fattened, or fattening, I get the same 5 hits – 3 for “fattening”, one for “fattened”, and one for “fatten” – i.e. everything with the stem “fatten”. It doesn't go all the way down to the root “fat”, and that's fine.

When I search for “fatty”, all I get is the one entry for “fatty”, as you explained above. That's fine too.

We had been adding inferred glosses for the uninflected English stems and roots of attested glosses, e.g.

<def>
<seg>I am <gloss>fattening</gloss> it up</seg><bibl corresp="psn:W">W10.138</bibl>
<seg><gloss subtype="i">fatten</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
<seg><gloss subtype="i">fat</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
</def>

Here, <gloss subtype="i">fatten</gloss> adds nothing to the search capabilities, because the stemmer can find “fatten” within “fattening”.

But does this entry with “fattening” get found when I search for “fat” because of the stemmer, or because of the <gloss subtype="i">fat</gloss>? It must be because of the inferred gloss, because the stemmer only stems as far as “fatten”.

In the case of “fatty”, where we know the stemmer doesn't operate on it, it still gets found when I search for “fat” because of the <gloss subtype="i">fat.

(“fattening” and “fatty” do NOT get found when I search for “fat” just because they contain the string f-a-t, because “fatal” and “father” are NOT found by a search for “fat”. To find anything with the string f-a-t, I would need to search for “fat*”.)

So the inferred glosses do play a role in improving the search. That said, I don't think we should be going out of our way to add inferred glosses for this reason.

Much discussion over the last few weeks regarding the placing of gloss tags for generating the Eng-Nx wordlist. I attempt to summarize our conclusions here for future reference.

1) Why do we place inferred glosses (<gloss subtype=”i”>)?

At various times, we have placed inferred glosses for augmenting the search engine on the website, and for generating the English word list.

We concluded that from here on, we ONLY need to place gloss tags for generating the English word list. Inferred glosses do sometimes enhance the web search engine, but now that the stemming analyzer is in place, we don't need to do any further markup to help it out.

2) How should we tag inflected English words?

Until last week, we had been inferring the root word (or stem where relevant) when a def is an inflected or derived form of an English word, e.g.

<def>
<seg>he is <gloss>fattening</gloss> it up</seg>
<bibl corresp=“psn:JM”>JM 1.2.3</bibl>
<seg><gloss subtype=“i”>fatten</gloss></seg>
<bibl corresp=“psn:ECH”>ECH</bibl>
<seg><gloss subtype=“i”>fat</gloss></seg>
<bibl corresp=“psn:ECH”>ECH</bibl>
</def>

This encoding means that this entry will show up three times in the English-Nxa’amxcin wordlist: under fat, under fatten, and under fattening. This seems like overkill, especially when these three words will sort one after the other in the English wordlist anyway.

ECH and SMK decided we would like to see the “fat” entries as follows in the print dictionary:

fat: fat

fatten: fatten, fattened, fattening

fatty: fatty

To accomplish this, we need to reduce the number of gloss tags we place in each entry. Inflected English forms (-ed, -ing) should not be gloss tagged; only their root or stem should be gloss tagged.

So “fattening” would now be gloss-tagged as:

<seg>he is <gloss>fatten</gloss>ing it up</seg>

MDH confirmed that the search engine is ignoring gloss tags, so the stemmer will operate on <gloss>fatten</gloss>ing the same as it would on <gloss>fattening</gloss>. (That is, it will continue to return all results with the stem “fatten” when someone searches for fatten, fattened, or fattening.)

MDH has created two sample Eng-Nx word lists based on the 6 files with “complete” status, one using all the gloss tags, and one omitting the inferred gloss tags. They are in moses/trunk/docs/glosses. We concluded that we don't want to programmatically ignore the inferred glosses, because many of them – especially the synonyms – are worth including. But we can refer to these lists to identify the inflected English words whose gloss tags need to be revised.

3) How should we tag English phrasal verbs?

Where appropriate, English phrasal verbs will be enclosed in a single gloss tag - e.g, <gloss>go after</gloss>. This will allow us to organize the headwords in the Eng-Nx word list as follows:

go

go after

go down

go up

, etc.

4) How can we distinguish English homophones in glosses?

English homophones in glosses will be distinguished with a secondary word (or phrase) in an @n attribute on the <gloss> tag, e.g.<gloss n="conflagration">fire</gloss>, <gloss n="back of boat">stern</gloss>. These will then be rendered as follows in the print dictionary:

fire (conflagration):

stern (back of boat):

We decided not to use parts of speech for @n values. We will always use synonyms. We need to select synonyms that will be clear to readers in the community.

I have now disambiguated the English homophones listed here, and updated the Notes on Definitions and Gloss Tagging document accordingly. Where one homophone was far more common in the data than the other, I only added an @n value on the less common one - e.g. watch (wristwatch).

Did this through XSL with some cunning language-detection code based on content and context, and it seems to have worked pretty well. The Names page now uses the @xml:lang attribute instead of its own cruder detection code to build output.

As planned last week.

Meeting to review the presentation -- my task now is to collapse six slides which begin with the picture of the filecard box into a single stepped diagram illustrating the old encoding process and the horrible binary result.

Section 2 is now down to 6 slides, with more detail and more extensive notes.

Following Sarah's post, I've done the following:

• Added a language filter so you can view names only in English or Nxaʔamxcín. This is a crude regex, but it works because English names always begin with caps, and Nxaʔamxcín names never do.
• Turned off the traffic light display in the names page.
• Added more processing to the path, to handle rendering of e.g. choice elements inside names.
• Excluded lexical suffix entries.
• Elaborated the captions and links a bit.

Here are a few requests for the Names page on the website:

DONE -exclude Lexical Suffix entries

DONE -fix the display of sic/corr, so that only “Wenatchi” displays, not “WenatcheeWenatchi” (See for example the entry for “Sam George”.)

DONE -put flora (plants) and fauna (animals) in the link text at the top of the page

-separate out the sorting into Nx-Eng and Eng-Nx pages. Ideally, users should be able to view the complete list, or any of the six lists by name type, sorted either by Nxa'amxcin name or by English name. The present setup with Nx and Eng names mixed together in the Name column is somewhat confusing. Continue to sort the Nx-Eng lists based on name tags in prons. For the present, exclude name tags in orths when generating these lists. Sort the Eng-Nx lists based on name tags in defs.

PENDING ECH'S FURTHER DISCUSSION WITH CCT:

Please also generate a printable version of the six lists of names by type. These only need to be sorted alphabetically by Nxa'amxcin name - i.e. only include the name tags within prons when generating these lists. Ideally they would be spreadsheets with the following columns:

Name (pron:seg type= “p”)
Source (following bibl ... if the pron:seg type= “p” is NOT subtype=“i”)
Definition (all defs)
Pronunciation (pron:seg type= “n”)
Source (following bibl)
Word Parts (hyph)

We've been running the live db with open access since the last time I rebuilt it, so in the process of doing other updates (such as rolling out the Java sorting collations) I've also added back the protection that we had before. In the process of doing this, I got bitten by the horrible eXist bug which enables you to lock yourself out of the admin account if you edit the admin user and forget to retype the password into the two password boxes (the effect is that you end up with a random admin password that you can never discover). As a result, I had to remove the server version of the app and replace it with a refreshed version of my local copy. This failed the first few times -- Tomcat tries to auto-deploy the app before it's completely uploaded the dbx files, so the uploaded .filepart files can not be renamed to overwrite the ones created by the live startup. It took two or three shots to get this problem solved. The only way seems to be to let it deploy, but stop it immediately in the Tomcat manager; then delete all the dbx, lock and log files; then upload them again; then restart it in the manager.

1) For the linguists' dictionary, we would like to see:

first phonemic representation in bold <orthography in angle brackets> [narrow transcription(s) in square brackets], for both forms and cits - e.g.:

ʔáyx̣ʷt <ʔáyx̌ʷt> [ʔáyəx̣ʷt]
√ʔáyx̣ʷ-t
1. be tired
2. tired, worn out

• √ʔáyx̣ʷ-tl kɬʔámnc
<√ʔáyx̌ʷ-tl kɬʔámnč>
[√ʔáyəx̣ʷ-t ləkɬəʔámənč]
he is tired of waiting (for you / me)

2) On the website, we would ultimately like things sorted by orthography.

This morning we decided that a simple and quick way to distinguish between homographs with different meanings is required to make the English lookup part of the dictionary less confusing. This will be achieved by adding a clarificatory word or phrase in the @n attribute of a gloss. Glosses will then be presented in the E-to-M view with this clarification in parentheses. Processing on the website will need to be changed to take account of this, and the print dictionary rendering will also have to be written with this in mind.

The problem of duplicate @xml:id attributes on entries has now become a serious issue for the print dictionary building, because I'm unable to properly process the entire collection properly to produce the book; to build the dictionary I have to use XInclude to create a single XML source file, and when I do that there are over 1600 duplicate ids which prevent some of the processing steps from being successful.

I've taken a quick look at where the duplicates tend to be concentrated, by adding the files in alphabetical order and looking to see how many duplicates occur with each addition. These files create no problems (i.e. they have no duplicates among themselves):

affix_glot-ix.xml
affix_k-m.xml
affix_n-t.xml
affix_u-CAPS.xml
c.xml
c-glot.xml
c-rtr.xml
glottal.xml
h.xml
h-phar-part1.xml
h-phar-part2.xml
l-affric.xml
lex-suff.xml
new-data-2013.xml
p-glot.xml
phar-w.xml
qw-glot.xml
s-rtr.xml
t-glot.xml
xw.xml

When I add the remaining files, one by one (and only one at a time), these are the results:

k.xml            100 duplicates.
k-glot.xml:         18
kw.xml:              2
kw-glot.xml:          2
l.xml:              3
l-fric.xml:          6
m.xml:              3
n.xml:             97
p.xml:              7
particles.xml:          4
pron.xml:          2
q.xml:              4
q-glot.xml:          3
qw.xml:              1
rescued.xml:         54
s.xml:              2
t.xml:             20
ww-glot.xml:          4
x.xml:              3
x-uvul.xml:          4
yy-glot.xml:          4

What I'm going to do is develop the dictionary output using only the valid files, and then add the others in as they get fixed. In the meantime, it might be worth having a go at some of the low-hanging fruit (the ones with only two or three duplicates). More will show up as we add those in, of course -- there will be duplicates across the currently-excluded files as well as those that they share with the "good" files. So the dictionary PDFs will shrink in size, but I'll be able to start doing things like generating page-references that depend on xml:ids.