HCMC Blogs

My contention about the change to docUtils.java having caused a regression which broke relative paths for the doc() function was borne out after I changed the file and rebuilt. Reported the bug formally on the bugtracker, and it is now fixed, so I have a fresh trunk build of eXist ready to go for MoEML. I'll deploy this first thing tomorrow before anyone else gets to work.

I am posting this exchange about inferred glosses so that I don't have to think it through all over again in the future!

SMK wrote:

Regarding the search engine, I blogged on 12/12/12:

"ECH's goal for the search engine in the web database is that, if a user searches for "fat", s/he will get results including fat, fatten, fattening, fatty. Our current settings, and our policies for adding inferred glosses, seem to be accomplishing this nicely. An entry which has "fatty" in its def is found by a search for "fat", because it also has an inferred gloss "fat". Searching for "fat*" also returns defs including fat, fatten, fattening, fatty ... but also fatal, fathom, father."

However, we also noticed the converse on 16/04/13:

When I searched for the inflected form “fired”, I also I got all the entries with “fire”.

BUT when I search for “fatty” or “fatten”, I don’t get all the entries with “fat”. What is the difference here?

MDH replied:

I think you're just discovering that a stemming analyzer is not an educated human. It doesn't understand semantics; it just knows how to strip off (some) inflectional endings and index the resulting stems, and then how to stem the search input and search the stemmed index with it. You will never find an automated search engine that gives you perfect results.

Right now, the search is paying no attention to whether things are in gloss tags or not; as I understand it, the purpose of the gloss tags is to construct and English-Nxa’amxcin list, not to aid in searching.

The situation with "fatty" is definitely a bit odd; it appears that if you search for that word, you it doesn't get stemmed prior to the search, whereas if you search for "fired" it does. Perhaps the stemmer avoids stemming -tty inputs because there are many which shouldn't be stemmed? ("batty", "natty", "patty", for instance.)

SMK continued:

OK, so when I search for fatten, fattened, or fattening, I get the same 5 hits – 3 for “fattening”, one for “fattened”, and one for “fatten” – i.e. everything with the stem “fatten”. It doesn't go all the way down to the root “fat”, and that's fine.

When I search for “fatty”, all I get is the one entry for “fatty”, as you explained above. That's fine too.

We had been adding inferred glosses for the uninflected English stems and roots of attested glosses, e.g.

<def>
<seg>I am <gloss>fattening</gloss> it up</seg><bibl corresp="psn:W">W10.138</bibl>
<seg><gloss subtype="i">fatten</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
<seg><gloss subtype="i">fat</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
</def>

Here, <gloss subtype="i">fatten</gloss> adds nothing to the search capabilities, because the stemmer can find “fatten” within “fattening”.

But does this entry with “fattening” get found when I search for “fat” because of the stemmer, or because of the <gloss subtype="i">fat</gloss>? It must be because of the inferred gloss, because the stemmer only stems as far as “fatten”.

In the case of “fatty”, where we know the stemmer doesn't operate on it, it still gets found when I search for “fat” because of the <gloss subtype="i">fat.

(“fattening” and “fatty” do NOT get found when I search for “fat” just because they contain the string f-a-t, because “fatal” and “father” are NOT found by a search for “fat”. To find anything with the string f-a-t, I would need to search for “fat*”.)

So the inferred glosses do play a role in improving the search. That said, I don't think we should be going out of our way to add inferred glosses for this reason.

Much discussion over the last few weeks regarding the placing of gloss tags for generating the Eng-Nx wordlist. I attempt to summarize our conclusions here for future reference.

1) Why do we place inferred glosses (<gloss subtype=”i”>)?

At various times, we have placed inferred glosses for augmenting the search engine on the website, and for generating the English word list.

We concluded that from here on, we ONLY need to place gloss tags for generating the English word list. Inferred glosses do sometimes enhance the web search engine, but now that the stemming analyzer is in place, we don't need to do any further markup to help it out.

2) How should we tag inflected English words?

Until last week, we had been inferring the root word (or stem where relevant) when a def is an inflected or derived form of an English word, e.g.

<def>
<seg>he is <gloss>fattening</gloss> it up</seg>
<bibl corresp=“psn:JM”>JM 1.2.3</bibl>
<seg><gloss subtype=“i”>fatten</gloss></seg>
<bibl corresp=“psn:ECH”>ECH</bibl>
<seg><gloss subtype=“i”>fat</gloss></seg>
<bibl corresp=“psn:ECH”>ECH</bibl>
</def>

This encoding means that this entry will show up three times in the English-Nxa’amxcin wordlist: under fat, under fatten, and under fattening. This seems like overkill, especially when these three words will sort one after the other in the English wordlist anyway.

ECH and SMK decided we would like to see the “fat” entries as follows in the print dictionary:

fat: fat

fatten: fatten, fattened, fattening

fatty: fatty

To accomplish this, we need to reduce the number of gloss tags we place in each entry. Inflected English forms (-ed, -ing) should not be gloss tagged; only their root or stem should be gloss tagged.

So “fattening” would now be gloss-tagged as:

<seg>he is <gloss>fatten</gloss>ing it up</seg>

MDH confirmed that the search engine is ignoring gloss tags, so the stemmer will operate on <gloss>fatten</gloss>ing the same as it would on <gloss>fattening</gloss>. (That is, it will continue to return all results with the stem “fatten” when someone searches for fatten, fattened, or fattening.)

MDH has created two sample Eng-Nx word lists based on the 6 files with “complete” status, one using all the gloss tags, and one omitting the inferred gloss tags. They are in moses/trunk/docs/glosses. We concluded that we don't want to programmatically ignore the inferred glosses, because many of them – especially the synonyms – are worth including. But we can refer to these lists to identify the inflected English words whose gloss tags need to be revised.

3) How should we tag English phrasal verbs?

Where appropriate, English phrasal verbs will be enclosed in a single gloss tag - e.g, <gloss>go after</gloss>. This will allow us to organize the headwords in the Eng-Nx word list as follows:

go

go after

go down

go up

, etc.

4) How can we distinguish English homophones in glosses?

English homophones in glosses will be distinguished with a secondary word (or phrase) in an @n attribute on the <gloss> tag, e.g.<gloss n="conflagration">fire</gloss>, <gloss n="back of boat">stern</gloss>. These will then be rendered as follows in the print dictionary:

fire (conflagration):

stern (back of boat):

We decided not to use parts of speech for @n values. We will always use synonyms. We need to select synonyms that will be clear to readers in the community.

I have now disambiguated the English homophones listed here, and updated the Notes on Definitions and Gloss Tagging document accordingly. Where one homophone was far more common in the data than the other, I only added an @n value on the less common one - e.g. watch (wristwatch).

ES added transcripts for accf3, fraf8, cltf6

Trying to abstract the combined keyword/text search into a separate library yesterday was very problematic, but I took a simpler approach this morning and simply copied and adapted the code from search.xq into advanced_search.xq. The result seems to be working perfectly -- the keyword/text search is done first to retrieve a set of @xml:ids, then the search is done on those ids, with additional filters provided by the other form controls.

Did this through XSL with some cunning language-detection code based on content and context, and it seems to have worked pretty well. The Names page now uses the @xml:lang attribute instead of its own cruder detection code to build output.

It was great meeting you all today and I'm looking forward to working with you all through the summer! I thought I would post one of my favorite newspaper articles from the project I mentioned today. Blayney was the oldest of the Scott brothers and the event that earned him the Distinguished Flying Cross is outlined in the article on the left. It's a pretty unbelievable story!

Happy hunting tomorrow.