Nxaʔamxcín (Moses) Dictionary Blog

Reminder to self on how to generate dictionaries:

• In Oxygen, select all the XML files in the dictionary directory and run the transformation moses_auto_orthography. This generates all the files with automatic orthographical forms in dictionary_test.
• Open the file utilities/generate_names-only_dictionary.xsl, and run it on itself. This uses the files from dictionary_test to create the smaller XML files in the names subfolder.
• Select the XML files in the names subfolder and run the transformations moses_xml_to_pdf_LEARNER_include_names and moses_xml_to_pdf_LINGUIST_include_names. This generates the complete set of pdfs in the names/pdfs subfolder.

A couple of days ago I ran this to get a candidate list of erroneous defs which should have been quotes:

xquery version "3.0";
declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace util="http://exist-db.org/xquery/util";

for $t in //TEI
for $e in $t//entry
let $quoteSegs := for $s in $e//quote/seg where
string-length(normalize-space($s)) gt 0 return translate(normalize-space($s), '*', ''),
$defs := for $d in $e//def/seg where string-length(normalize-space($d)) gt 0 return translate(normalize-space($d), '*', ''),
$dupes := distinct-values($quoteSegs[.=$defs])
where count($dupes) gt 0
order by util:document-name($t), $e/@xml:id
return concat(util:document-name($t/@xml:id), ' / ', $e/@xml:id, ' : ',
string-join($dupes, ' | ')) 

I've now written XSLT (utilities/flag_superfluous_defs.xsl) to insert a comment before each of these candidates.

Met to discuss the sequencing and grouping of related items in the print dictionary. Complicated but doable; SK will construct a large example tree, and we'll derive the rules from that. There's grouping and nesting, and then sorting within groups.

In some cases, the person entering data in the Lexware days used the same band name for glosses of example sentences as for defs, so we have some situations in which a def:seg duplicates data in a quote:seg. I've used this XQuery to identify candidate duplicates for SK to look at:

xquery version "3.0";
declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace util="http://exist-db.org/xquery/util";

for $t in //TEI
for $e in $t//entry
let $quoteSegs := for $s in $e//quote/seg where string-length(normalize-space($s)) gt 0 return normalize-space($s),
$defs := for $d in $e//def where string-length(normalize-space($d)) gt 0 return normalize-space($d),
$dupes := distinct-values($quoteSegs[.=$defs])
where count($dupes) gt 0
return concat(util:document-name($t/@xml:id), ' / ', $e/@xml:id, ' : ', string-join($dupes, ' | '))

I ran some queries on where we have placed <note> elements thus far, and concluded that they should be rendered as follows in the print dictionary output.

1) Cultural, comparative, and editorial notes should appear in the print dictionary, and on the website. (Internal, referToEwa, and referToElders notes should NOT appear in the print dictionary, nor on the website.)

2) Render these elements immediately followed by all the (immediate child) <note>s they contain:

entry (some editorial notes that begin with a non-English word begin with a small letter)
form (some editorial notes that begin with a non-English word begin with a small letter)
sense
def
def:seg
cit
quote:seg

3) A <note> inside a <pron> should be rendered after its preceding-sibling <seg> and <bibl> pair. If there is no <bibl>, render the note after its preceding-sibling <seg>.

Similarly, a note inside a <quote> should be rendered after its preceding-sibling <seg> and <bibl> pair, or <phr> and <bibl> pair.

4) <note>s should never appear within these elements. (I have added this reminder to the Dictionary Editing Manual.)

div
pron:seg
hyph
xr

5) I also tried to standardize the formatting of <note>s to begin with a capital letter and end with a period. However ...

-Most comparative notes have the form <note type="comparative" resp="psn:MDK">CA. {a Coeur d'Alene word}</note>. They do not currently end with a period. MDH, can you please do this globally if you think it is necessary?

-There are 4036 editorial notes in the database, of which 1688 are "Meaning unclear." and 1560 are "Root entry added based on attested complex forms." MDH, can you help narrow down the finding and fixing of initial capitals and final periods in the other 788 editorial notes??

6) I have not tried to formalize or summarize MDK or EB's language in the cultural notes. I think we should render them in smaller type, and mention in the introduction that these are reproduced exactly as collected / contributed. This is consistent with what other dictionaries have done.

7) Delimit <note>s with « ».

Had long productive meetings today looking at a range of different print dictionaries and deciding that we like the organization and structure of the Klallam dictionary best; it also accords well with what we've done so far; all entries appear individually, while there are various other lookups (English, as well as morphologically-organized nested structures). SK is creating a detailed spec for generating the list of nested items organized by morphology. I have a list of minor tasks too:

• Regenerate name dictionaries to check they're now complete (done and confirmed).
• In print dictionary, show only notes which are comparative, cultural or editorial (basically done, except that there are too many contexts in which notes appear; we need to rationalize that, and we're in the process of doing it).
• Replace Search on main menu (done).
• Figure out why "Other entries" link doesn't appear when sense element is empty.
• When there are multiple form elements, add a border and margin to the bottom so they're clearly delimited.
• In the print rendering, where there are multiple forms, any after the first are fully outdented; they should only be partially outdented.
• These are the complete contents for the introductions for name dictionaries: Animal Names, Plant Names, Personal Names, Place Names, Tribal Names, Story People. Each of these needs to be turned into a complete TEI document ready for XInclude.

The audio files for the pronunciation guide were horribly noisy, so I've cleaned up all the mp3s and then regenerated the ogg versions. Then I rearranged the file structure to get the aup files and other detritus out of the way of the bits required by the website. Tedious job, cleaning up audio.

• Generated oggs for two new audio files.
• Fixed the normalization issues with the pron page, except for one ("text run starts with a composing character"), which seems to be an artifact of the XSLT processor as far as I can see. The only remaining invalidities are caused by the non-self-closing <source> tag, which is an eXist bug that I've diagnosed and reported.
• Rewrote the table-rendering code so that the pron guide table doesn't end up being sortable by clicking on the header (which can't work because it's full of colspans and rowspans).
• Fixed a bug whereby <note>s were not being rendered when they were children of <sense> (I wasn't expecting them there).
• Fixed another bug which resulted in divs inside spans in the output, also caused by note elements. Normal pages now validate.
• Cut the site menu down to six items -- it was getting too long for a non-scrolling menu -- by moving all the dictionary pages to a sub-page called "Dictionary". Also removed Fonts from the menu, and substituted a link in the About page.
• Rewrote the PDF generation so that links to morphemes are only included in linguist output mode, so the learner dictionaries don't have links at all. Linguist dictionaries generated without related morphemes have bad links, of course, but I can probably get rid of that build pipeline.
• Rationalized the file structure a bit to segrate XML files from the PDFs generated from them.

Today I've made the following changes to the PDF dictionary generation:

• choice/sic/corr is now handled correctly.
• The linguist dictionary now sorts by orthographical headword.
• Auto-orthographization is now done with phonemic segs instead of hyphs.
• Name entries are only included if the name shows up in the pron.

I've also added a morpheme-linking feature, which enumerates the morphemes in the hyph and provides a page-reference to the entry for each of those single morphemes. Obviously this only works in the dictionaries which include all the related morphemes, but you can see it at work if you look at the names-only_linguist_with_related.pdf file.

• The output we want for the names dictionaries is the one without component morphemes, learner version.
• Linguist dictionaries should have headwords in orth form, and should sort by them.
• Auto-orths should be done from the phonemic pron, not from the hyph, to avoid the reduplication problem.
• In names dictionaries, only include an entry whose name tag is in a pron.
• sic/corr are still not working properly in the PDF output. That's because this is the template for <def>, and it does not process tags inside <seg> (so a <persName> containing a <choice> just gets output as text):

    <xsl:template match="def">
      <xsl:for-each select="seg">
        <xsl:if test="preceding-sibling::seg">
          <fo:inline>; </fo:inline>
        </xsl:if>
        <xsl:value-of select="normalize-space(.)"/>
      </xsl:for-each>
    </xsl:template>
  

  Obviously more subtlety is required here.

The audio is stored in home1t/moses/www/audio, all together in one folder. This includes aup files and their associated folders (Audacity projects), mp3s generated by CB, and the OGG files I auto-generated from the mp3s.

We're linking to them like this:

<ref type="audio" target="mosaud:filename.mp3">[word]</ref>

From that, I'm generating an HTML5 <audio> element with two <source> element children, one for mp3 and one for ogg (my Linux Firefox still won't play mp3s). I've substituted a single play button for the standard controls, which are overkill for these tiny extracts.

One problem is that eXist currently outputs the <source> elements with closing tags instead of as self-closers, and that generates an error in the HTML5 validator. I've filed an eXist bug for that.

Another problem we have is that the audio is extremely noisy. It could easily be cleaned up; the question is whether it would be better to clean up the full stretches and re-capture the individual words, or just clean up the segments we're using.

Fixed a display issue (we should show corrs, not sics, from choice elements). Other bugs exist, though, and three more dictionary types are required.

I now have an XSLT file called generate_names_only_dictionary which runs on itself and generates a whole stack of eight different XML files in dictionary_test/names. A subsequent pair of transformation scenarios (moses_xml_to_pdf_*_names_only) run on these XML files to generate 16 PDFs. These break down as follows:

• all the names
• only the fauna
• only the flora
• only the storyPeople

For each of those four, there are two different content sets:

• 1 with only the name entries
• 1 with the name entries plus all the morphemes that constitute them

For each of those two, there are 2 PDFs:

• 1 in learner format
• 1 in linguist format

So there are 16 different PDFs.

None of these dictionaries have any introductory material at all; someone will have to write that stuff, once we know what they're going to be used for and who will be using them.

I think this should cover all the options we can imagine for names-only dictionaries, and I have a two-stage pipeline for creating them so it's easy to regenerate them whenever we want.

We've had to redo two of the global transforms we did to automate some markup tasks, because they proved to have made changes not expected or wanted; this is mainly because the original Lexware encoding (and thus the resulting XML generated from it) varies so greatly from file to file that what works for one file doesn't necessarily work for another. Lessons learned. Luckily SVN helps figure out what went wrong, and allows us to recover from it.

As part of the preparation for work on the auto-hyphenator, I've generated a list of all the distinct forms of morphemes and what they link to, initially using this XQuery (which takes a long while to run):

xquery version "1.0";

declare default element namespace "http://www.tei-c.org/ns/1.0";

for $mText in distinct-values(//m[@corresp])
return concat('morpheme form: ', $mText, ' links to ', string-join(distinct-values(//m[text() = $mText]/@corresp), ', '))

and then trimming the results to remove everything that links only to m:UNASSIGNED, as well as removing links to m:UNASSIGNED from the lists of other links (should have built that into the XQuery). We can now use this list to spot candidates for auto-assignment (starting with those forms which only ever link to a single morpheme entry).

Faced with the task of creating a print dictionary consisting only of the name entries, I was initially stumped because of the incidence of duplicate xml:ids across the collection. Previously, my print dictionary processing has depended on inclusion by manual selection of only those files whose status is complete, across which there are no duplicate ids. However, the names are sprinkled throughout the whole collection, including lots of files which have not yet been edited, and therefore have duplicate ids in them.

After some thought, I set up this process, running in dictionary_test:

• First, the auto-orthography transformation is run against all the files in the dictionary directory (the live files) to create auto-orthographized versions in dictionary_test.
• A new file called master_all.xml XIncludes all the entry files in dictionary_test. This file is obviously invalid because of the duplicate ids, but it can be processed with XSLT.
• Next, a transformation called generate_names-only_dictionary.xsl pulls out all the name entries, along with all the completed root, stem and affix entries to which the name entries link in their morpheme elements, and creates from them a file called master_names-only.xml.
• Finally, the moses_master_to_pdf_LINGUIST transformation scenario is run on the master_names-only.xml file to generate the PDF dictionary.

In the process, I found and fixed a couple of errors including a duplicate id between two name entries, and also noticed a new problem we'll have to work on: the TEI Schematron embedded in the RelaxNG schema for the <gloss> element disallows the presence of @subtype when there is no @type (quite reasonably, perhaps), but we're using @subtype by analogy with what we use on <seg>, while having removed @type from the schema. I guess we should probably handle this by creating a new datatype and using @type on <gloss> instead of @subtype.

Per SMK's blog post task #4: all the duplicate defs have now been collapsed. This was done based on identity through deep-equals, so there may be some cases where near-duplicates still exist, but I don't see a way to automate the decision on whether something which is a near-duplicate should be deemed a duplicate.

Attempted to carry out instruction #3 in SMK's blog post "Global changes to speed up editing", but I can't find a single example of a situation which matches the case. After getting no results on sample files with XSLT, I ran the following XQuery on the whole db to try to find some cases to work with:

xquery version "1.0";

declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace util="http://exist-db.org/xquery/util";
declare namespace hcmc="http://hcmc.uvic.ca/ns";

declare function hcmc:cleanUpNode($inNode as node()) as xs:string{
	translate(normalize-space($inNode), '*', '')
};

for $d in //def[following-sibling::cit][count(seg) = 1]
let $seg := $d/seg
where $d/following-sibling::cit/quote/phr[hcmc:cleanUpNode(.) = hcmc:cleanUpNode($seg)]
return $d/ancestor::entry

along with some variants of it, I couldn't find a single hit, so I've given up on this one.

Tasks 1 and 2 in Sarah's "Global changes to speed up editing" are now complete. The second was a bit complicated because identifying the context for it was predicated on the existence of a hyph, but the hyph had been commented out in the preceding task (#1). However, a very messy regular expression match against the commented-out hyph did the job, after some tweaking.

NOTE: This includes major changes throughout the files, so, for the record, these changes all took place in one revision, rev 1346.

Per SMK's request on the blog.

ECH and SMK finished their editing work yesterday, so this morning I did some final tidying up (lots of minor font and formatting issues), added some extra detail to the references, and created the submission package (three different versions of the file, one with figures all moved to the end, etc.). I think we're finally done! Posted it on a server for ECH to submit, because it's too big to attach to an email.

List of tasks for August:

• Create a print output consisting of only the name entries (entries with <f name="name">/<binary value="true">) using the existing dictionary XSL:FO code.
• Work on the automatic morpheme linker routines outlined in SK's document entitled Autohyphenator. Start with simple ones and work towards the more complicated ones.
• Carry out all the tasks in SK's previous post.

I have some changes to make arising out of this, but what remains is not major.

Tasks arising out of today's meeting:

• Add MLW to the ODD/schema (personography item).
• Add handling for an <entry> with no <form> element (xml:id="unknown") [SMK adds: we will also want to suppress this "unknown" entry from the print dictionary.]
• Build a PDF dictionary with only the affix files and lex-suf (for reference by team).
• In lex-suf, and any affix files which contain matches for the XPath below, do an identity transform which in this context:

  seg[@subtype='i'][following-sibling::bibl[1][@corresp='psn:MDK']]
  

  removes the @subtype value.
• Remove traffic lights from names page (if straightforward).
• Add light-edited (yellow) and light-proofed (green) to the document status values in ODD and schema, then add traffic-light handling.
• Add all light-edited and better docs to the XInclude file and validate to catch any duplicate ids.

All ready for next week now.

I am posting this exchange about inferred glosses so that I don't have to think it through all over again in the future!

SMK wrote:

Regarding the search engine, I blogged on 12/12/12:

"ECH's goal for the search engine in the web database is that, if a user searches for "fat", s/he will get results including fat, fatten, fattening, fatty. Our current settings, and our policies for adding inferred glosses, seem to be accomplishing this nicely. An entry which has "fatty" in its def is found by a search for "fat", because it also has an inferred gloss "fat". Searching for "fat*" also returns defs including fat, fatten, fattening, fatty ... but also fatal, fathom, father."

However, we also noticed the converse on 16/04/13:

When I searched for the inflected form “fired”, I also I got all the entries with “fire”.

BUT when I search for “fatty” or “fatten”, I don’t get all the entries with “fat”. What is the difference here?

MDH replied:

I think you're just discovering that a stemming analyzer is not an educated human. It doesn't understand semantics; it just knows how to strip off (some) inflectional endings and index the resulting stems, and then how to stem the search input and search the stemmed index with it. You will never find an automated search engine that gives you perfect results.

Right now, the search is paying no attention to whether things are in gloss tags or not; as I understand it, the purpose of the gloss tags is to construct and English-Nxa’amxcin list, not to aid in searching.

The situation with "fatty" is definitely a bit odd; it appears that if you search for that word, you it doesn't get stemmed prior to the search, whereas if you search for "fired" it does. Perhaps the stemmer avoids stemming -tty inputs because there are many which shouldn't be stemmed? ("batty", "natty", "patty", for instance.)

SMK continued:

OK, so when I search for fatten, fattened, or fattening, I get the same 5 hits – 3 for “fattening”, one for “fattened”, and one for “fatten” – i.e. everything with the stem “fatten”. It doesn't go all the way down to the root “fat”, and that's fine.

When I search for “fatty”, all I get is the one entry for “fatty”, as you explained above. That's fine too.

We had been adding inferred glosses for the uninflected English stems and roots of attested glosses, e.g.

<def>
<seg>I am <gloss>fattening</gloss> it up</seg><bibl corresp="psn:W">W10.138</bibl>
<seg><gloss subtype="i">fatten</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
<seg><gloss subtype="i">fat</gloss></seg><bibl corresp="psn:ECH">ECH</bibl>
</def>

Here, <gloss subtype="i">fatten</gloss> adds nothing to the search capabilities, because the stemmer can find “fatten” within “fattening”.

But does this entry with “fattening” get found when I search for “fat” because of the stemmer, or because of the <gloss subtype="i">fat</gloss>? It must be because of the inferred gloss, because the stemmer only stems as far as “fatten”.

In the case of “fatty”, where we know the stemmer doesn't operate on it, it still gets found when I search for “fat” because of the <gloss subtype="i">fat.

(“fattening” and “fatty” do NOT get found when I search for “fat” just because they contain the string f-a-t, because “fatal” and “father” are NOT found by a search for “fat”. To find anything with the string f-a-t, I would need to search for “fat*”.)

So the inferred glosses do play a role in improving the search. That said, I don't think we should be going out of our way to add inferred glosses for this reason.

Much discussion over the last few weeks regarding the placing of gloss tags for generating the Eng-Nx wordlist. I attempt to summarize our conclusions here for future reference.

1) Why do we place inferred glosses (<gloss subtype=”i”>)?

At various times, we have placed inferred glosses for augmenting the search engine on the website, and for generating the English word list.

We concluded that from here on, we ONLY need to place gloss tags for generating the English word list. Inferred glosses do sometimes enhance the web search engine, but now that the stemming analyzer is in place, we don't need to do any further markup to help it out.

2) How should we tag inflected English words?

Until last week, we had been inferring the root word (or stem where relevant) when a def is an inflected or derived form of an English word, e.g.

<def>
<seg>he is <gloss>fattening</gloss> it up</seg>
<bibl corresp=“psn:JM”>JM 1.2.3</bibl>
<seg><gloss subtype=“i”>fatten</gloss></seg>
<bibl corresp=“psn:ECH”>ECH</bibl>
<seg><gloss subtype=“i”>fat</gloss></seg>
<bibl corresp=“psn:ECH”>ECH</bibl>
</def>

This encoding means that this entry will show up three times in the English-Nxa’amxcin wordlist: under fat, under fatten, and under fattening. This seems like overkill, especially when these three words will sort one after the other in the English wordlist anyway.

ECH and SMK decided we would like to see the “fat” entries as follows in the print dictionary:

fat: fat

fatten: fatten, fattened, fattening

fatty: fatty

To accomplish this, we need to reduce the number of gloss tags we place in each entry. Inflected English forms (-ed, -ing) should not be gloss tagged; only their root or stem should be gloss tagged.

So “fattening” would now be gloss-tagged as:

<seg>he is <gloss>fatten</gloss>ing it up</seg>

MDH confirmed that the search engine is ignoring gloss tags, so the stemmer will operate on <gloss>fatten</gloss>ing the same as it would on <gloss>fattening</gloss>. (That is, it will continue to return all results with the stem “fatten” when someone searches for fatten, fattened, or fattening.)

MDH has created two sample Eng-Nx word lists based on the 6 files with “complete” status, one using all the gloss tags, and one omitting the inferred gloss tags. They are in moses/trunk/docs/glosses. We concluded that we don't want to programmatically ignore the inferred glosses, because many of them – especially the synonyms – are worth including. But we can refer to these lists to identify the inflected English words whose gloss tags need to be revised.

3) How should we tag English phrasal verbs?

Where appropriate, English phrasal verbs will be enclosed in a single gloss tag - e.g, <gloss>go after</gloss>. This will allow us to organize the headwords in the Eng-Nx word list as follows:

go

go after

go down

go up

, etc.

4) How can we distinguish English homophones in glosses?

English homophones in glosses will be distinguished with a secondary word (or phrase) in an @n attribute on the <gloss> tag, e.g.<gloss n="conflagration">fire</gloss>, <gloss n="back of boat">stern</gloss>. These will then be rendered as follows in the print dictionary:

fire (conflagration):

stern (back of boat):

We decided not to use parts of speech for @n values. We will always use synonyms. We need to select synonyms that will be clear to readers in the community.

I have now disambiguated the English homophones listed here, and updated the Notes on Definitions and Gloss Tagging document accordingly. Where one homophone was far more common in the data than the other, I only added an @n value on the less common one - e.g. watch (wristwatch).

Did this through XSL with some cunning language-detection code based on content and context, and it seems to have worked pretty well. The Names page now uses the @xml:lang attribute instead of its own cruder detection code to build output.

As planned last week.

Meeting to review the presentation -- my task now is to collapse six slides which begin with the picture of the filecard box into a single stepped diagram illustrating the old encoding process and the horrible binary result.

Section 2 is now down to 6 slides, with more detail and more extensive notes.

Following Sarah's post, I've done the following:

• Added a language filter so you can view names only in English or Nxaʔamxcín. This is a crude regex, but it works because English names always begin with caps, and Nxaʔamxcín names never do.
• Turned off the traffic light display in the names page.
• Added more processing to the path, to handle rendering of e.g. choice elements inside names.
• Excluded lexical suffix entries.
• Elaborated the captions and links a bit.

We've been running the live db with open access since the last time I rebuilt it, so in the process of doing other updates (such as rolling out the Java sorting collations) I've also added back the protection that we had before. In the process of doing this, I got bitten by the horrible eXist bug which enables you to lock yourself out of the admin account if you edit the admin user and forget to retype the password into the two password boxes (the effect is that you end up with a random admin password that you can never discover). As a result, I had to remove the server version of the app and replace it with a refreshed version of my local copy. This failed the first few times -- Tomcat tries to auto-deploy the app before it's completely uploaded the dbx files, so the uploaded .filepart files can not be renamed to overwrite the ones created by the live startup. It took two or three shots to get this problem solved. The only way seems to be to let it deploy, but stop it immediately in the Tomcat manager; then delete all the dbx, lock and log files; then upload them again; then restart it in the manager.

1) For the linguists' dictionary, we would like to see:

first phonemic representation in bold <orthography in angle brackets> [narrow transcription(s) in square brackets], for both forms and cits - e.g.:

ʔáyx̣ʷt <ʔáyx̌ʷt> [ʔáyəx̣ʷt]
√ʔáyx̣ʷ-t
1. be tired
2. tired, worn out

• √ʔáyx̣ʷ-tl kɬʔámnc
<√ʔáyx̌ʷ-tl kɬʔámnč>
[√ʔáyəx̣ʷ-t ləkɬəʔámənč]
he is tired of waiting (for you / me)

2) On the website, we would ultimately like things sorted by orthography.

This morning we decided that a simple and quick way to distinguish between homographs with different meanings is required to make the English lookup part of the dictionary less confusing. This will be achieved by adding a clarificatory word or phrase in the @n attribute of a gloss. Glosses will then be presented in the E-to-M view with this clarification in parentheses. Processing on the website will need to be changed to take account of this, and the print dictionary rendering will also have to be written with this in mind.

The problem of duplicate @xml:id attributes on entries has now become a serious issue for the print dictionary building, because I'm unable to properly process the entire collection properly to produce the book; to build the dictionary I have to use XInclude to create a single XML source file, and when I do that there are over 1600 duplicate ids which prevent some of the processing steps from being successful.

I've taken a quick look at where the duplicates tend to be concentrated, by adding the files in alphabetical order and looking to see how many duplicates occur with each addition. These files create no problems (i.e. they have no duplicates among themselves):

affix_glot-ix.xml
affix_k-m.xml
affix_n-t.xml
affix_u-CAPS.xml
c.xml
c-glot.xml
c-rtr.xml
glottal.xml
h.xml
h-phar-part1.xml
h-phar-part2.xml
l-affric.xml
lex-suff.xml
new-data-2013.xml
p-glot.xml
phar-w.xml
qw-glot.xml
s-rtr.xml
t-glot.xml
xw.xml

When I add the remaining files, one by one (and only one at a time), these are the results:

k.xml            100 duplicates.
k-glot.xml:         18
kw.xml:              2
kw-glot.xml:          2
l.xml:              3
l-fric.xml:          6
m.xml:              3
n.xml:             97
p.xml:              7
particles.xml:          4
pron.xml:          2
q.xml:              4
q-glot.xml:          3
qw.xml:              1
rescued.xml:         54
s.xml:              2
t.xml:             20
ww-glot.xml:          4
x.xml:              3
x-uvul.xml:          4
yy-glot.xml:          4

What I'm going to do is develop the dictionary output using only the valid files, and then add the others in as they get fixed. In the meantime, it might be worth having a go at some of the low-hanging fruit (the ones with only two or three duplicates). More will show up as we add those in, of course -- there will be duplicates across the currently-excluded files as well as those that they share with the "good" files. So the dictionary PDFs will shrink in size, but I'll be able to start doing things like generating page-references that depend on xml:ids.

I've created a new MosesPhonemicCollation jar for sorting based on the phonemic representations. I've also forked the dictionary build process based on a parameter called "dictionaryType", which can be "learner" or "linguist". The former produces a dictionary based on the orthography, sorted with the MosesOrthographyCollation, and the latter produces one based on the phonemic transcriptions, with the new collation. The "alphabet" guides that run across the bottoms of pages are also appropriately different. I've abstracted the front matter into a separate file, and I'm auto-including the personography, although I'm not processing it yet.

The idea of having a single collation to sort everything in our db is now impractical, because the orthographical sorting rules clash with the transcriptional sorting rules, so I've created a new, simpler MosesOrthographyCollation class for sorting the orthography only. It's working well, but there are still some outstanding questions about it. In the meantime, we can't update the website because we don't have orthographies there yet, so this is only going to be used in for the print dictionary generation.

This has had to be redone a couple of times due to changes in the list of glyphs, but it's working now and tested with the print dictionary system.

We concluded that we need a different alphabetical order for the community dictionary vs. the linguists' dictionary.

The community dictionary should indeed follow the order in the 2006 language program dictionary - that is:

a aa ə əə č c cʼ h ḥ ḥʷ i ii k kʼ kʷ kʼʷ l lʼ ll llʼ ɬ ƛʼ m mʼ n nʼ p pʼ q qʼ qʷ qʼʷ r rʼ š s t tʼ u uu w wʼ x xʷ x̌ x̌ʷ y yʼ ʕ ʕʼ ʕʷ ʕʼʷ ʔ

The linguists' dictionary should follow the order in MDK's 1981 dictionary:

ʔ a ạ c c̣ cʼ ə ə̣ h ḥ ḥʷ i ị k kʼ kʷ kʼʷ l ḷ lˀ ḷˀ ɬ ƛʼ m mˀ n nˀ p pʼ q qʼ qʷ qʼʷ r rˀ s ṣ t tʼ u ụ w wˀ x xʷ x̣ x̣ʷ y yˀ ʕ ʕˀ ʕʷ ʕˀʷ

Today I got the following bits working:

• Running headers showing first and last items on dictionary pages.
• Front matter with independent numbering in roman numerals.
• Footers on dictionary pages showing alphabetical order (the order itself isn't yet finalized).
• Correct alignment and page-numbering for recto/verso pages (I'd screwed up on that).

Next is the implementation of the English-Moses glossary.

Proofing and rewriting. I've now finished section 4. The conclusion remains to be written, and the intro will have to be reworked at the end.

Worked through sections 1 and 2, and into 3, merging previous changes and suggesting more.

I now have some basic rendering for definitions and examples, so we're getting closer to something that looks like the final product will look. There are two chars missing from the fonts, but the author of the fonts may be able to add them for us (yay!). Other than that, things are looking good.

The author of the Aboriginal fonts kindly fixed the r-with-caron rendering problem we'd identified, within an hour of our reporting it, so it looks like that will be our font of choice. Tested and working now.

After looking with SMK at the way some characters are rendering in the PDF, I've created a function called hcmc:renderingFixups() which does some character substitution. Specifically, we're replacing i-with-dot-below + combining accent aigu or grave with i-with-accent plus combining dot below; and a similar thing with i-with-short-stroke. This last may not be as pretty as we hope, and there are still outstanding problems with the dot below m (off centre to right) and l (off to the left). We may be able to fix the latter by flipping briefly to another font, but that's very ugly. Still, these are minor issues, and so far CharisSil is working well for us.

SMK suggested CharisSIL, which I've downloaded and tested with XEP; it looks good. I'll now try integrating that into FOP. I've also updated the Collator after discovering that we had not included barred lambda without a following apostrophe, causing one (actually erroneous) example of that to be sorted to the beginning, a mystifying thing for a while. I've made a little more progress with the rendering of entries, but I don't have a model to work from yet so it's just exploratory.

Up to now, we've been using the excellent Gentium Plus fonts for our dictionary website, and they're working well; they're attractive and cover all the glyphs we need.

However, GentiumPlus comes only in regular and italic flavours; there's no bold version of the font. When we use "bold" on the website, the browser renderer automatically fattens-up the regular font to give the impression of bold; it's not particularly pretty, but it works. Unfortunately, that can't be done when generating the PDF. Neither Apache FOP nor the commercial XEP generator we use has the capability of automatically generating a bold version of a font from a regular version (and they would presumably argue that it shouldn't be done, because it's ugly, and would be especially noticeably so in print). So we're faced with three choices in the print dictionary:

• Eschew bold text altogether. This is not ideal, because dictionary entries depend for their readability on the textual differentiation of lots of information bits which appear next to each other.
• Choose a different font, one which has both bold and italic variants, for the whole dictionary.
• Choose a different font only for bits of the text which need to be bold.

We'll have to think carefully about this. I've written to the team for thoughts and suggestions.

Just reminding myself: I need the MosesCollation.jar file to provide the sort collation for the dictionary, and in eXist this is found automatically as long as it's in the WEB-INF/lib directory. However, locally, I have to add it manually to the transformation scenario -- on the first tab, click on Extensions, then Add, and find the jar file.

I've finished the first draft of my bits -- over to ECH...

Redrafting the third part of the article encouraged me to do even more refinement of the schema; I've now nailed down the use of @type and @subtype more thoroughly, and tweaked the encoding of <gloss> as a result. It uses @subtype="i" instead of @type, for consistency with <seg> and <phr>, even though it makes no use of @type (at the moment). The article is coming along, and I hope to get the first draft of my bits finished tomorrow.

Built a fresh checkout of eXist trunk, and reworked the app to run in it. Had to change relative paths to XSL files in XQL files to full paths from /db for some reason. Also fixed a bug in the personography rendering, which after much confusion turned out to be caused by my having moved the schemas around in the db. Only thing left to do: add password protection.

And better tested and documented. I think it's finished now, but we've also learned enough to be able to update it easily. I also learned that if I compile it targetted at Java 1.7, it will cause errors on Pear, so I compiled it for 1.5.

It's remarkably fiddly to get all this stuff right.

This took way longer than I expected, and I ended up resorting to tables to keep the layout under control. Not ideal, but nothing else would work.

Sarah says:

I think we decided on Monday (talking with Ewa) that you should go ahead and use the full def:segs instead of the glosses in the "get related words" and "other entries containing this morpheme" lists. The glosses are only intended as headwords for the English-Nx word list.

Our example was cìqqnúnn, with the def:seg "I accidentally dug up something". The gloss tags are "dig", "dug", and "accidentally". We want the Nx word to appear (with its def) under all three headwords in the English-Nx word list, but we don't want a learner to look at the "get related words" list and think the word can be used to mean just "accidentally".

I've moved the NetBeans project for the RuleBasedCollator class MosesCollation into our SVN tree, and then I rewrote it to include orthographic characters and English (upper and lower case) so that we can sort all three types of string using the same collation.

I had to do this twice because the current version of NetBeans from the Precise repo, which was 7.0.1, has the most disastrous bug imaginable: no file changes are saved to disk. This is undetectable while you're running it, because file buffers are changed, and jar files are built from the buffers. I lost all my work to this bug, and had to repeat it after removing the repo NetBeans and installing 7.3 from the download installer.

Discussion of print dictionaries, collation for orthography, vacations and timing, and other issues.

Opening old NetBeans 6 projects in NetBeans 7 resulted in errors because the JUnit libraries couldn't be found. I had to right-click on Test Libraries, choose Add JAR/Folder, then choose /usr/share/java/junit4.jar to resolve the missing dependency. Took a while to figure that out, curses...

We sort our Moses entries currently based on the phonemic representation, using a Java comparator I wrote specifically for the project. Now we're going to have orthographical representations, the sort order will have to be amended to take account of that. I'm therefore reviving the NetBeans project for the MosesCollation, and beginning to update it.

This is the current sort order.

We'll need to add the following characters to the list:

• č (u010d) (does it sort before or after c?)
• š (u0161) (does it sort before or after s?)
• x̌ (x + u0323) (does it sort before or after x?)

I'm now trying to get the Gentium Plus font working with FOP, to handle our non-ascii characters. It is possible to use the "simple method", which involves giving the FOP processor a special config file telling it to parse the system fonts to find the font it needs. However, because I want to be able to make PDF generation a part of the portable webapp, I need to do it in the "hard" way as well, so I've started figuring it out.

The process is confusing, because FOP behaviour has changed. It seems to involve three steps:

• Generate a font metrics file for each font you want to use. This was quite hard, because I was trying to follow these instructions, which are out of date, as I reported to the Oxygen forum here; this is what I eventually had to do, at the command line:

  java -cp lib/fop.jar:lib/avalon--framework-4.2.0.jar:lib/xercesImpl.jar:lib/commons-logging-1.1.1.jar:lib/commons-io-1.3.1.jar:lib/xmlgraphics-commons-1.5.jar:lib/xml-apis.jar org.apache.fop.fonts.apps.TTFReader /usr/share/fonts/truetype/gentium-plus/GentiumPlus-R.ttf GentiumPlus-R.xml
  
  java -cp lib/fop.jar:lib/avalon--framework-4.2.0.jar:lib/xercesImpl.jar:lib/commons-logging-1.1.1.jar:lib/commons-io-1.3.1.jar:lib/xmlgraphics-commons-1.5.jar:lib/xml-apis.jar org.apache.fop.fonts.apps.TTFReader /usr/share/fonts/truetype/gentium-plus/GentiumPlus-I.ttf GentiumPlus-I.xml
  

• Creating a FOP configuration file which tells it where the fonts are.
• Calling FOP and pointing it at that configuration file.

Still working on the second and third steps...

Wrote, tested and delivered a utility XSLT file for handling problems in the glottal.xml file, per SMK's request.

I've started work on the XSL:FO/PDF generation code, working with some test files for which I've auto-generated the orthographies. I have a basic layout done, for letter-sized paper, and a parameter system built in which enables me to add other paper sizes later. I'm working with FOP, because if we can get what we want with it (and it's looking good so far -- columns work) then we can deploy anywhere. The biggest hurdle right now (after reminding myself of how page-masters and sequences work) is getting the Unicode characters to display correctly. That's next on my list.

More refinements to the orthography generation. We've now decided that we should base the orthography on the hyph rather than the pron, because that way we can perhaps insert intrusive schwas more easily; morpheme boundaries appear to be significant for this purpose. I'm also orthing the phonemic phrases in citations, and since these are partly hyphenated, I'm handling them slightly differently, splitting on the morpheme boundaries, but using the same conversion code. I have a hook in for the schwa insertion if we can formalize the rules for it.

Wrote the orthography-generating XSLT, and tested and refined it with SMK working on the l-affric file. The only remaining outstanding questions are: what to do with dotted n (change to nn, or keep as-is, or remove the dot); and how/whether to insert the extra schwas we see in older examples of the orthography.

Also tested some XQuery to determine how practical it will be to link some morphemes in hyphs automatically to their source morpheme. There are many instances where a particular string has only one existing morpheme link, so there are lots of candidates. My XQuery could be used to build a lookup table for all instances of a string which has a single existing corresp, and we could use that to auto-link a lot of m elements.

Continued refining the names page, in response to feedback, including adding print settings in the stylesheet to hide the menu stuff. Also added counts of entries into the status page, so that we can see where we're at.

On the plane yesterday I worked on the rewrite of the Names page, which is now working. It's a sortable table, showing all things tagged as names, along with links to their entries. In the process I brought in some JS for the table sorting, and modularized some of the entry link display code. I also found some bits of code which look obsolete, that I might be able to get rid of; I should do that asap. Uploaded the new code this morning, then worked on a bug with the menu display caused by my changing some of the GET params.

Had to move back to LibreOffice 3.6 because of a major bug in handling of custom animations in 4.0. My bits are now done, and ECH is finishing the last parts of hers.

Finally managed to get some data into LEXUS, and out in XML format -- it's pretty ugly, and seems to be in its own namespace, although it nods at LMF. Did more work on the presentation (new diagram). I think my bits are basically done.

Managed one more diagram today, along with some thinking and ideas for showing how ODDs work.

Creating photos, diagrams and screenshots, and putting the first half of the presentation together.

All our current values for <person>/@role have now been documented in the ODD file, and I've also put a system in place for reading that data out of the ODD file at transform time (using <xsl:document>) and using it to create the web display text. This will be handy in future, and it's a good approach because it encourages us to do more documentation, in more detail, in the ODD file, with the sense that it's going to be made public. The ODD file is now in the db, of course, which is also probably good. It might make sense to put the rng file there as well, and make both forms of documentation available from the site.

Added a new xml file to the svn repository. It's called new-data-2013.xml, and it's for new words PCS contributes, beyond her comments on words already in the data.

Now that I am working on the feature structures for the affix files using the new feature system I am noticing changes that are needed. I have documented them in comments at the top of the feature system file. (3) is implemented already

1. Need to add new daughter to name="affix", i.e. value="syntactic"
2. Need to add new fDecl name="syntactic" category, and list of symbol values that are daughter of "syntactic"; one of these latter is going to be "interrogative" (it is a type of enclitic particle with syntactic function).

3. Moved attributiveHabitual and repetitive to secondaryAspectual category.

Please note: The categories are determined on the basis of grammatical function and meaning. Therefore, for example, the outofControl morpheme is included with Control morphemes rather than with Reduplication on the basis of its function as a control marker rather than on the basis of its formal characteristics as a reduplicative morpheme. For the most part the categories reflect those provided in Nxa'amxcín grammar of Marie Louise Willett (2003), which in turn is based on all the work of Dale Kinkade.

Finally finished my bits of the paper relating to feature structures and interoperability. Now I can focus on the presentation...

MDH has posted a list of entries whose hyphs include verb morphology at:

http://pear.hcmc.uvic.ca:8081/moses/site/wordlist.htm?fromLang=moses&show=verbs

Here are the criteria we used to generate the list.

A verb entry's hyph will include one or more of:

m:n-CTL

m:t-TR

m:min

m:nun

m:nwˀaɬn

m:stu

m:tuɬ

m:ɬ-DIR

m:xit

m:cut

m:waxW

A verb entry's hyph will NOT include any of the nominalizing morphemes:

m:s

m:sxWs

m:ul

m:min-INST

I've also implemented "Get related words", which is more subtle than getting all the entries for a morpheme, and I've cleaned up some other iffy code and extended the schema a bit.

Had to resort to removing all the existing link elements following the first (main) stylesheet, then inserting a new link for the supplementary stylesheet. Also had to eschew the use ofd @title on link.

I have the cookies and session settings working OK, and abstracted to a little module, so that will be handy in future. I also have the widget controls on the page working, and retaining the correct settings. Now the problem is that I'm trying to turn stylesheets on and off using their "disabled" property, but it's really not working at all. Both Firefox and Chrome have problems, but they exhibit different behaviour. The spec says I can write to the disabled attribute, but it doesn't seem to work on Chrome if the stylesheet was disabled initially in the HTML code; with Firefox, the first disabled stylesheet seems to be active when the page first loads. Frustrating and annoying. I may have to resort to deleting and inserting link elements in the header, which is very crude, but might actually work.

Wrote, tested and deployed some XSLT that converts or comments out some now-obsolete types of feature structures.

Made a start on three different views of entries. I have pipelines set up, and need to tweak the XSLT for each case; following that, I have to decide how to save the setting (session variable, cookie or URL parameter, or combination).

Met for most of the day with EB, SC and the rest of our team, and made some sound decisions about different views of the database. These are some notes:

Dictionary views

Elders' view (perhaps not needed, because EB and SC interface between them and the dictionary):

• Orthographical form
• Meaning
• Hyphenated form

Learner's view:

• Orthography
• Sound (EB has recordings of contributors K and A, as well as Nellie Moses Friedlander. Or perhaps PC could record the entries.)
• Phonetic (an entry could have more than one pron:seg type="n")
• Meaning
• Hyphenated form
• Usage
• Paradigm (for verbs)
• Related words: all with the same root or stem
• Cross-references: <xr>s

Print dictionaries

• Both directions
• Moses-English organized by root, but with prefixed versions included with "see root".
• Additional front matter? Grammar background information per Nancy Mattina's work in latest community-published dictionary? Charts of subject and object suffixes (where possible, using the same root with different transitive markers)?

Informants

We should gather and include information on informants (genealogy, dates, etc.) so that we can generate relationship info and trees, and provide a good credits page if we need to, but keep that info confidential for the moment. One source for info on deceased participants would be here, especially the St. Mary's Mission, Nespelem City Cemetery, Little Nespelem Cemetery, and Nespelem Catholic Cemetery (Sacred Heart?) records. Other cemetery records which might be helpful are Cashmere Cemetery and ____ Creek Cemetery in Monse.

Extra features

Portal page - with photo of Moses Mountain from EB? The web database should have the same title as the community's print dictionary: Nxaʔamxčín Nwwáwəlxtənt. EB and SC will confirm the spelling with PC.

A good map of the area, on which we could also put the placenames and link to their entries, would be good.

Names need to be divided into personal, place, story characters, orgs, flora and fauna. For flora, we could try to get a copy of Ethnobotany of the Southern Okanagan. Personal names will eventually be hidden in entries.

Generating orthography

• Start from the hyph.
• c goes to c-wedge.
• x-dot goes to x-wedge.
• any other dotted character loses the dot and is doubled.
• Some schwas will be inserted, rules to be determined.

Met with ECH and SMK to rework the whole feature structure thing, and we went back to a flat model. We'll work with a flat model until we're sure everything is functional and accounted for, and then we'll start adding structure and dependencies as appropriate. This seems the cleanest approach, and will let us get productive again. I've encoded a clean new version of the feature system with no @dcr: attributes or comments, and we'll go from there. I'll start adding back the @dcr attributes soon.

Following this morning's discussion of feature structures, we are no longer treating names and loanwords as baseTypes (since names, and potentially loanwords too, can be multi-morphemic).

The feature structure for names is being changed from

<fs>
<f name="category">
<symbol value="proper-noun"/>
</f>
</fs>

to:

<fs>
<f name="name">
<binary value="true"/>
</f>
</fs>

Monomorphemic names will also have the <fs> of a root:

<fs>
<f name="baseType">
<symbol value="root"/>
</f>
<f name="name">
<binary value="true"/>
</f>
</fs>

The feature structure for loanwords is being changed from:

<fs>
<f name="baseType">
<symbol value="loanword"/>
</f>
</fs>

to:

<fs>
<f name="loanword">
<binary value="true"/>
</f>
</fs>

A loanword stem entry would have this feature structure:

<fs>
<f name="baseType">
<symbol value="stem"/>
</f>
<f name="loanword">
<binary value="true"/>
</f>
</fs>

Spent the day tidying up the last of the things on my list. I've implemented authentication through eXist rather than Tomcat, which makes things cleaner in the deployment department, but slightly more messy when it comes to users, groups and permissions in the db itself. I've also got the Snowball analyzer working with English stemming, and it works out of the box, with syntax highlighting and everything. Things I'll need to remember for the next one:

• In controller.xql, when doing a dispatch/forward to an XQuery library, url parameters are passed on automatically. If you erroneously pass them explicitly as well, you get an array in the XQuery.
• CSS background-size sometimes fails on Chrome with an SVG image if the root <svg> element doesn't have preserveAspectRatio="none".
• To use the Snowball analyzer, in collection.xconf:

  <lucene>
              <analyzer class="org.apache.lucene.analysis.snowball.SnowballAnalyzer">
                  <param name="name" type="java.lang.String" value="English"/>
                  <param name="stopWords" type="java.util.Set">
      				<!-- using set from StopAnalyzer.ENGLISH_STOP_WORDS_SET -->
                      <value>a</value>
                      <value>an</value>
              [...]
              </analyzer>
  </lucene>
  

Today I ported the old Cocoon app to a fresh build of eXist, and got everything working. There are two or three little niggles to sort out, but basically everything is up and running. In the process, I've cleaned out a whole pile of obsolete code, cleaned up a lot of things, and simplified a bit. Another few hours and we'll be ready to deploy this as a replacement for the current app, which is pretty flaky sometimes, and which I suspect is responsible for the death of Tomcat once in a while.

Notes for future ports:

• Don't forget to put any jars you created into WEB-INF/lib.
• Edit controller-config.xml to set up your "/site" location for the app code.
• There's still a bug with the passing of duplicate parameters into XQuery functions. I suspect the add-parameter thing is unnecessary, and params get passed through automatically -- have to test this.

"Questions for elders" now come with an email link that pre-populates an email message with the id and the question.

Meeting to discuss what we'll do during the visit of the CT folks next week. Agreed on a dinner on Sunday, presentation of the db on Monday morning, and discussions re orthography, dictionary components, a user view of the db, etc.

Aspect

Need to make some decisions about how to indicate transitive/intransitive distinctions here. The categories in green are the affixes that need to be added to the feature system.

Imperfective-aspect ʔac (allomorph c)

This aspect prefix is used with transitive stems

Imperfective-aspect sac (allomorph sc)

This aspect prefix is used with intransitive stems and cooccurs with -mix. However, there are a few examples where it shows up on transitive stems (see MLW 2003: 309).

Imperfective-aspect s-IMPF

This aspect prefix is used with intransitive stems and usually cooccurs with -mix; it is not clear whether its meaning/usage is different from that of sac-. There are a few examples where it shows up on transitive stems (see MLW 2003: 309).

Imperfective-aspect ʔas This aspect prefix may be composed of a combination of ʔac- and s-IMPF. It occurs on transitive predicates and is rare (see MLW 2003: 310).

Nonperfective-aspect mix-IMPF

This aspect suffix is used with two non-perfective constructions: imperfectives (prefixed by sac- or s- according to MDK 1982) and irrealis/unrealized forms (prefixed by kas- (i.e. kaɬ-s)).

I'm pretty sure we're going to be able to have nesting/hierarchy both in our feature structure declarations and in our feature structures inside entries. I have one possible simple instance relating to clitic particles which will make a good test case, if ECH confirms what I believe to be the intended analysis based on her text.

Following ECH's analysis of the feature structures, I'm refactoring them, breaking out the general categories into smaller ones. Once that's done, we can see about how we might recombine them to create feature structures at higher levels, both in the declaration and in the documents.

There's a proliferation of groups of bibls which apply to the same preceding item (seg, phr, etc.), so I've written some XSLT to collapse them to a single bibl. My plan to move from bibl/@corresp to @resp on the preceding element has been thwarted by the fact that <phr> cannot take @resp. I'll need to re-think. I don't like <bibl>, which doesn't really make sense, and for quotes we can use @ref instead, but I'd like to use a unified mechanism for <seg> and <phr>.

Read through the latest version of LR's paper and sent comments. Now working on ECH's feature structure paper, and it looks as though we're going to break down our feature/value sets into smaller groups.

I went through ECH's document, Moses Feature System.odt, and noted several cases where we may need to further refine the feature system.

Is it our goal that every affix (well, besides lexical suffixes and lexical prefixes) have a unique feature structure? If so, we will need additional features to disambiguate the following:

- the applicative suffixes ɬ-DIR, tuɬ, and xit. (MLW 2003 could not identify differences in usage for these sufffixes.)

- ɬ-DIR and ɬ-EP can be distinguished with the symbol values "applicative" and "external-possession", respectively, but their meanings overlap somewhat. They are currently combined into one entry, "ɬt" in the database.

- all locative prefixes (kat-LOC, k-LOC, kɬ-LOC, kʼɬ-LOC, n-LOC, niʔ, t-LOC)

- the directional prefixes (ʔal, c, lc)

- the nominalizer-instrumentals min-INST and tn.

- the "count" prefixes: kʼɬ-DER (for counting sacks or bags), kɬ-DER (for ordinal numbers), and k (for counting people)

- the irrealis mood prefixes: kas for verb stems, and kaɬ-PR for noun stems. (Should these be allomorphs within one entry? Or two entries, cross-referenced?)

Snippet for same, producing tab-delimited data for pasting into spreadsheet:

declare default element namespace "http://www.tei-c.org/ns/1.0";

for $e in //entry[descendant::symbol[@value='proper-noun']]
order by $e/@xml:id
return concat($e/@xml:id, '	', $e/ancestor::TEI/@xml:id, '.xml')

Snippet to generate data for pasting into spreadsheet:

declare default element namespace "http://www.tei-c.org/ns/1.0";

for $t in //@target[parent::ref[ancestor::xr]]
let $id := substring-after($t, 'm:')
where not(//entry[@xml:id=$id])
order by $t/ancestor::TEI/@xml:id, $t/ancestor::entry/@xml:id
return concat($t/ancestor::TEI/@xml:id, '.xml	', $t/ancestor::entry/@xml:id, '	', $t)

To make this (and other things) simpler, I added @xml:id attributes to the root elements of all our files. For the record, to save a couple of minutes, here's the code to get duplicate ids for pasting into a spreadsheet. Don't forget to set the number of results high enough.

declare default element namespace "http://www.tei-c.org/ns/1.0";

for $id in distinct-values(//entry/@xml:id)
let $c := count(//entry[@xml:id=$id]),
$docs := if ($c gt 1) then //TEI[descendant::entry[@xml:id=$id]]/@xml:id else ()
where $c gt 1
order by $id
return concat($id, '	', $c, '	', string-join($docs, ', '))

Wrote some XSLT to generate new @xml:id values from the phonemic rather than phonetic representations, and ran it on all the unedited or rescued documents. This saves the editor a lot of work, but actually has resulted in an increase in the number of duplicate ids across the project, which is now pegged at 1401. Gawd.

SK found a new informant, VH, so I've added her to the odd file and schema.

For the record, this is what I did to test the Autophonemicizer on 21Dec12:

My test file was l-affric.xml.

I checked the first few already-edited entries, and no problems had been introduced by the autophonemcizer.

Then I carefully checked the transformation on lines 1811 to 2596, and cut-and pasted the correctly transformed pron:segs and hyphs into the main file. (I already did the quotes in this file on 13Dec12.) There were a couple of problems due to typos in the inputs, but nothing that we could have predicted or improved.

Then I pasted the rest of the autophonemicized file into the main l-affric file, so it is done.

Martin ran the autophonemicizer on all files with status="unedited" and status="rescued" on 24Dec12. Besides l-affric, the ones with status="editing" and status="additions_needed" are lexical suffix and affix files. In these, the prons are all already edited, the hyphs have been commented out, and the quotes will eventually be commented out too, so there is nothing to be gained from autophonemicizing these files.

On SK's instructions, I've run the autophonemicizer on all files with @status="unedited" and "rescued". Everything validates, and I've committed to SVN and uploaded to the db.

The autophonemicizer now handles prons, hyphs and quotes, and seems to be working well. One final tweak might be to re-generate the @xml:id of the entry based on the phonemicized hyph as well, but we may want to leave that until later, when we can just more reliably use the first seg in the pron.

SK will test, and if she's happy, she'll send me a list of files this should be run on, and I'll run it on Monday.

We had a common situation in unprocessed files in which the same transcription would appear in multiple segs in the same pron, with different bibls. I've written XSLT to detect and collapse these instances, and after some testing and tweaking we ran it on the whole collection today. This puts us in a better position to do phonemicization of the segs, because we don't have to allow for duplicates during that process.

Phonemicization has also been extended to deal with R, and we've realized that we can add a phonemic seg based on the existing hyph (text content), so the code that already works for phr should be usable. I have to finish that bit, and also rewrite the phr stuff so that it does some collapsing of its own at the end of the process.