Nxaʔamxcín (Moses) Dictionary Blog

I've been trawling through the Moses XML files trying to figure out a useful structure for the XML, and I'm realizing that the way we've been working -- moving files between directories to indicate their status -- is going to be an unhealthy way to work in subversion. Every time you move a file, you actually have to do an "svn remove" command to remove the file in its original location from tracking, and an "svn add" to add it into its new location. This will be tiresome and error-prone.

We really ought to be making use of the TEI revisionDesc element to track the changes we make to files.

Each change we make to the file would be documented in a <change> element, and the <revisionDesc> element itself has a "status" attribute which we can use to track what stage each file is at. I can fix it so that the web application looks at this status attribute to decide whether to display the content of the file or not, which means that all the files could always be stored in the database; we wouldn't have to put them in a special folder to "publish" them.

All the files would stay in one folder, and wouldn't have to be moved around; we could open any file to find out its situation and status.

So this is the proposal:

• Go through all the files in these folders, and identify _which_ of the folders each file should actually be in, based on its status:
  • rescued
  • tei_for_xform
  • tei_xml_ECH
  • tei_xml_done
  • tei_xml_editing
  • ready_to_edit_xformed
  I noticed that I have multiple copies of various files in these folders, so we need to establish which is the current version.
• For all of the files, create a <revisionDesc> element with its @status attribute showing one of the following values:
  • status="unedited" (files from ready_to_edit...)
  • status="editing" (files from tei_xml_editing)
  • status="rescued" (file from rescued)
  • status="edited" (files from tei_xml_ECH)
  • status="complete" (files from tei_xml_done)
• Merge all these directories into one, called "xml", containing all the files.
• Use this to create the Subversion repository (which will also include the cocoon code, documentation, schema etc.).

After that, I'll be able to upload all files into the db without worrying about what status they're at, because the db will only "publish" files which have a status of "complete". ECH will be able to do a find-in-files to discover which files have status="edited", meaning that they're awaiting tweaks and approval from her, and SK will be able to do the same to determine which files she's currently working on. When I run XSLT transformations, I'll be able to target the transformations at files with a specific status value.

Every time a file is edited, we'll add an entry to the top of the <revisionDesc> element explaining the changes we've made (very briefly, unless there's a good reason to go into detail).

Waiting for approval from the team before moving forward with this.

Here's an update on the status of the files in the tei_xml_editing folder:

affix.xml
-To do: proofreading against MDK cards from line 1870, phonemicizing throughout, checking questions in Comment tags.
-Add entries for missing affixes, per blog posts 1-4 Mar 2010. Go through affix paradigm sheets to determine which ones still need entries, and add entries for these affixes. CHECK ALSO pron.xml before creating a new entry!

h-phar-part1.xml
-Done by SMK and ready to be checked by ECH, but we'll wait 'til I have finished h-phar-part2 and combined the files again.

h-phar-part2.xml
-Ready for editing by SMK. Needs to be combined with h-phar-part1 when completed.

lex-suf-new.xml
- This file contains entries SMK created for 4 lexical suffixes which could not be found in the main lex-suff.xml file. We subsequently realized that many lexical suffix cards had never been entered into the Lexware database, so the next step here is to enter the rest of those cards. Then, if further lexical suffixes are still unaccounted-for, we can create new entries for them. CHECK ALSO lex-suf-nom.xml before creating a new entry!

lex-suf.xml
-ECH or SMK need to phonemicize all the examples.
-This file still needs to be checked against the Lexware printout.
-ECH needs to check SMK's work.
-The file needs to be proofed against MDK's cards, and the missing lexical suffixes need to be entered, as noted above.
-CHECK ALSO lex-suf-nom.xml before creating a new entry!

particles.xml (FORMERLY KNOWN AS affix-part.xml)
-This file contains only the first two entries from the particle card file. These two entries need editing, and the rest of the cards need to be entered. We will enter just the <form> and <def> information for the particles, not the dictegs, because the same dictegs appear elsewhere in the data (with their root morphemes). Later on, we will search for the examples of each particle, tag them, and link them to the particle entries programmatically.

qw-glot.xml
-Edited by SMK as far as line 469. The rest of the file is ready to edit.

rescued_final.xml
-This file contains the entries MDH recently rescued, which had been lost in an earlier transformation of the data. So when SMK comes across a missing entry in an alphabetical file, she can search for it in this file and paste it into the correct place in the alphabetical file.

These two files were left in a state of partial conversion after being partly edited before we wrote some universal conversions that automated some of the editing. We've now carried out all the required conversions on these files to bring them to the same state that the rest of the files awaiting editing are at. In the process, I discovered that in the case of gloss phrases which had multiple asterisked words, only the first was being tagged as a gloss, so in the file collapse_forms_etc.xsl, I abstracted the code which tags glosses to make it into a recursive template which would tag all glosses. This now means that there may be some instances of multiple glosses untagged in the bulk of the files awaiting editing; if that's the case, then I can extract that template and make an identity transform with it to apply the same fix to the rest of those files.

We've now re-organized the file structure somewhat, so that we have tei_xml_editing for files which SK is in the process of working on, tei_xml_ECH, for files edited but awaiting ECH's approval, and tei_xml_done, for files ready to go into the database. If this structure remains stable, I'll base the subversion repository on it.

Following up on our questions from March 2010 about how to mark up unattested glosses , we concluded the following:

An unattested gloss should be in its own <seg>, have type="u", and have a following <bibl> tag identifying the editor who supplied the gloss - generally ECH:

So this:

<seg><gloss>knot</gloss>ted up (one knot) <<gloss>tie</gloss>></seg>
<bibl>Y13.2</bibl>

Would change to this:

<seg><gloss>knot</gloss>ted up (one knot)</seg> <bibl>Y13.2</bibl>
<seg><gloss type="u">tie</gloss></seg> <bibl>ECH</bibl>

I will need to process these manually.

When we left off this project in March 2010, affix.xml and lex-suff.xml still needed a subset
of the steps from collapse_forms_etc.xsl run on them to clean up their dicteg entries. Martin had attempted this on 16Sep10, but I checked these files today, and the transformations didn't work. We concluded that it will be fastest for me to make these changes to the dictegs manually.

See the file status report blog entry, 29Mar10, and the "Automating some repetitive tasks through XSLT" blog entry, 18Mar10, for details on what needs to be done to each of these files.

Although all ENTRIES missing from qw.xml were successfully rescued into rescued.xml, I noticed various missing LINES which didn't make it into qw.xml, and were NOT rescued either. Some of these losses were due to data entry typos by CDH, but some are systematic: The following bands were lost in the transformation from the entries_separated stage to the partially_transformed stage:

xf
gn
gc
q
cul

g and k bands immediately following these bands were also lost.

However, after discussion with Martin, we decided we have come too far to go back and rescue these bands. I will re-type this data from the Lexware printouts as I go.

MDH wrote:

I've now tweaked the XSLT and run a second operation, which brought back 1906 entries, as opposed to nearly 3,000 from yesterday's run. SK will examine these and see if we've now hit the nail exactly on the head.

SMK adds:

I checked this new version of rescued.xml against qw-glot.xml, l-affric.xml, and the first 10 pages of Lexware printout of qw.xml. All the missing entries from these three files were successfully rescued. No "non-missing" entries were found from l-affric.xml or qw.xml. Two "non-missing" entries from qw-glot.xml were still rescued:

<!--From file: Q'W.xml-->
<ENTRY level="002" id="">
<ds/>
<infl>stative</infl>
<stt>q'ʷə+√q'ʷác'‐t</stt>
<g>they are *full</g>
<k>JM3.7.5</k>
</ENTRY>

<!--From file: Q'W.xml-->
<ENTRY level="002" id="">
<ls.ls.dm/>
<infl>intransitive</infl>
<i>kən q'ʷ+√q'ʷác'=əl=tn</i>
<g>I *fill|ed up my basket</g>
<k>JM2.110.2</k>
<var>kən q'ʷəc'+√q'ʷác'=ɨl=tn̩</var>
<g>I *fill|ed several baskets</g>
<k>JM3.7.7</k>
<i>q'ʷəc'+√q'ʷác'=ɨl=tn̩</i>
<g>he *fill|ed several baskets</g>
<k>JM3.7.6</k>
<i>q'ʷəc'+√q'ʷác'=ɨl=tn̩ lɨx</i>
<g>they *fill|ed several baskets</g>
<k>JM3.7.8</k>
</ENTRY>

We couldn't figure out why these were included in the rescue operation, but since all the truly missing entries have been found, it doesn't matter that a few extra entries have also been found.

Martin then proceeded to run the transformations outlined in the previous post on rescued.xml to create rescued_final.xml.

This is what we've done to get the missing entry items into a state where they can be merged back into the main collection whenever SK finds that one is missing:

1. Generated a rescued.xml file starting from the complete set of unmerged_xml documents, collated using xincluded-complete-list-expanded.xml, and running a new transform called retrieve_lost_from_unmerged.xsl.
2. Manually fixed nested <ENTRY> tags (there were only six instances).
3. Ran rescued.xml through remove_empty_tags_from_rescued.xsl, which removes the initial empty tag that caused all our problems; it also turns any child::infl tag into a comment (SK thinks this will be the simplest way to preserve that information). The output is called rescued_empties_removed.xsl.
4. Ran rescued_empties_removed.xsl through expand_separated_xml.xsl to produce rescued_empties_removed_expanded.xml. In this case, I tweaked the XSLT file to preserve comments from the preceding steps, recording the file of origin of the entry, and any <infl> tag value.
5. Ran rescued_empties_removed_expanded.xml through glottal_conversion.xsl to produce rescued_empties_removed_expanded_fixed.xml. This corrects a bunch of Unicode character representations.
6. Ran rescued_empties_removed_expanded_fixed.xml through collapse_forms_etc.xsl to produce rescued_empties_removed_expanded_fixed_forms_collapsed.xml. This makes a set of changes SK identified as being uniform and helpful to reduce the amount of repetitive work she has to do.
7. Renamed rescued_empties_removed_expanded_fixed_forms_collapsed.xml to rescued_final.xml

Where entries are missing from the other files, SK will now go to rescued_final.xml and retrieve them from there; they should be in the same condition as the other entries they're being merged with.

Note to self: it was difficult to reconstruct this conversion sequence because half of it took place before the blog was in place. In future, make sure anything like this is blogged in great detail.

My first shot at pulling back missing entries did catch all of the missing ones, but also found some which were not missing. After SK examined the results, we now theorize that where the entry had an <infl> element with an @mode attribute, then it was retrieved during the original operation; so these entries can now be excluded from our set. I've now tweaked the XSLT and run a second operation, which brought back 1906 entries, as opposed to nearly 3,000 from yesterday's run. SK will examine these and see if we've now hit the nail exactly on the head.

Met with ECH and SK today, and got everything going. This is what's been done:

• Following my own and ECH's research into the entries which were not converted at the separation stage, I wrote an XSLT file to harvest all of those entries from the unmerged files. SK will check a significant number of these to make sure that we've got all the missing ones. It's possible we may have got others that weren't missing, too; that's not so much of a problem, but I'll probably try to fine-tune the process to keep the numbers of entries down anyway. Then when we're sure we have the correct subset, I'll try to reconstruct the conversion process that the other files went through, and run this file through it. Then the missing entries can be merged back into the existing files as SK works her way through them.
• The schema has been updated. We were working from a 2006 schema with some manual modifications to the RNG file, for two reasons: first, we are using an element (<dicteg>) which has now vanished from P5, and second, at the time when we created the schema, Roma wasn't processing customized ODD files properly (IIRC). Now, though, I've put the whole thing on a more formally-correct footing, and built an ODD file to add <dicteg>, and to modify various other bits of the content model to allow (for instance) <bibl> inside <entry>. At the same time, we benefit from some other changes to P5, such as the availability of @type and @subtype on <gloss>, which enables us to mark editor-supplied (unattested) glosses as <gloss type="u">, making it possible to suppress them or or deprecate them in output if necessary.
• SK is set up on Spartan, working directly on the server copies of the files.
• I have asked for a Subversion repository for the XML. This is going to be essential, since there will be three of us working on the content. Once sysadmin have set this up, we'll have a little training session and I'll make a cheat sheet for SK and ECH.