Nxaʔamxcín (Moses) Dictionary Blog

JW on the eXist list solved my problem, by pointing out a slight syntax change that makes util:expand() work as it should. This is the working code to retrieve tagged hits inside a modified container:

for $e in collection('/db/moses')//tei:entry[ft:query(., $find)]
    order by ft:score($e) descending
      return
      <tei:entry xml:id="{$e/@xml:id}"  n="{$e/ancestor::tei:TEI/tei:teiHeader/tei:revisionDesc/@status}">
        {util:expand($e)/*}
        </tei:entry>

It works really well, and all the normal search string tricks (plus and minus, quotes etc.) work perfectly. Stayed late to get it deployed to the server.

I have the search working well, but there's a dilemma: due to what seems to be a bug in eXist, I appear to have a choice between retrieving results with the hits marked up with <exist:match> tags, which enables me to highlight them, and retrieving the results with their status value attached to them as an attribute. In other words, if I massage the retrieved entries in any way, to attach the attribute to them based on the status of the file they come from, the <exist:match> tags are lost, and I can't highlight the hits on the page; but if I don't add the status value, I can't display the appropriate red, amber or green box next to each hit.

Because this looks like an eXist bug, I've spent half the afternoon trying to rebuild eXist and Cocoon using our script from last year, but that doesn't seem to work, unfortunately; I'm still trying to figure out why it fails.

I now have the search returning results rendered into XHTML. The last stage is hooking up the JS to make the AJAX work.

Completed entries now show with a green light, unedited ones with red, and everything else with orange.

Implementing a simple search engine, this is what I've done so far:

• In collection.xconf, added this section, to enable fulltext searching of the specified nodes:

      <lucene>
        <text qname="tei:entry"></text>
        <text qname="tei:pron"></text>
        <text qname="tei:def"></text>
        <text qname="tei:hyph"></text>
        <text qname="tei:note"></text>
      </lucene>
  

• Confirmed that simple queries like this now work:

  declare default element namespace "http://www.tei-c.org/ns/1.0";
  declare namespace tei="http://www.tei-c.org/ns/1.0";
  declare namespace ft="http://exist-db.org/xquery/lucene";
  
  for $e in collection('/db/moses')//tei:entry[ft:query(., 'visiting')]
        return $e
  

• Added a pipeline for outputting the XML results of the search, and confirmed the results are showing up with appropriate exist:match tags.
• Created a basic pipeline for the eventual XHTML output.

Next steps:

• Create a search.xsl file, which will render a search form.
• Decide whether we should retrieve results into the existing page, or whether we should reload the page anew and echo out the search parameters.
• Decide on how to render the results (full items, I think).
• Test the Lucene query syntax, and evaluate it against the flexibility of our own Java library. We could rewrite the Java library to support the Lucence ft: searching, or if it provides enough power on its own, adopt it in preference to ours.

In preparation for writing a simple search, I tweaked the menu so it's clearer what page you're currently on.

Doing a lot of background reading in preparation both for the next phase of the project (more formal publication of the dictionary interface and data) and a proposed article. Most notes are on the printouts of articles, but key points are:

• I must implement an OAI-PMH interface for the dictionary. There are good examples available through OLAC (where we should also be registering the dictionary when it's publicly usable). The mechanics of this are not complicated, but we don't yet have the metadata in a usable form.
• Each entry should have a single-URL access point, so that it can be cited. There should also be a link for citing it.
• For citation, we need versioning. After considering a range of options, we decided that the best way to provide it is this: whenever an entry is retrieved, it has a "cite me" link on it, and that retrieves a formal citation pointing to the entry, with a date, which is derived from the latest date of any entry file in the database. I need to implement a little XQuery function to retrieve that date. We can't really do versioning beyond that, for (say) individual files, because any given entry will retrieve into itself entries from its constituent morphemes, and will link out in other ways to other entries.
• We need a subdomain for the project. However, moses.uvic.ca may not be acceptable for various reasons, and Nxaʔamxcín won't do as part of a domain name; we want ascii if possible. ECZH is considering the options.
• Entries for morphemes retrieved into the middle of an existing entry should be handled more intelligently. Right now, they simply embed themselves as if they were part of the parent entry. What I think we need is this:
  • When an entry is retrieved initially, a skeleton tabbed panel control is created, with a tab and panel for each of the constituent morphemes. The panel for each morpheme should have an @id attribute which is constructed from the combination of the parent and the child @xml:ids. All panels and their tabs should initially be display: none.
  • When a child entry is retrieved, it is placed into the appropriate panel, and its tab and panel are shown. Exactly how this works may be complicated, because it may involve showing the the whole tab control for the first time, or showing just the new panel, and flipping from an already-visible panel to the newly-populated one.
  • Panels should be (say) max-height: 8em; overflow-y: scroll, so that they don't take up more than a small part of the parent entry. They should be visually distinct from the main entry content.
  This will enable the reader to retrieve the full information about constituent morphemes if required, and switch between them, but without overwhelming the original entry.
• There's a range of documentation that needs to be completed in XML files peripheral to the main entry content, including documentation of terms, description of phonetic alphabet usage, and personography.

I went through the list of 160 cases where a @sameAs has a value but there's currently no corresponding @xml:id on an entry.

Most of them are affixes which are in the list for Ewa to add to affix.xml. We also need to add "min-INST" to that list.

Entries also need to be created for the following lexical suffixes:

nwˀaln
tn
ups
wil

Other problems will be resolved when pron.xml is edited.

Some of the problems were typos. Interestingly, oXygen does not flag typos in @sameAs the same way as in an @xml:id. For example, it doesn't make the file invalid to have a raised w in a sameAs.

There are also a few specific cases I need to ask Ewa about.

Ran some checks to see how many @sameAs attributes are not pointing at existing @xml:ids. There are 160 cases where the @sameAs has a value (is not empty), but there's no corresponding @xml:id on an entry:

skʷtas in entry skWtas
kn in entry kascaʔqWmixn
lqs in entry scəslqs
lqs in entry scəslqs
nwˀáln in entry cx̣cx̣nwˀalnˀ
sac in entry saṣc̣ụṣkṣtəxW
n-LOC in entry naḥaʔe
m-MID in entry ḥacm
OC in entry ḥacc
k-LOC in entry kḥacn
min-INST in entry ḥacmin
t-TR in entry ḥacpnuntn
OC in entry ḥacḥacc
n-LOC in entry nḥacawˀsn
min-INST in entry ḥacmintn
tn in entry ḥacmintn
kɬ-LOC in entry kɬḥacmintn
min-INST in entry kɬḥacmintn
tn in entry kɬḥacmintn
n-LOC in entry naḥcusmn
min-INST in entry naḥcusmn
n-LOC in entry naḥcapxn
ap in entry naḥcapxn
k-LOC in entry kḥac̣ḥac̣qinˀxn
CHAR in entry kḥac̣ḥac̣qinˀxn
tn in entry ḥacʼxWn
ḥaḥənt in entry sḥaḥəntaɬxW
ḥalˀxʷ in entry kasḥalˀxWmíx
ḥalˀxʷ in entry ḥalˀxWn
n-LOC in entry naḥalˀxW
ḥalˀxʷ in entry naḥalˀxW
ḥalˀxʷ in entry ḥalˀxWulˀəxW
ḥalˀxʷ in entry ḥalˀḥalˀxWxn_kn
kn in entry ḥalˀḥalˀxWxn_kn
kn in entry kn_ḥalˀḥalˀxWxn
ḥalˀxʷ in entry kn_ḥalˀḥalˀxWxn
kn in entry kn_tḥalˀḥalˀxWanaʔ
t-LOC in entry kn_tḥalˀḥalˀxWanaʔ
ḥalˀxʷ in entry kn_tḥalˀḥalˀxWanaʔ
t-LOC in entry tḥaƛʼm
OC in entry ḥaƛʼƛʼ
OC in entry sḥaƛʼƛʼ
OC in entry sḥaƛʼƛʼtn
tn in entry sḥaƛʼƛʼtn
OC in entry ḥaƛʼƛʼcin
OC in entry sḥaƛʼƛʼnus
OC in entry ḥaƛʼƛʼamˀinˀaʔ
kat in entry katḥaƛʼƛʼanaʔ
OC in entry katḥaƛʼƛʼanaʔ
ʔin in entry ʔiscḥamcin
sac in entry ʔiscḥamcin
limt in entry ḥamlimt
kn in entry kn_ḥamlimt
limt in entry kn_ḥamlimt
OC in entry ḥamm
n-SUBJ in entry ḥamˀcn
n-SUBJ in entry ḥamˀn_lx
t-TR in entry ḥamˀntm_lx
OC in entry ḥassus
CHAR in entry ḥatḥat
n-LOC in entry snḥawaɬcʼaʔtn
tn in entry snḥawaɬcʼaʔtn
ayˀ-PAST in entry kasḥawˀiɬtmnayˀuxW
ayˀ-PAST in entry sḥawˀiɬtmnayˀm
m-MID in entry sḥawˀiɬtmnayˀm
sac in entry scḥawˀi
sac in entry scḥawˀiymix
kn in entry kn_ḥawˀiyasnccwˀaxaʔ
sac in entry scḥawˀiymix_lx
kn in entry kn_scḥawˀiymix
sac in entry kn_scḥawˀiymix
sac in entry scḥawˀiymix_kt
sac in entry scḥawˀiymix_kW
sac in entry scḥawˀiymix_kWp
m-MID in entry ḥawˀiym
m-MID in entry sḥawˀiym
n-SUBJ in entry kasḥawˀiyn
t-TR in entry ḥawˀiyntxW
xW-SUBJ in entry ḥawˀiyntxW
n-SUBJ in entry ḥawˀiyminn
n-SUBJ in entry ḥawˀiyminnayˀ
ayˀ-PAST in entry ḥawˀiyminnayˀ
n-SUBJ in entry ḥawˀiyminn_lx
n-SUBJ in entry cḥawˀiystunn
ɬ-DIR in entry cḥawˀiɬtunn
n-SUBJ in entry cḥawˀiɬtunn
ɬ-DIR in entry ḥawˀiɬt-2
t-TR in entry ḥawˀiɬt-2
t-SUBJ in entry ḥawˀiɬt-2
ɬ-DIR in entry ḥawˀiɬt-2
t-TR in entry ḥawˀiɬt-2
n-SUBJ in entry ḥawˀiyxitn
n-SUBJ in entry cḥawˀixitn
ʔin in entry ʔiscḥawˀi
sac in entry scḥawˀiys
s-POSS in entry scḥawˀiys
n-SUBJ in entry ʔalḥawˀiyn
sac in entry scḥawˀiyaltuxW
m-MID in entry ḥawˀiyaltm
sac in entry scḥawˀiyaɬxWuxW
kn in entry kn_ḥawˀiyáɬxW
m-MID in entry ḥawˀiyaɬxWm
m-MID in entry ḥawˀiylupm
min-INST in entry ḥawˀiymin
min-INST in entry ḥawˀiyminn-2
ham in entry chamhamilx
CHAR in entry chamhamilx
kn in entry kn_hamp
kn in entry kn_hampncut
n-LOC in entry nahamp
CHAR in entry hamhampul
n-LOC in entry nhampatkW
k-LOC in entry khamhampawˀs
k-LOC in entry khamhampakstmn
m-MID in entry hampmnatkWmˀ
m-MID in entry hampmnatkWmˀ
wil in entry hapyˀwil
wil in entry hapyˀwil
m-MID in entry hapyˀwil
n-LOC in entry nhapyˀatkWn
kn in entry kn_sḥaypx̣əxW
OC in entry hə̣ll
t-TR in entry huymn
ayʼ in entry huymnayʼ
t-TR in entry chuymntls
Ø-OBJ m-SUBJ in entry huystm
m-OBJ in entry ʔachuymstms
ɬ-DIR in entry huyɬn
Ø-OBJ n-SUBJ in entry huyɬn
ɬ-DIR in entry kashuyɬn
Ø-OBJ n-SUBJ in entry kashuyɬn
n-LOC in entry nahuyxn
CHAR in entry sʕWaʔʕWaʔ
CHAR in entry sʕWʕWaʔʕWaʔ
askʼayˀt in entry ʕWəcʼpaskʼayˀt
kat in entry katʕWətʼqWatkW
ups in entry ʕˀWənˀps
ups in entry ʕˀWənʕˀWanps
n-LOC in entry nʕˀWənˀʕˀWanˀpsn
ups in entry nʕˀWənˀʕˀWanˀpsn
tn in entry nʕˀWənˀʕˀWanˀpsn
CHAR in entry sʕˀWəsʕˀWəs
CHAR in entry sʕˀWəsʕˀWəs
CHAR in entry sʕˀWʕˀWəsʕˀWəs
niʔ in entry niʔqʼWacʼlqs
n-LOC in entry naqʼWacʼt
kn in entry kn_qʼWqʼWacʼltn
tn in entry kn_qʼWqʼWacʼltn
kn in entry kn_qʼWəcʼqʼWacʼltn
tn in entry qʼWəcʼqʼWacʼltn_lx
OC in entry qʼWalˀlˀx
m-MID in entry qʼWaɬtmm
CHAR in entry sanspax̣pax̣t
CHAR in entry ṣə̣nṣə̣nt
CHAR in entry ṣə̣nṣə̣nt_lx
CHAR in entry ṣə̣nṣə̣ntwilˀx
OC in entry ṣụʔụʔtʼ
niʔ in entry niʔṣụʔṣụʔtḷqs
lqs in entry niʔṣụʔṣụʔtḷqs
solcas in entry ṣolcaṣ 

Most of these will be expected, but some may be typos.

I've converted the site so that it uses the brand-new GentiumPlus fonts, which are supplied as WOFF files as well as TTFs; I'm using @font-face to get the browser to download the fonts, and it seems to be working nicely (except on Opera, which doesn't support WOFF files yet). They're pretty large, but it's worth it. I noticed some missing characters in SMK's browser the other day; hopefully that will now be solved.