Mariage

I've implemented search result caching in Mariage, and done a bunch more work to bring it up to speed with what I learned in the Graves project. However, I'm now faced with a problem in search design which also afflicts MoEML, summarized here:

Imagine you want to find "amour" in your documents. You search for "amour".

It finds (say) thirty documents which contains "amour". It returns the first ten (it's paging in sets of ten results), and it sets about giving you all the keyword-in-context display results for each document.

Now, the first document has 200 instances of "amour". So the search code has to do a kwic expand operation on all 200 of those results in order to give you 200 keyword-in-context fragments for that document. These operations take a long time, so it takes ages for your results to come back.

If your results page contains ten documents, each of which has 200 hits, you're now processing 2,000 hits to give a single page of results.

In the Graves project, this isn't an issue, because all the documents are tiny (one diary entry). But in Mariage and MoEML, we have a combination of very small (one poem, one little article) and very large (Satire Menipée, Stow) documents.

One option is that instead of returning all the hits for a document, you just return (say) the first five, with a note "195 more", and the option to search only that document. If you take that option, you see hits only from that document, but paged in sets of ten.

Another option is to treat the search as a search of the collection itself, so that every hit is a separate "result"; in that case, in our imaginary scenario, the first 200 hits (i.e. the first 20 pages of results) come from the first large document, and you have to get to page 21 before you see anything from the next document.

Another option is to search at the granularity of smaller fragments rather than full-scale documents (Stow chapters, etc.). The problem with that can be seen in this example, where search results from the same play are scattered around because each scene is searched as if it were a separate document.

I have a vague notion that you might let users search "FOR DOCUMENTS" (in which case they'd get summaries with the first one or two hits, with documents ordered by hit-count) or "IN DOCUMENTS" (in which case each individual hit in a document would be a separate "result" on the page. But I'm not sure how easy that would be for users to understand.

Fixed the bug where the docTitle was repeated at the beginning of introductory fragment documents. Also created an application icon for use in the XAR file.

Worked with GN to fix some issues we've identified with AJAX searches which are pushed into the history stack with JavaScript. The solution (in search.js) depends on a couple of factors:

• When a page reloads as a result of the use of the back button, the window's onload event does not fire, therefore the URK is not parsed and the search is not done. The solution here is to use the window's onpopstate event to trigger the same behaviour.
• When a search is sent off and a new URL-encoded version of that search is ready to be pushed into the history stack, check that it's not identical to the current one; if it is, don't put it in the stack, otherwise you'll have two copies of the same URI in the stack.

Current solution tested and working on Chromium and Firefox.

As of today:

• JS/AJAX search is now working, using a simpler AJAX approach that I might generalize to replace js.jx.
• Search params are pushed into the window history.
• Loading a saved search URI populates the form and runs the search.
• A large list of French stopwords is implemented, although we're still using the standard analyzer.
• Pages are now being served as XHTML from eXist, which makes the CSS work properly on Chrome.

Lots of work still to do, including making search highlighting apply to document display, and making image search retrieve an image "fragment".

I erroneously gave CC the instruction to use pc/@force="weak" for hyphens that should be retained; and she misunderstood and added it to hyphens that should be dropped. This resulted in a bit of unnecessary encoding in a short text. I've fixed the text, revised the instructions and tweaked the XSLT handling, and with luck everything should work for the next text.

I have a basic XQuery search file working, with some indexing configuration, and I'm getting results back. Lots more to do, though:

• Format results for return, with document metadata to go along with the kwic summaries.
• Handle insertion of image fragments for gravures -- not sure how I'll actually do that, at this point.
• Hooking the AJAX return into the page with JS.
• Figuring out how/when to restrict search to normalized documents. This may required some configuration on the content div.

The other thing I've been working on is finding a decent approach to a stemming analyzer for these documents, and a stopword list. Looks like I may be able to use the French Lucene analyzer, but I'm not sure how well stemming will work.

• Confirmed with CC that the hyphenation diagnostic works for her.
• Enhanced it to sort by doc id, and to ignore documents with a specific entry in the revisionDesc. Also added instructions for how to encode both non-breaking hyphens and the revisionDesc to the top of the result. Sent it off to CC.
• Added <pc[@force='weak']> to the schema, along with a Schematron rule to ensure it only wraps a single hyphen.
• Added handling for this element to the HTML output, in which context it's turned into character U+2011 (non-breaking hyphen).
• Enhanced the normalization transformation so that it's sensitive to the non-breaking hyphen and handles it correctly.
• Tested with a fake example temporarily added to a file. Will wait on real encoded samples for better testing.

Wrote an XSLT routine that lists out all the instances of hyphenated linebreaks in context, with line numbers (so it requires Saxon PE) to allow CC to look for any that need to be tagged for preservation when rendering continuous views of the document.

We're beginning to get to grips with the nitty-gritty of the primary source document styling, and finding that there are a few things about the original XML encoding of style which are either slightly wrong or incomplete. That means we'll have to make minor changes to some of the CSS in the original documents. I've done some work on Candale 1. The issue we're going to hit is that these changes may have unpredictable effects on the display of these documents on the current site. So I'm checking with CC to see if if we can call a halt to uploading them to the current version of the site, and focus on the new site.

I now have quite a nice view of the original text now, with linebreaks and hyphens removed, long s's switched out, and so on. But one possible problem is that there may be cases where a hyphenated compound happens to fall on a linebreak. I don't know how often that happens in these texts, if it happens at all, but if it does, we need to avoid deleting that hyphen when we remove the linebreak. That means finding all those instances and tagging them with a specific attribute that makes it possible to preserve the hyphen.

Obviously we don't want to be reading the whole document set again to find all those things, so I thought I'd write a routine that lists all the hyphenated linebreaks with their line numbers so CC can easily scan them and identify those that need to be tagged:

   ...il la laissa sur sa bonne-foy, & par cette maniere...  (line xxx)

     ...lettres ſur ce neceſſai-res. A ces cauſes voulant... (line yyy)

I've written to see if that will be a workable approach.

At least it looks that way from a quick check.