teiJournal

Finished writing and doing basic testing of highlight-matching.xsl. The purpose of this file is to create supplement then normal way eXist tags search hits. When single items are searched for using the normal eXist &= and |= operators, each hit is tagged with an <exist:match> tag, This tag can then be used in the output to highlight the hits. However, when we search for an "exact phrase", we have to use the XPath contains() function in the XQuery. In this situation, the hits are not highlighted, because eXist isn't using its tokenized index. Therefore we need to keep track of the phrases which the user has searched on (only those searched using the + or AND operators), and then process each text node in the output to see if any of those phrases appear in the node. If they do, we need to highlight them.

This is achieved by first harvesting the search phrases from the <teiHeader> of the search results document, and putting them into a sequence called $phrasesToTag. Then we match on each text() node, and (assuming there are any phrases to highlight), we pass that node to a recursive template called tagMatches. This checks each of the elements in $phrasesToHighlight against the text node, using the contains() function; if it finds the phrase within the text node, it passes the text node and the phrase to another recursive function called tagMatch, which highlights all instances of that phrase.

The only limitation at the moment is that if there are multiple phrases, and more than one of them shows up in the same text node, only the first will be tagged. This is annoying, but it's not really a big deal; it's unlikely that multiple matches of phrases will show up often in the same text node, and in any case, the text node itself (or its ancestor node) will be returned as part of the hit report due to the first phrase being tagged in it, so we won't worry so much about this.

There's also probably room for some optimization in this code, but that will come later.

I've found the solution to an old problem: how to get the request URI into the output. I can pass it as a parameter into an XSLT transformation like this:

<map:transform type="saxon" src="xsl/highlight_matches.xsl">
  <map:parameter name="browserURI" value="{request:requestURI}?{request:queryString}" />
</map:transform>

It doesn't include the protocol, server, or Tomcat port, but those are not important anyway, really.

I'm also beginning work on the code which highlights the phrase matches through XSLT. Actually, it seems most likely that this will be achieved through a call to a Java class, as documented here.

Working on the Java classes alongside the XQuery code that's calling them, I've now extensively revised both, to give results which seem to work well, and give valid, usable output. The list of phrases to highlight is now provided as an array by the Java class, and the XQuery iterates through that array to create a list of items, resulting in actual XML elements rather than textual data. When there's no search string, a dummy text body is returned for each document header, so that the resulting corpus file is valid.

Because testing with a Java class can be risky, Greg's set up a test environment for me to hack at, with a working Tomcat/Cocoon/eXist stack containing teiJournal.

In this environment, I've deployed the Java classes I wrote, and integrated them into the XQuery. This is how you do it:

1. Add the jar file which contains your classes to the Cocoon/WEB-INF/lib directory.
2. Add a namespace for your class like this:

   declare namespace su="java:SearchParser";

   My class is not qualified by a hierarchy; it's a top-level class in a library called xqSearchUtils.jar. It appears that the name of the jar file does not have to match the name of the class; the two classes in my jar file are called SearchParser and SearchBit.
3. Construct an instance of your class like this:

   declare variable $sp := su:new($search);

   This is equivalent to Java:

   SearchParser sp = new SearchParser(search);

   In this case, $search is the variable containing the search string the user has submitted.
4. Call functions on the object like this:

   declare variable $searchStringBit := su:escGetXQueryClauses($sp);

   which is equivalent to Java:

   String searchStringBit = sp.escGetXQueryClauses();

The only slight complications I encountered concerned the escaping of strings coming back from the Java object; to handle this, I added extra functions to the object to escape strings for XML. However, I have one slight issue remaining: the "phrases to highlight" stuff is coming back as a string, even though it's a set of XML <seg> elements. I need to figure out how to get elements from it.

Once that's all working, I need to turn on XACML, and figure out how to nail down the use of Java so only my classes can be called.

Added handling for the AND, NOT and OR keywords, so they are actually converted to the equivalent +, - and nothing-at-all equivalents before the search string is parsed. Strings which use either now give equivalent results.

Added another function to the Java class, to enable the calling XQuery to embed XML detailing any phrases which will need to be highlighted in output. This is explained in the comments in the class file:

* This is how you would use this class:
* 
* 1. Get the search string as input from a form, and collect it 
* in the XQuery using request.getParameter().
* 
* 2. Create a SearchParser instance, passing the search string to 
* the constructor.
* 
* 3. When you need the XQuery clauses to tack onto your XQuery 
* code, call the SearchParser's getXQueryClauses() method. What 
* you get back can form part of a string passed to util:eval().
* 
* 4. When you need to embed phrases that your XSLT can find and 
* highlight, call the SearchParser's getPhrasesToHighlight() method.
* This gives you back a <seg type="searchHitPhrase">...</seg> element
* for each phrase which is included in the search. This is necessary 
* because, when we search using eXist's &= or |= operators, the hits
* come back tagged with <exist:match> elements, but when we use the 
* XPath contains() function (which is necessary for doing phrase 
* searches), they don't come back highlighted. XSLT can use the <seg>
* elements to perform some later highlighting on search results, 
* perhaps by adding <exist:match> elements which can be rendered in 
* the normal way later in the XSLT.

Having decided, for the moment, to offload the heavy lifting of the search string parsing to Java code, I rewrote the contents.xq page so that it just does a simple |= job on the search string contents, so I can carry on working on the rest of the page. This seems to work fine. Next, we move on to the XSLT, which is going to be complicated too: it will have to find all the matches that were achieved with contains(), which will not be tagged up with <exist:match> tags, and either tag them up or highlight them in some other way. That means I really do need to know what the search string was, even at the XSLT level. Perhaps this tagging should take place in a preliminary XSLT operation, prior to the main rendering of the page.

Also, it strikes me that the Java class I'm writing ought also to be able to spit out a list of the phrases that will need highlighting, in a format that's easily parsed by XSLT. That would make things quicker and simpler.

My two simple classes for parsing a Google-style search string and emitting XQuery clauses are now finished and tested; I've added extra features for handling redundant spaces, returns, and other deformities in the input, and everything seems to be working pretty well. The only slight difficulty is the problem of handling optional phrase matches; with mandatory phrases, we can use [contains(., "the phrase")], but with optional phrases (not preceded by +) we have to fall back to adding all the components of the phrase to a [. |= "the phrase and other stuff"] clause. This is not ideal, but I don't see a comparable XQuery clause I can use, and I don't want to break the system down any further. Another option would be to treat all phrases which are not minused ("must not contain") as being mandatory; we should consider that, or even make it a switch.

The next stage is to build the jar package, code links to it into the XQuery, and test it on a sandbox system that Greg is setting up, where we can mess with the XACML settings and turn on Java binding.

After some serious brainstorming with others in the lab, it seems that it's going to be virtually impossible to parse a Google-style search string in XQuery. The best approach is probably going to be using a Java class, which can be called from inside XQuery, to parse out the components of the search string and return a block of XQuery clauses which can be appended to the query as appropriate. This will entail enabling and configuring XACML in order to protect the db against the running of arbitrary Java code.

Started writing the Java package in Eclipse; so far it can parse out each of the components of the search string successfully, and decide which type of components they are. Next I need to build the XQuery in the most efficient way (for instance, amalgamating all of the single-word must-contain components into one &= clause).

Marked up the editorial introduction to the volume.