Mariage

The site has been running well for days, so I'm getting more confident in it. I did a lot more uploads today, and all went smoothly.

Also spent some time with LSPW and LCC discussing some markup issues. I'm looking forward to getting down to some serious documentation of the markup policies we've been evolving; these are often mentioned on the blog, but sometimes not, and really ought to be codified at this point. I think the best approach might be a series of blog posts by me on various markup topics, which could then be checked over by the others, and eventually merged into a single document and translated for the website; the documentation on the site right now is inadequate.

I also need to do some more explicit documentation of the site structure and backend code, so I can hand it off to MJ for some focused work in the coming weeks. There should be an "if I get hit by a bus" document as well, maintained regularly on the blog. All our complex projects should have this.

Did some tidying up and enhancement of the Apparatus/similarity demo app in Delphi, making it more amenable to screenshots for the DH2010 proposal. I have some hope that XPath can be made to work, but not this week; probably best to do this all again in Java.

After an attempt to get the site running on the main Tomcat through a cunning alias, which failed due to permissions complications, we've gone for a deployment in the Tomcat-dev. Cleared out two old projects ("dms" and "dhcn", obsolete ETCL sandboxes), I uploaded the Mariage cocoon, and had sysadmin chown the entire folder to apachsrv (so that eXist can modify its db), then chown the site directory back to me, so I can do development. The net result for Tomcat-dev is probably beneficial, because it's lost two projects and gained one. I've asked for the virtual host to be reconfigured.

Currently doing background research and collecting my notes for the DH 2010 submission on similarity metrics and the Courval texts. I've written a couple of paragraphs on the background, and started compiling a bibliography. Lots of work still to do. Also investigating the possibility that we might be able to use XPath in the Delphi demo app, with some success (helped by the ADOM author DK); but there are some limitations that might make it a bit complicated for a general-purpose tool. I'll have to think about it a bit more.

On a roll today. In an effort to demonstrate that something was wrong with the XQuery generator, I created a really simple project in a test Cocoon, and found that the <exist:match> tags were coming back exactly as expected. That meant there was something particular that I was doing in my Mariage find.xq file which must be resulting in the loss of the tags. After much hacking around, I found a solution -- although I still don't understand why it works. First of all, I had to make sure I was adding the exist namespace to the <teiCorpus> element in which I was returning the results; I hadn't been doing that, which meant that the match tags had an unbound prefix. Second, and most important, I had to rewrite the actual file so that I constructed the output document in a slightly different way. I had been generating it using a lot of embedded functions, with local variables containing (for instance) search results, then querying that local document in order to get its contents and drop them into the return document. This worked, and the search is now functional on my local test copy of Mariage.

The only thing I can think of that might have caused the problem is that during the process of constructing the document in memory, and then querying it, elements in the exist namespace were somehow lost. In any case, I now have a solution, and we can think about deploying it first thing Monday. I can chop down that Cocoon in the same way I chopped down the latest one, and just push it up either into the main Tomcat or into the dev one. Greg and I can decide on Monday.

Unfortunately, porting the fix back to the old (mariage_3) project didn't help; still getting no search results there. All the more reason to rush out a new deployment.

Next I'm going to try extracting ScanCan in the same manner, and see if that works as advertised...

...from a stack of things, from discussions about how to handle the embedded narrative in the Agre'emens, to fixing validation issues in the Cholière, to scripting some of the build-and-test processes for new Cocoon/eXist builds, to getting the server restarted multiple times...

WM reported that the indexing bug has now been fixed, so I checked out and rebuilt eXist. I have a local deployment working, and I've started rationalizing the database structure a little. Everything seems to work well, except that the search functionality that inserts <exist:match> tags STILL seems to be broken. After posting on the list about this a couple of weeks ago, I've had no response, so I conclude that it's not a priority. I'm going to have to find an alternative, and since I have my own Java library that was written for the teiJournal project, that does a lot of this work, and does a decent job of it, so I might try using that instead. I spent quite a while getting the index configuration set up, and I presume the actual indexing is working, but match-tagging isn't.

Extended my bash script for pulling down the in-progress markup documents so that it integrates those which are already being shown on the site into the site tree correctly; this allows me to update the site more easily. Also added handling for the <mentioned> tag, which should now result in guillemets (per instructions from CC).

Over the last couple of days, the site has been very flaky because of the eXist indexing problem. We're no nearer to a fix, as the latest messages on the eXist suggest a similar error (index corruption due to failure to store temporary fragments) is still happening in the 1.4 RC. The trouble seems to be triggered by any serious search on the Mariage search page, which does store temporary fragments; in attempting to minimize the scale of these, or the processing required, I tried to add new range indexes this morning, but that borked the whole system again. When Greg comes in, we'll take it down, remove all dbx files except for dom.dbx, symbols.dbx and collections.dbx, restart the db and use the Java admin client to reindex everything.

I've posted a warning on the main page about the search problems.

The search page on the site seems to have become spectacularly vulnerable to the index corruption we've been experiencing with eXist. It's now almost never actually working properly. In the absence of a stable eXist 1.4 as yet, we're kind of stuck, but this is getting worrying. I may have to work on a Mariage stack that's based on the 1.4 RC to get around the problem. On the 1.4 RC, the search isn't working either yet, but that's due to XQuery changes which I need to get a handle on anyway, so perhaps I should be doing that as a priority. I'll look at it tomorrow.