Mariage

I've now marked up five pages of Corps des époux, and encountered more new features to document:

• The text requires linking to specific locations in texts in the anthology; in the cases I met today, these were page references inside Choliere (which is still not really in a state to be up on the server -- that's another matter). There was no way of doing this, but I've created one today: ever <fw> tag which is @type="pageNum" or @type="folioNum" generates an XHTML @id attribute generated from its content, so you can now write a link like this:

  <ref target="xhtml.xq?id=choliere#p_82">

  to link to page 82 of the Choliere. This will only work if there is such an <fw> tag in the XML, and in cases where there are duplicate page numbers (as in the Amboise), only the first instance will get an id. Older and smaller documents that don't have explicit <fw> tags usually have a <pb n="xx> formulation, and these will not get ids and not be linkable. It may be that we therefore want to go back and mark up those documents appropriately; but if they're short it doesn't really matter, and if they're long, they're most likely going to get reworked anyway.
• Blockquotes are common and some have biblio references at the end of them:

            <cit rend="block">
              <quote>
                Si toutes nos fringuantes, qui ont si grande envie de sçavoir aux despens de leur pucellage que c’est de s’acoupler avec le masle, estoient assubietties à la rigueur d’une telle loy … i’en tiens la plus-part si doüillettes qu’elles aimeroient mieux estre filles toute leur vie, qu’avec une si mal plaisante ceremonie commencer leurs nopçages… 
              </quote>
              <bibl>(<ref target="xhtml.xq?id=choliere#p_83">p. 83</ref>)</bibl>
            </cit>
            
  

We also have a potential collision between our editorial notes and footnotes in original texts (the Cabinet Satyrique is the main case). After some discussion, I've changed editorial notes so that they show up looking like little buttons, leading to a popup, whereas original footnotes should be encoded in-place, as they originally looked (at the bottom of the page where they appear). The Fantastique Repentir will be the test case for this.

Changes haven't yet been migrated to the server, because it's a bit fragile; I have a brand-new stack for Mariage built with 1.4 stable, with a couple of extra tweaks, which I'm planning to deploy on Monday.

The eXist db now seems to require a full re-index whenever you upload files into it; they don't get properly indexed just through the upload process. My guess is that this is because we actually deployed a pre-release version, and there were some tweaks after it was fully released. My plan now is to build a new version with the Cocoon/eXist package I built on Monday for MJ, test that locally and then replace the server copy with it. I also want to make a further change to conf.xml, through my build script, as follows:

<db-connection cacheSize="48M" collectionCache="24M" database="native"
        files="webapp/WEB-INF/data" pageSize="4096" nodesBuffer="-1">

becomes

<db-connection cacheSize="48M" collectionCache="24M" database="native"
        files="webapp/WEB-INF/data" pageSize="4096" nodesBuffer="1000">

With the setting at -1, the system consumes all available memory when doing indexing; with it set to 1000, it keeps only 1000 nodes in memory at a time, leaving memory free for other threads.

I now have a Java class working and tested which can do the NCD calculations. I may wrap it in a larger class which can also do all the normalizations, for clarity. The testing has revealed some interesting aspects of the USM which bear on the question of "What is similarity?", or better, what on earth could constitute an objective measure of the similarity of two pieces of text. If you take one piece of text and compare it with another which consists of dozens of copies of itself, the similarity is very high (because of the repetition, which is sort of factored out by the compression); in one sense, this is true (they are very similar), but in another, it's not (they're of radically different lengths).

I've started re-working the Choliere text, which EM prepared as part of her workshop last year, to go up on the site. It's generally pretty good, but there are many fixes needed. This is what I've done so far:

• Fixed a lot of @rend attributes which were not CSS.
• Changed <fw type="pageNum"> to @type="folioNum", since these are folio number showing only on the recto.
• Added lots of header information (still waiting for some from CC).
• Added the <tagsDecl>/<rendition> elements for describing the basic page layout.
• Moved all folio number tags before the running title headers, because they float. This is standard on our site; it makes the rendering easier.
• Enhanced some of the @rend attribute CSS in the front to make the rendering better approximate the original.

The most urgent problem that remains is the issue of the little marginal subject headings; these are exactly like those in the 1609 Sonnet, where we've handled them with <argument> tags, which are rendered into the margin. However, there, each little heading signalled the end of one <lg> and the beginning of another, which is logical (new heading, new topic, new line group). In the Choliere, whole chapters constitute a single paragraph, so there's no division into which you could insert an <argument> tag. I'm still working on this. I think the tag is the right one to use, but the schema is blocking me. In addition, I'm not at all sure that these unbroken blocks of text are rightly described as paragraphs (they're marked with <p>). They really are "anonymous blocks". But changing to <ab> still doesn't allow me to insert the <argument> tag.

Finally, chapter headings are marked as <p> tags containing <title> elements. I think we really should be using <head> elements there.

Working through the first of the modern articles, I'm finding some particular issues to record:

• We need to link from citations in the text to biblio refs in the document itself, in the <back>. I've settled on doing this with <bibl corresp="#dejean_2005"> (pointing at the xml:id of the <bibl> tag with the full reference).
• We need to link out to references in the central bibliography file. For that, I've settled on <bibl corresp="../biblio/biblio.xml#mercier_2005">, where the path is relative within the db structure and the corresponding directory structure on my system, and in any case the type of link can be recognized through the presence of biblio.xml. These references will need to be pulled into the article when it's being rendered.
• We need to link to documents in the anthology itself. These are not really citations; they're references and intended as links. For this, I've gone for <ref target="xhtml.xq?id=operateur_cephalique">; which should work in context (all documents are accessed through URLs in the root of the site), and which can also be expanded to a full URL if necessary.
• Footnotes often include references, which will lead to the same kind of embedding problem we have with references referencing other references. I think a site-wide solution might be achieved for this in JavaScript, whereby the popup element is cloned so two items can be displayed at the same time.
• I've used <cit rend="block">, which will need some specific handling. Once this is done, it might be repurposed to handle those references which include multi-paragraph quotes.

I'm marking up the first of the modern articles, with the idea that when it's done, I'll document the markup practices I've used, and then subsequent articles can be marked up by the team. I've now done the bibliography, but there are some outstanding questions about formatting differences between this biblio and the main site bibliography; for instance, in the main biblio, authors' forenames are shown in parentheses after the surname, whereas in the article they weren't. Also, the main biblio seems to use "dir." for editors, where the article biblio uses "éd." (unless I'm misunderstanding -- could these be different roles?). For now, I've standardized this biblio to match the main one.

Continuing work on the search system, to get it to a more usable state:

• doc_with_refs.xq now incorporates match-tagging using the old fulltext index system. This works to apply <exist:match> tags for any-word or all-word searches.
• <exist:match> tags in documents are now rendered highlighted in light grey, with yellow text (in the default theme).
• In the case of exact-phrase matches, we have a different problem, because using the shorthand eXist &= or |= matches won't work. At the moment, I'm just highlighting all the words in the phrase. In the long run, we need to use the ft:query syntax to take advantage of the new Lucene features, but that doesn't seem to work at all in the Cocoon block build; that's something I'm still trying to figure out. It could be a problem with our eXist setup (although it seems to comply perfectly with the instructions), or with the Cocoon build process.

The deprecated function name request:request-parameter had survived in the project XQuery from years ago. This is deprecated, and will be removed from eXist 1.5/1.6, so I've changed all the instances to the current request:get-parameter, and tested. All OK.

Wrote the first class, for zipping an object and getting the size of the zipped stream, and wrote some JUnit tests for it. Working well. Did this in NetBeans. Takes a while to get familiar with Java again.

Added the menu entry for the references, and also updated the accordion.js file with the new menu entries (that hadn't been done for a while).

I've started work on the highlighting of search hits inside documents when you jump to the document from a search hit. This is what I've done so far:

• Search hits in the results are now links to the documents in the case of text documents.
• Their link also includes as parameters the searchType and the searchString, so the XQuery which retrieves a document can now know about these. The hit number is also included as a parameter, in case this is useful, but it may not be; we might have to create a unique id for each hit, and pass that id into the document retrieval process somehow.