Mariage

This post describes the problem I worked on this morning. It turned out to be much more complicated than I'd anticipated. It wasn't an XSLT problem at heart; it was an XQuery issue. The root of it was this: During the process of creating a list of documents for the contents pages, the XQuery makes copies of the <biblStruct> and <bibl> elements inside the <sourceDesc> of each document. However, in the process, it needs to inject some extra attributes into them, so it can't just copy them to the output; it has to reconstruct them. However, the bibl element content is mixed content (or at least, it was till today) when titles and so on are mentioned. There's no easy way to copy mixed content in a reliable way; you end up getting all the elements followed by all the text, or the elements don't get through at all, or any number of failure patterns depending on the strategy you take. In the end, I decided on a compromise solution: where the <bibl> element contains mixed content, that content should be wrapped in a single <note> element, so that it's copied intact to the output.

Having fixed the output of the 1621 Sonnet document by the addition of a <note> element inside the <bibl>, I then discovered that the XSLT in index.xsl was not expecting any mixed content, so it just copied text to the output on the page. Once I changed the value-of to an apply-templates, I found that there weren't actually any templates for title elements at all (in this case, I needed monograph titles to appear in italics), so I added those to teiGeneral.xsl, along with appropriate classes in the CSS file mariage_layout_typography.xsl.

The difficulties thrown up by what looks like it should be a simple processing issue are typical of a project which has tended to grow organically rather than being planned at full scale before the markup was done. But that's the kind of project we have, for better or worse.

Fixed a couple of typos at CC's request in the Sonnet files, and in the process included her short descriptions of the texts, which go into <bibl> elements in the <sourceDesc>. This threw up a new problem. Up to now, these descriptive bits have been one-line plain text items, so the handling code is not expecting anything like titles or italicized text; these new descriptions include text titles that should be italicized. I've had to remove that from the content, until I can rewrite that code. Adding this as a task...

PG and LW are now comparing and proofing their markup, and as part of the process I've posted the Sonnet texts online. There are many display issues -- most are certainly caused by me (in other words, they're caused by my not having written the required display code for features of these texts which didn't appear in other texts). Some, though, might be due to markup issues, so the RAs can fix anything that's obvious as part of their proofing, and can also flag the other issues. Here are some I already know about, for the record:

• If you go to the 1609 text and search for "Mercure , penſé", you'll see that the following line, which has an <unclear> tag in the XML, just appears as if it were normal; in other words, nothing signals the "unclear" status of the text. This is my fault, because we hadn't used <unclear> before so I haven't written any handler for it.
• I'm displaying the folio page numbers when I shouldn't be -- only page numbers in <fw> tags should be displayed, not those in <pb/>s.

Including this as a task, so I remember to get around to working on these specific items.

As suggested in the previous post, I've now made ids unique across the two documents, rather than just the one.

I've been looking again at how the jclCompression libraries work, and it's sent me back to some thinking about how I should be saving this data in the first place. It needs careful planning. There's a range of conditions we need to avoid and allow for which make the simple bundling of files into an archive inadequate. These are my decisions:

• We'll use a plain zip format. Compression/file size are not major issues, and zip is very portable.
• Since the two input files ("left" and "right" texts) might have the same filename (being taken from different locations on a disk, but named identically), we'll need to store some info about them which specifies exactly where they came from. This means we can't zip them with their original names, so we might as well zip them with "left.xml" and "right.xml".
• Other files include similarities.xml (the current state of the similarities list), a putative combined file (described in a previous post, based on teiCorpus), and a metadata file.

There are two or three initial things I need to do before actually implementing a file save:

1. Rework the id-creation code so that it enforces uniqueness across the two files, not just within one file. This is necessary since we'll be creating a single corpus file incorporating the two documents. Perhaps we should think about the possibility of users creating multiple-document archives here, and try to create ids which are not only unique across two documents, but have more chance of being unique even when more files are added. However, the user can themselves enforce this right now by setting the id prefixes in the settings XML tab, so maybe we can just leave that to the user.
2. We need to figure out an appropriate metadata file format, which can be used to describe the contents of the archive. Dublin Core in RDF???
3. Figure out how to structure the corpus file, and create a routine for building it.

It's become clear that what we actually have here is a "project", and that we ought to be able to save all the data (source XML files, NCD values, lists of modified lines, current state of processing, settings in the main screen etc.) in one big file -- a compressed archive -- and reload it. I've begun creating the infrastructure to support this (the usual Modified variable, filename tracking, SaveChanges prompting etc.). I've got the New and Save stuff all written with the exception of the actual code which saves the file; this is going to be based on a TJclCompressionArchive object, which looks simple enough from the one demo that comes with it, but I don't know if it'll work out in the end. I would have used the open-source ZipMaster component, but it's currently uncompilable in Delphi 2009. I don't see why the JCL component won't do the job, though.

The task of confirming or rejecting proposed matches may be a long one -- in fact, in the case of our Sonnet data, it will be very long -- so we need to be able to save and reload the current state of the task. I've added a property called currItem which is set by the review window when it closes, and saved to disk as an attribute of the root <similarities> element in the data file.

Woke up with some good ideas about how to allow the user to vet the correspondences that have been found. At the end of the comparison and adjustment process, the matches are now sorted by similarity with the most similar first. Then the app pops up a new window with a list of the lines to be matched in two listboxes, and displays each match as highlights in the two list boxes, allowing the user to specify whether a match is a direct correspondence, and interesting parallel, or is completely unconnected. They can work through the entire set like that, or when they've done all the significant matches, they can stop. I've also added a feature so that you can double-click lines to select a match which is somewhere else in the list directly, and then specify that as a good match. Once a correspondence is made, its lines get a grey background, so you can see the remaining unmatched lines easily.

Next I think I need to do this:

• Allow the user to choose to discard all non-significant matches before saving (that would be all matches which are below a certain level, or more likely, those which are etNone in their equivalence field (meaning the user has not specified them as matches or interesting correspondences).
• Create a file format in which all this data can be saved and reloaded as a block. The obvious thing is a zip file containing all the input and output files. This would enable you to stop work on a set of correspondences, then start again later.
• Write save/load code for the file format.
• Add one additional file type to the output format: a collocation of the two documents as a <teiCorpus>, incorporating with all the match data.
• Write XSLT for the document above so that it can be displayed as a "critical edition" based off either of the two texts.
• Figure out if it's worth writing code to incorporate more than two texts, or whether the user should do that manually.

Found the JEDI jclCompression library has implementations of BZip2 and GZip, so I've added those as options, and also added an "Average of all 3" option. The results from each of these are quite different, but they agree from a relative point of view; ZLib is always highest, with GZip next, and BZip2 usually lowest, but they're equally good at distinguishing between texts. The differences are noticeable, and none of them now agree with the results from NCD on Linux, which is odd; LZip did in my earlier testing. However, there are now lots of options for calculating the results, which is useful, and we should be able to figure out what provides the best algorithm by producing good human-readable output based on the results. That's my next task.

Reworked the GUI to make room for more user controls, and added recursion options for the adjustment calculation, then rewrote the code to allow for controlled recursion with a breakout if no further changes are happening, and progress updates. Then we ran some tests. We can do 999 recursions on our test data without achieving stability, and the results are actually not good; there's a tendency to end up with a spurious mapping between the lines such that one strong correspondence in a block can cause the rest of the alignment to fall in with it. There must be a sweet spot at which you've maximized the returns from the contextual assessment, without allowing "strong" areas to take over their surrounding correspondences. We need to do more testing here to see whether there's actually any value in recursion or not.