Devonshire

Got some collation files back from Cara, and pushed them into the DB. This threw up two problems, not with those files, but with the MS markup file: eventually narrowed those down to one markup error (extra <div> surrounding a poem), and one XSLT problem (<note> tags nested inside the poem were not being processed properly). Also fixed a problem with name lookups; that seems to work OK now on all the poems we've tested.

Checked with Ray re a discrepancy in the requirements for rendering, whereby the old print-style rendering makes effective popups difficult to achieve. The outcome is that we should make the expansive view (where each reading gets its own line) the default, and go for highlighted popups there, while defaulting to an unhighlighted view of the line in the run-together print view.

Cara has found one bug in Apparatus, which seems to stumble over a blank "line" (i.e. an l tag with no readings) in the Collate file. I'll work on that tomorrow.

Documented installation and usage of the Apparatus application as thoroughly as I could in a simple HTML page, then packaged up the application as a zip file, with BAT and SH files for running it. Posted it on the lancenrd account and sent email to Cara et al. with the download link.

Added two improvements, the first at Ray's request, and the other because it makes sense:

• The popup message that shows the lemma and reading for categorization now shows two versions of each, one with the wrapping tags and one without. This involved an improvement to the utility function which spits out a dom node in XML code, making it recursive.
• When loading a Collate file, the app now pops up an input dialog asking for the xml:id of the poem. This is safer than leaving it to the user, because it's easy to forget to add the correct id.

Met with the ETCL folks, and got the following requests for fixes/changes/new features in the critical apparatus output:

• Sigla in the crit app should be clickable, leading to (or popping up) a proper MS description. This will be based on a mapping XML file for all the witnesses which Karin will supply.
• All readings from the same line should be displayed (initially at least) in one block, separated by a space 5ems wide, to conform with traditional practice. This should be switchable, though, because we have no space limitations and can give readers the option to see each item on a separate line if they wish.
• The user should be able to switch on orthographic variants, which are currently invisible.
• Instead of the word "Omitted", we should use "omit." in italics.
• Clicking on the line number should pop up a copy of the base text line, with the lemma highlighted (if possible -- this might be done with JavaScript).
• Karin needs to be a user in the DB, with permissions on the Devonshire collections, so she can upload apparatus files while I'm away.

Adding this as a task.

I have two examples of critical apparatus documents ready to work on now, so I started work on the XSLT and CSS to render them.

Working from Ray's 2004 Henry VIII edition as a model, I set up the XQuery to harvest the correct apparatus information from the database, and set up the XML output so the apparatus is internal to the document (and is marked as such in the <variantEncoding> tag). Right now, I'm suppressing the orthographic variants just by hiding them in the CSS, so they can be displayed if necessary. I have a couple of remaining issues (<del> tags in the lemma are giving the whole lemma, or sometimes the whole line, a grey background for some reason), but the basics are there.

In the process of working through the files detailed in the previous post, I made the following changes to the Apparatus app:

Added some proper error handling for the condition where the template.xml file is not found. Previously, this would prevent the app from saving data, but now it can just save a text file in place of the full XML file, so there's less risk of data loss if the template file is missing.

Built a manifest file to include version info in the package. However, I'm not sure whether this is done correctly, and whether it's actually working. The info on this is sparse.

Tested on Ubuntu again, just to make sure the app remains portable after recent rebuilding of the GUI.

Today's progress, on the LDev003 and LDev004 files:

1. Re-generated them from the Collate files with my application, which fixed the problems caused by commas vs semicolons etc.
2. Categorized all the readings using my app -- that works fine. I've followed the categorizations Cara added last time around.
3. Worked through Cara's comments in the original files, and fixed remaining problems where I could.
4. Added comments for all items I'm not sure about, or where there needs to be further input from Cara and/or Ray.

Sent the results to Cara, with explanation, and the following notes:

One fairly common problem relates to additions in the witnesses. The word "added" is often, but what that means is not usually clear. For example:

Perhaps the "addition" is actually an addition in the witness (which should be tagged with an <add> tag), or simply a word which exists in the witness but not in the base text (in which case it should just be in the reading). For instance, if the base text has this:

    one three

and the witness has this:

    one two three

then the app tag should look like this:

    <app>
        <lem>one three</lem>
        <rdg>one two three</rdg>
    </app>

although it could also look like this:

    <app>
        <lem>one</lem>
        <rdg>one two</rdg>
    </app>

or this:

    <app>
        <lem>three</lem>
        <rdg>two three</rdg>
    </app>

The main point is that there must be enough in the <lem> tag to show where the extra bit in the witness actually shows up.

However, if the witness itself has "two" in the form of a scribal addition, then the app tag should look like this:

    <app>
        <lem>one three</lem>
        <rdg>one <add>two</add> three</rdg>
    </app>

Does that make sense? I've tended to use the <add> tag, especially where you mentioned using it, but I suspect that usually we're not dealing with additions in the witnesses, we're dealing with bits of text that show up in the witnesses but not in the base text -- not the same thing at all.

I think these files are close enough for me to use as the basis for writing XSLT/output code for the Web view of the MS, though, so I'll start on that next week.

New features/tweaks added to Apparatus today:

• Readings are now separated based on both commas and semicolons, so it can handle Collate's inconsistency in this respect.
• The Categorize function now shows the full <lem> and <rdg> tags as XML code, to provide more info for the user. This is based on a new utility function called serializeDomNodeToString, in a class called MdhXmlFunctions in ca.uvic.hcmc.utils.
• The Categorize function now provides a counter to show you how far you've got through the process.
• It also provides an Abort button so you can cancel the categorizing operation whenever you wish.
• Button icons are now larger, and buttons also.

The next stage is to work through the two files I have from Cara, and add the appropriate comments and type attributes, then I'll have two basically working files.

Worked through Cara's comments on the two collation files she's processed, and sent this back to her (archived here because I need to keep track of it):

Just working through the comments in your file:

I noticed that you've marked the orthographic readings rather than the substantive ones. I think that's the reverse of what we agreed; since orthographic is more common, it should be left unmarked, and only the substantive variants should be tagged as type="substantive". That's what my app does (I got it working yesterday). If you think that's wrong, let me know why. It just seems to me to make more sense to have less code to maintain.

With regard to this:

	<!-- I'm not sure what to do with items like this where something has been deleted. -->
	<app loc="2">
		<lem><del>was</del></lem>
		<rdg wit="#LDev145">/</rdg>
	</app>

I'm not sure what's actually being encoded here. I'd read the XML like this:

• The base text had "was", but "was" was then deleted in the base text.
• The witness had a diagonal stroke (slash character) instead of the deleted "was".

If that's correct, then you don't need to do anything. If that's not what's represented by this app tag, could you tell me what it should be?

In this case:

	<app loc="2">
		<lem>taken</lem>
		<rdg wit="#LDev145" type="orthographic">takin</rdg>
		<rdg wit="#STC13860_12">was added</rdg> 
<!-- Can we change this so it reads "was taken" and the "was" is tagged as an addition? -->
	</app>

you could change the second <rdg> tag like this:

<rdg wit="#STC13860_12"><add>was</add> taken</rdg> 

Remember that what is in the <rdg> tag is taken to be a replacement for what is in the <lem> tag. Here, that means that "was" has been ADDED in the witness (it wasn't in the witness originally, but a scribe has added it). If what you mean is that the word "was" appears in the witness, but not in the base text, then this is all you need:

<rdg wit="#STC13860_12">was taken</rdg>

meaning that where the base text has "taken", the witness has "was taken".

On your comment here:

	<app loc="5">
		<lem>they slave to remain</lem>
		<rdg wit="#AAH06">thie servant to remayne</rdg> 
<!-- I'd break this one up into word-by-word comparisons. -->
	</app>

if it is a good idea to break it up into separate <app> tags, then you'll have to do it manually, unless you can get Collate to do it for you initially. My program can't really deduce whether a long phrase should be broken up or not. In a case like this:

	<app loc="9">
		<lem>my desyer</lem>
		<rdg wit="#LEge20">thy desire</rdg> 
<!-- split up: thy is substantive, desire is orthographic -->
	</app>

you'll need to split up the code manually into two app tags. I could write a routine that would enable you to specify an app tag and have it broken up automatically, assuming that it will always be the case that every word in the lemma has a matching word in the reading (i.e. they both have exactly the same number of words), and they should be split at word-boundaries. This would be worth doing if this situation is likely to arise frequently; if it's more often the case that the word-numbers won't exactly match, or the app tag should be split, but not into a new tag for every word, then it's going to be more straightforward for you to do it manually, I think.

On this one:

	<app loc="12">
		<lem>ploues</lem>
		<rdg wit="#LDev145" type="orthographic">plowithe</rdg>
		<rdg wit="#AAH06" type="orthographic">Plowithe</rdg>
		<rdg wit="#STC13860_12 #LEge20" type="orthographic">Ploweth</rdg> 
<!-- this is a mistake that Collate keeps making. "Ploweth" is omitted in LEge20 -->
	</app>

Looking at the original, I think this might actually be a mistake my app is making, not Collate. However, it's triggered by an inconsistency in the Collate output:

ploues ] plowithe LDev145.txt , Plowithe AAH06.txt , Ploweth STC13860_12.txt; omitted LEge20.txt

Here, the ] delimits the lemma from the readings, and a comma should delimit each of the individual readings. However, Collate seems to have arbitrarily decided it's going to use a semicolon to delimit the last reading, from the LEge20 text. Is this configurable? If so, can we make it consistently stick to the comma? My app is conflating the last two readings because it assumes the delimiter is the comma. In the meantime, I can try to rewrite my code so that it will work with mixed delimiters. I suspect that will be a bit more error-prone, but maybe not.

Whatever happens, I think the workflow is going to require checking from you at each stage. We can make it go more quickly, but I don't think we'll be able to automate it completely, especially when Collate is less than 100% reliable.

I'll go back to my app and try to rework the problem above. When I've done that, I'll work from your examples and run the same two files through my ap, using it to categorize the rdgs as type="substantive" or not. Where you've marked items which need further thought from you or Ray, I'll put your original comment back in.

Eventually fixed the scrollbar problems by deleting and recreating the controls. Somehow the scrollpanes had become disconnected from the text areas, and there seemed to be no way to re-connect them.

Once that was solved, I was able to get back to working on the XML output. I found that I was able to get the text content of a node using the DOM level 3 getTextContent() method, now available in Java 1.6, which simplified getting the reading and lemma text for display. Popping up dialog boxes for choosing whether a variant was substantive or not proved fairly simple, and it was easy to add/remove/modify the value of the type attribute on the <rdg> tags based on the response.

What proved remarkably difficult was serializing the XML back out as text, to go in the textarea for saving. For the record, here's the code I ended up with:

    private String saveDomDocToString(){
        try{
            javax.xml.transform.TransformerFactory transformerFactory =
                            javax.xml.transform.TransformerFactory.newInstance();
            javax.xml.transform.Transformer transformer = transformerFactory.newTransformer();
            transformer.setOutputProperty(javax.xml.transform.OutputKeys.OMIT_XML_DECLARATION, "yes");
            java.io.StringWriter xmlout = new java.io.StringWriter(); 
            javax.xml.transform.stream.StreamResult result = new
                                javax.xml.transform.stream.StreamResult(xmlout); 
            transformer.transform(new javax.xml.transform.dom.DOMSource(doc),result);
            return xmlout.toString(); 
        }
        catch(Exception e){
            JOptionPane.showMessageDialog(this, e.toString(), "Error serializing the XML",                                            JOptionPane.ERROR_MESSAGE);
            return "";
        }
    }

I've tested this fairly thoroughly, even injecting Japanese characters into the input from Collate and seeing them successfully work through to the output, so I think this is robust from the character encoding point of view. Next I need to look at the notes Cara gave me in edited files last week, and see what other concerns remain to be addressed.