Devonshire

Began working on the parsing of the generated TEI code, and got as far as finding and iterating through all the <app> elements and finding their <lem> children, but then got stuck on getting the node value of the <lem> element. I probably have to get its text child element and use that.

Then I hit a severe problem with the GUI, which is very puzzling. All of a sudden, text areas don't scroll, even when full of text. I've tried everything to make them do so, but nothing works. This is a bit of a show-stopper, so I'll have to deal with it before I get onto anything else.

Made a few steps forward:

• Fixed some GUI issues when the app starts for the first time, with no settings in the prefs. You have to be careful to get the initial values right, otherwise some components can be sized badly.
• Added icons to the buttons (mainly for Pro-D, but in the process I extended my CarefulResourceLoader class to handle Icon as well as IconImage).
• Figured out how to load an XML string from a text box into a DOM. This wasn't obvious from the standard examples, which assume you're loading a file from disk. For the record, here it is:

  Document doc;
  DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
              DocumentBuilder builder = factory.newDocumentBuilder();
              String xml = "<?xml version=\"1.0\"?>\r\n" + jtpOutput.getText();
              doc = builder.parse(new InputSource(new StringReader(xml)));
  

Next is using DOM methods to iterate over the readings and let the user choose whether they're substantive or not; then add the relevant attributes; then write the doc back as text to the text box.

Quick meeting with Ray and Cara in which we discussed how best to enable the production of collations in TEI to proceed quickly. The best approach appears to be:

• Cara will use my Apparatus app to generate the TEI.
• She'll tweak Collate output where necessary before generating TEI.
• Then she'll use a new feature I'm going to add to my app, which will do this:
  1. Run through each <app> tag in the XML.
  2. For each <app> tag, go through each <rdg> tag.
  3. For each <rdg> tag, pop up a dialog box which shows the lemma and the reading.
  4. In the dialog box, show a checkbox which the user will check if the variant is substantive.
  5. At the end of the process, write the decisions back to the XML document in the form of type="substantive" attributes on the <rdg> tags.
• Next, she'll check the XML manually to make sure there are no errors.
• Once that's done, she'll set the xml:id attribute for the poem and save the file.

This will take me a little while to code, because I'll need to learn about DOM manipulation in Java, but it's all stuff I need to know about anyway.

Got two new collation files from Cara, for LDev003 and LDev004, and began tweaking the Apparatus application code to handle the new setup:

• Added clauses to detect if the new line number encoding is being used instead of the old. The app can now handle both <l n="1"> and <l 1>.
• Added some parsing to find the square brackets used instead of angle brackets for TEI tags. This is slightly complicated, because a closing square bracket is also used to delimit the end of the lemma, so the code only converts paired square brackets, leaving the orphan delimiter intact.
• Noticed that [sd][/sd], which is not a TEI tag, appears in the data. Karin tells me it's a substitute for <>, meaning scribal deletion, which causes problems in Collate. Added code to convert that back to l>>.
• Commented out code previously added to replace substitutions for square brackets ($ and _), since these are no longer being used, and the underscore causes problems because it's used in witness filenames.
• Added linebreaks and indents to format the output more readably.

Having got the app working, I saved two output files and sent them to Cara so she can check them out and add the type="substantive"<rdg> tags) where appropriate (the default will be orthographic, since it's more numerous). Once that's done, I can start writing rendering code.

Met with Cara and worked through what Collate does and doesn't do, and came up with a strategy for both of us. Then I went back to my application and coded some of the consequences into it. Among the details:

• Angle brackets in XML tags will be substituted with $ and _ respectively, to distinguish them from the square-bracket delimiters. My app now restores them to angle brackets in the XML output.
• All readings should commence with a lemma (even if blank) and a closing square bracket, so any line which doesn't have a closing square bracket is a faulty hard wrap. My app deals with that by recombining it with the preceding line.
• The words "omitted" and "added" can be assumed to be intrusions by Collate. I will look at ways of replacing them with appropriate TEI, or perhaps square-bracketing them in the output.
• Line group tags will be omitted from Collate's output.
• Regularization will NOT be done at the level of Collate; after my app has produced TEI output, each reading will be marked manually as "substantive" or "orthographic", and the latter will probably be the unspecified default because there are more of them. Users will be able to suppress either type if they wish, in the rendering.

I'm now waiting on a new batch of collations from Cara to test my app out.

Having got the application to a point where it can do useful work, I began running it against the 26 collation files Cara has created. I found significant variations in how the Collate output is formatted between the files, as well as some new features that weren't in my test files, which I don't yet understand. Wrote to Cara with lots of questions, and a request to meet next week and work through the details.

After a discussion with Karin, we've arrived at a slight reformulation of the markup of notations and similar blocks:

1. There are several instances of <seg rend="below"> which occur inline; it seems simpler to insert linebreaks between them, instead of using a rend attribute. For instance, this trailer:

<trailer>
  <seg rend="box">finis</seg>
  <name type="author" key="TH_HOWARD">T. H.</name>
  <seg>
    <figure>
      <figDesc>arrow shaped drawing pointing to the right</figDesc>
      <graphic url="ms_images/figures/arrow_horizontal.png"/>
    </figure>
  </seg>
  <seg rend="below">
    <name type="author" key="TH_HOWARD">
      <unclear reason="faint">T Hou</unclear>
      <gap/>
    </name>
  </seg>
  <seg rend="below">
    <figure>
      <figDesc>arrow shaped drawing pointing upwards to finis</figDesc>
      <graphic url="ms_images/figures/arrow_vertical.png"/>
    </figure>
  </seg>
</trailer>

<trailer>
  <seg rend="box">finis</seg>
  <name type="author" key="TH_HOWARD">T. H.</name>
  <seg>
    <figure>
      <figDesc>arrow shaped drawing pointing to the right</figDesc>
      <graphic url="ms_images/figures/arrow_horizontal.png"/>
    </figure>
  </seg>
  <lb/>
  <seg>
    <name type="author" key="TH_HOWARD">
      <unclear reason="faint">T Hou</unclear>
      <gap/>
    </name>
  </seg>
  <lb />
  <seg>
    <figure>
      <figDesc>arrow shaped drawing pointing upwards to finis</figDesc>
      <graphic url="ms_images/figures/arrow_vertical.png"/>
    </figure>
  </seg>
</trailer>

Using a rend attribute suggests that the position is somehow unexpected, whereas in cases like this, all we're seeing is a normal line break, I think (or all of these segs are block elements, which amounts to the same thing).

3. The align values are slightly confusing, because in e.g. the case above, the intent is to align the text to the right _within_ the flow of the poem; in other situations, the intention is to specify that the text is floated in the margin, to the right or left. These two types of alignment need to be distinguished. I would suggest we do something like this:

• align(left|center|right) = inside the regular line;
• float(left|right) = floating in the margin

Furthermore, it makes sense to me that we should actually use CSS notation, which is more standard:

text-align: left|center|right; float: left|right;

This will help to clear up the confusion, I think.

4. Two slight inconsistencies:

• There are seven instances of "align(centre)" and seven of "align(center)".
• There is one instance of "below_align(right)" and one of "align(right)below".

I'm posting this on the blog so that we have a record of it. How do the two of you feel about this?

A while ago Karin gave me a new version of the MS document which has some updates (although not all of the ones which are planned, as yet, I don't think). This required some tweaks to the XSLT, which I've done. I've uploaded the new document into the db, and the site pages seem to be working fine.

Fixed the problem with extraneous <lg> tags, after confirming with Karin that all lines are consecutively numbered. Added more GUI extras to make life easier -- dialog boxes now remember their previous folders between sessions, and the ".xml" extension is forced when saving a file.

I also figured out the Look-and-feel issue; you have to set the look and feel before initComponents() creates all your components. I've now set it to use the look and feel of the platform, which seems to work; on Windows it looks like Windows, on Linux it uses GTK, and to test on Mac I joined the Apple Connection developer club and downloaded the last test build of their Java 1.6 machine (from September last year). This runs the app just fine.

Finally, I tried re-coding for backward compatibility with the 1.5 engine. One minor problem was the FileNameExtensionFilter class, which is new to 1.6; after coding around that successully using a class based on FileFilter, I hit a showstopper with the new GroupLayout manager, which the NetBeans IDE uses by default. That would take serious work to code around, so I gave up. We don't need this app to be very portable really.

The Apparatus app is now working; it can read a simple Collate file and output a complete, valid TEI file. I'm still waiting for a bit of feedback from the TEI list on how exactly the <app> tags should be included in the separate apparatus file; I wanted to put them into a div, but they can only appear at a lower level, so right now I'm putting them inside a pointless <ab> tag.

Looking at the other collations (I've been working mainly with the first, LDev003), I see that they're more complicated than the first:

For instance (working with the regColl version (which is the simplest to use), in LDev003 we see something like this:

<l n="1">
<l n="2">
yt ] / added LDev145-TM1039-TP1162.txt
<l n="3">
<l n="4">
<l n="5">
they ] thy LDev145-TM1039-TP1162.txt LEge20.txt

meaning that nothing happens till line 2, then we have one addition in one of the witnesses; then nothing till line five, when we have a slight difference in the two witnesses. My app can happily deal with this.

However, if I look in LDev004, I see something like this:

<lg n="1"><l n="1">
<lg n="1"><l n="2">
that whyche  ] the thing that STC_24650.5_05 STC_26053.5.3
<lg n="1"><l n="3">
hathe in  ] omitted STC_26053.5.3
<lg n="1"><l n="4">
but ] And STC_24650.5_05 STC_26053.5.3

In other words, line-group signals are included along with line signals. Now, these seem to be superfluous, in fact, because line numbering is consecutive through the poem, so line numbers are all we need for reference. Therefore the simplest solution might be just to strip out all the line-group signals before processing. However, I'll have to take a quick look through the remaining files before I know for sure if this is reliable or not.

I also tested the app on Ubuntu, where it works fine after installing Java 1.6. (OSX is out until Apple gets its VM up to version 1.6). Played around a bit with the Look and Feel settings to see what's available -- not much, really, except that on Ubuntu, as a user, you can force the app to use GTK widgets, which makes it look more like a normal application. I learned a bit about cursors and JFileChoosers too. This has been well worth doing from the point of view of learning Java, irrespective of the use for this project specifically.