Devonshire

Testing the app without persistent state was getting annoying, so I did a bit of research and discovered the java.utils.prefs features for storing and retrieving preferences. Documentation is pretty sparse, but eventually I found this tutorial which shows how easy this stuff is to use. I soon had the app saving and restoring its window position and size, and the position of it splitter bars. That's all I need for this app, but future apps will probably benefit from a full class which saves the state of all relevant components in a JFrame, like the FormState class I wrote for Delphi.

The Preferences class stores data in an appropriate place based on the underlying platform; on Windows, that turns out to be the system registry, and all user-level Java app preferences are stored in the same location: HKEY_CURRENT_USER\Software\JavaSoft\Prefs. That means it's important to have your class hierarchy properly rooted and unique, so I refactored the application so that its main package is ca.uvic.hcmc.apparatus.gui.

Managed to refactor the GUI of the app by starting a new JFrame, building the new structure, and then copying and pasting the top-level components over. In the process, multiple instances of some components were added, and could not be found for deletion, and multiple consecutively-numbered instances of their event handlers were also added. I fixed all this by editing the .form and .java files in a text editor, with the IDE shut down. Once I'd got the new GUI working as the old one did, I shifted the old into an archive package (which required me to add to its import clause the libraries from the main package on which it depends. I'll delete it as soon as I'm sure the new one is fully functional.

Next, I wrote the code to factor out the individual readings and witness sigla. This seems to be working just fine; I now have a block of <app> tags which reflect the Collate output correctly. Now all I need to do is figure out how to create a complete TEI file including this info. Then I have to save the file.

I've also added a label and text box for the base text id, which will need to be included in the output file. This can't be completely derived from the input file name, but enough of it can (e.g. LDev003) to hint at what it should be, and how to find it.

Made more progress on the Java application (learning all the time):

• Input text box now displays line-breaks correctly. Previously, incoming linebreaks (u000d) were ignored (presumably because of the platform). I now convert them to u00du00a (=\r\n), which result in real line breaks in the edit control; the processing code also now looks for that combination. It remains to be seen whether, on other platforms, these show up as double line breaks.
• The file opening code is now able to search for sigla and automatically populate the sigla table with values. It does this by searching for anything ending in ".txt" delimited by whitespace, which seems to work reliably for our data set.
• All blank lines in the base text are now correctly detected, and shown in the output as an empty <lem> tag, with a comment that the line is missing in the base text.

The final stage is to parse out all the witness readings and create <rdg> tags. To do that, I'll have to split each line based on siglum+whitespace+comma, to get each reading into a separate string, then read the string up to the first siglum to get the reading, and the subsequent sigla to go with it.

Meeting with Ray and Karin, during which we decided:

• We'll work from the output of Collate, because this is what's working best for Cara, so I'll continue work on my Java conversion app.
• We'll keep watching the Juxta project with a view to using it in the future, and if we get the time, we'll go through the testing process again and submit bug reports on their SourceForge site.
• The witness list will constitute a single separate file.
• The apparatus for each poem will constitute a single separate file.
• The browser rendering will provide a full traditional apparatus below the poem, but will also pop up a copy of the apparatus for any given line when you click on a symbol in its left margin.

Karin has finished the new version of the XML file, so I can start rewriting the XSLT. One problem is that the change to TEI which allows spaces between lines has not yet found its way into ROMA, so we'll probably have to knock up a little ODD file to make that work; instances of spaces between lines are currently commented out.

Also discussed the collation a little; it seems Cara has finished about half the poems, although I have only 26 of her collations. Given this, it seems even more sensible to finish the Java app that will generate our P5 versions from Collate's output.

I've continued work on my Java app to generate P5 critical apparatus from the output from the old Mac Collate program. I'm now able to generate all the <app> and <lem> tags, and I've created a table where the user can specify the sigla of in the input file, and alternative sigla to be used in the output if required. I'm now figuring out how to parse individual lines into different readings, and learning in the process how to use Java vectors. The NetBeans IDE is still working well, with the exception of some of the property editors for Swing components -- the table editor is especially flaky, and best avoided.

I've been testing JUXTA, and I find I'm not unimpressed with it so far. First, I did a simple test with three plain text variants of a Larkin poem. The application found all the differences with the exception of leading spaces (indents on even-numbered lines), and it generated a useful apparatus. However, it numbered the lines including breaks between stanzas, so the first line in stanza two of was numbered 6 instead of 5. This is despite the fact that I set it to Ignore Whitespace. When I set it to pay attention to whitespace, in order to see what it would do with the missing spaces at the beginnings of lines 2 and 4 in variant 2, I saw some very odd behaviour relating to the combination of "Ignore case" being ON, but "Ignore whitespace" being OFF. In the end, I found I had to turn all three Ignore switches OFF (whitespace, punctuation and case). If I didn't do that, I would get an apparatus that suggested there were differences in capitalization when actually there were differences in spacing.

Having done that, I found that the line numbering was a total mess. The program found two variants on line one (the second of which was actually on line two), and then placed a fourth-line variant on line three when it was actually on line four. This was using a VERY simple input set. My conclusion is that JUXTA is not usable with plain text input.

Next, I tried markup up the poem variants according to their guidelines (JUXTA uses a proprietary XML format). The first thing I discovered is that whitespace matters in the GUI of JUXTA; if you pretty-print or indent your tags in the XML file, the output will retain some (not all -- it's inconsistent) of that indenting and whitespace, making the appearance of the poems harder to parse in the JUXTA GUI itself. The collation it generated was a bit better and more reliable, but still it was a little odd; missing spaces were signalled by a ~ character with no indication of where in the line they were missing from, and an inserted line in variant 2 was identified as such, and located in the right stanza, but it wasn't clear where it had been inserted.

Nevertheless, it seems as though this might be a useful tool, assuming we have the resources to mark up our source texts. I'm going to look at the actual code of the app, and see if I can figure out how we might improve or work around these issues.

I need to get up to speed with Java to write some stuff we require for special processing in Cocoon, for a number of projects, so I'm taking this opportunity to write a little text file processing tool in Java, rather than in Delphi. It'll make it more portable too.

Worked through the NetBeans 5.5 tutorial on basic Swing GUIs, and got a GUI written for my application. The new GUI design tool combined with Java 1.6 is a great step forward, and it's now almost as easy to design the basic GUI in Java as it is in Delphi, although obviously there are lots of aspects I don't yet understand. The Java 1.6 requirement means it'll only run on Windows and Linux for the moment -- the Mac JVM is behind -- but that's a temporary limitation. Got the GUI resizing OK, figured out how to pop up a dialog box and load a file (pinched some file loading code from Stew). Then I started writing the string processing.

The idea is to parse out the output from Collate to produce a series of P5 <app >elements containing <rdg> elements and an optional <lem> element. I didn't get as far as generating working output -- I'm stuck on an array index out of bounds error -- but I'll get there pretty soon. Then we'll be able to use Collate to generate the core data, and process it into P5. We should also be able to automate the production of the base text from the main DMS transcription using XSLT, so the whole process will be fairly streamlined except for generation of the witness transcriptions.

I've begun work on the critical apparatus issue, building on what Cara has already done, reading through all the relevant P5 Guidelines sections and playing around with JUXTA. There are a number of questions we need to answer before we can move forward with this, and I think they'll need some thought and discussion:

1. I'm starting from the assumption that we want our witness and collation data to be encoded in P5, but we still need to decide exactly how that should be done. Specifically:
   • Should witness and collation information be kept separate from the main transcription document? (I think this is advisable, because it will bloat the document and make it hard to edit otherwise.)
   • If so, should witness and collation information for each poem be kept in a separate file, or should they all be encoded in one large document? This rather depends on the number of witnesses each poem has in common. If many poems in the base text have witnesses that appear in the same external source, then it would make sense to centralize the witness and collation information into a single file. If, on the other hand, there are few instances of this, then we could create a separate witness/collation file for each poem. The latter will obviously result in smaller files which are easier to edit and manage.
   • If the apparatus is external, how should it be linked into the locations in the base text? The most appropriate method recommended in the Guidelines would seem to be location-referenced, with a loc attribute on the <app> tag, its value consisting of the xml:id attribute of the target poem div, followed by a space, followed by the line number in the target poem. Will this work reliably? Are there circumstances in which variants will need to be encoded which refer to a location in the base text which is not identifiable in this way?
   • Should metadata about the witnesses themselves be kept in the same file as the collation/variant data? All witness metadata could be listed in a separate file, referenced by all the collation files. This would centralize and information and ensure that only one copy of the same information exists (i.e. there's only one description of any particular witness). On the other hand, if there are no witnesses in common between different poems, then there's no point in this; we can keep witness metadata in the same file as the apparatus itself.
2. How are we going to generate the P5 collation information? There seem to be two approaches to this: one is to use Collate, as Cara has done, and then write a little application to process its data into a P5 file. I'm going to write a little Java thing to do that. The other is to use Juxta, which I'm investigating at the moment. Juxta can produce an XHTML apparatus, which I think could be fairly easily processed into P5 using XSLT. Cara has reported that she got unreliable results with Juxta, so we'll have to test it carefully to see if we can rely on it.
3. We need a set of sigla (taking a form which is usable as an xml:id attribute) for the witnesses. The collations Cara's created with Collate use the file name of the text file containing the variant. This would do perfectly well, but we need to list and document all the sigla in a <witList>.
4. How are we going to output the information? Ray wants to see something that looks like a traditional critical apparatus, which should be easy to reproduce, but where should that appear -- at the bottom of the poem page, or alongside it? If alongside it, how do we tackle the problem of the apparatus having more text than can be accommodated while keeping it synchronized on the page with the base text? Should lines for which there are variants have a little clickable symbol next to them, which will pop up the apparatus for that line?
5. Do we intend to make the apparatus so precise as to enable the complete reconstruction of all witnesses from it? If so, we'll need to be much more precise in encoding the start and end points of each apparatus entry. Personally, I'm not sure that this is worth the trouble, because in any case we'll have to generate either text files (if we choose to continue using Collate) or special markup files (if we use Juxta) for each witness anyway; it would make more sense to read and output those if necessary, rather than laboriously reconstruct variants based on our apparatus markup. However, the ability to reliably reconstruct texts would give us more flexibility in terms of creating interesting and innovative display formats in the future.

I'd like to get some feedback on these questions before I carry on.

First, install and test the app to see if it's stable and reliable (Cara reports inconsistent results on her tests). The docs make it look fairly straightforward; Stew and I had a look at it today, and read through all the Help.

If it works OK, go to the source code and see if it's going to be possible to extend it to make it produce output in TEI format.