Capital Trials at the Old Bailey

Met with SD yesterday to discuss his questions and concerns about the structure of the XML file. He's been proofing the data for the last couple of weeks. As a result of his reading, he's realized that the current XML structure is lacking in a few areas. So, we're going to make the following changes:

• recorder_report (rec_rep in the XML) currently "has many" cases (case in the XML). rec_rep also "has one" rec_rep_head, which has rec_rep_start_date, rec_rep_end_date, rec_rep_pub_date, and rec_rep_notes elements. According to SD, in some cases the recorder report dates can change for each case within the same recorder report. So rather than rec_rep_head being a direct child of rec_rep, it will be moved to being a child of each case WITHIN the rec_rep. So the hierarchy will be: rec_rep -> HAS MANY case -> HAS ONE rec_rep_head
• respite elements will have a child respite_ref field so that SD can add in any possible references for a respite.
• A new element, "judge_respite", is being added as a trial_file child. Unlike the trial file -> HAS MANY respites relationship, a trial file only HAS ONE judge_respite.
• execution_mode, which is a child of outcome, is going to be expanded to allow for multiple elements. So the relationship will become: trial file -> HAS MANY outcomes -> HAS MANY execution_modes. This is an optional field.

I also approached SD about using an Excel spreadsheet to gather his raw data, rather than a word processor (which is his current method). Since the current method to convert his raw data to XML is quite cumbersome and error-prone (not by any fault of SA, who did an admirable job with GREP), moving SD to a spreadsheet will make the conversion process faster and more streamlined. I think it'll also make his own work simpler since he'll be able to find and manipulate his data much more easily.

To add to my list of SIMILE and Google Visualization, I'm also investigating Flot, a plotting library built on the JQuery framework. It looks to do almost exactly what I need it to do, and its learning curve is not nearly as steep as the other tools. After some fiddling and exploring Flot is the front-runner.

I spent a large chunk of today exploring possible Javascript-based visualization tools for the histogram feature, which is intended to show trends over time based on any number of criteria. There seem to be two clear frontrunners:

• SIMILE Widgets ( http://www.simile-widgets.org/ ), specifically Exhibit and/or Timeplot.
• Google Visualization API ( http://code.google.com/apis/charttools/index.html )

Both are good options but quite complex and, as always seems to be the case, fall just short of the functionality needed. Writing some custom code might be unavoidable.

Made some good progress on the search interface today. Instead of having one Search class to handle both searching and returning one record, I wrote a separate Search_Single class that handles fetching a trial file based on a passed ID. Since a trial file can have multiple trials, respites, and outcomes (but only one criminal), the Search_Single class grabs those associated records via separate queries. Meanwhile, the main Search class simply does a LEFT JOIN on the trials, outcomes, and respites table to find at least one record that matches the search criteria.

I chose the trial_file as the 'main' table for the search because it's the parent for trials, respites, outcomes, and criminals. Since there's only one criminal per trial file but potentially multiple trials, respites, and outcomes, the trial file makes the most sense because we don't want the same criminal to show up multiple times in the search results, which would happen if we were to search with the trials table as primary.

I've sent the 1800 decade XML data file to SD for an initial proofing. Stewart did the bulk of the raw-to-XML transformation using a series of GREP statements. For this final transformation, I used XSLT to convert raw text to 'normalized' values that match against the RNG validation file. Here's how the validation currently breaks down:

• The XML file has 41535 lines
• 116 of these lines (0.003%) do NOT validate
• Of these errors, nearly all of them (100 / 116) are either empty jury_type elements or empty crime_text elements
• The rest of the errors are mainly due to crimes or outcomes that didn't fit into any category (at least, to my untrained eye)

The remaining errors will need a 'trained' eye to be fixed on a case-by-case basis. Once SD is happy with the data then he'll return it and I'll perform a further transformation that will import the data into the MySQL database (for which the XSLT spreadsheets have, luckily, already been written).

I've also been spending some time this morning ensuring that the XSL files used in the transformation are well-documented.

The XML data file for the 1800s decade of data is down to 0.003% invalid lines, almost all of which are empty jury_type or crime_text elements.

While writing the XSLT stylesheet for the Bailey project XML data, I stumbled across a misleading 'gotcha' when using the XPath function empty(). Basically, empty() will ALWAYS return false as long as the element being tested exists. So, if your XML chunk looks like this:

<alias />

And your XPath looks like this (in this case, within XSLT):

<xsl:when test="empty(./alias)">

That test will always return false since <alias> exists. To test whether an element is empty, use the string() function. For example, if we want to determine whether <alias> is empty (which it is):

<xsl:when test="not(string(./alias))">

string() will return the string value of the element, which, in this case, is empty (but not empty() - hah).

P.S.: Martin reminded me that empty() is actually for sequences, which explains the weirdness when used with an element. But that, of course, leads one to wonder why empty() doesn't throw an error when used with a non-sequence...

In preparation for the completion of the XSLT stylesheet that will transform the XML data into a valid data set, SD has made some revisions to the list of outcomes and outcome groups. These revisions are mostly just involve refining current outcomes and making them more explicit (e.g. expanding "Executed" to "Hanged", "HangedOnSite", "Gibbeted", "Dissected", "Burned", and "Drawn").

I've got some work ahead of me to integrate the new outcomes - mostly just accounting for them when encountering irregular data in the XML - but for the most part it should be fairly straightforward. The new outcomes are in the XSLT, so now it's just a matter of adjusting the stylesheet.

Met with SD today so that I could get more familiar with the project. SD went over how he'd like the data displayed, and we talked about the current state of the project in general. He is pleased with the progress and is excited to see what lies ahead. He'll double-check the list of crimes, respites and outcomes before I send him the XML file for proofing. After we hammer down the conversion and validation process for the current data set (1800s), then the rest of the data (i.e. the rest of the 19th century) will fall into place relatively quickly.

So, the project is progressing well and both SD and myself are happy with the results thus far.

After much massaging with XSLT, the 1800s data XML file, which began as a raw RTF document and was transformed to an almost-well-formed XML file by SA, is almost error-free and ready to go to the proofer. In approximately 41,200 lines of data, there are 116 validation errors, which will need to be corrected manually by SD or his proofer. I tried to cover all of the irregularities when writing the XSLT stylesheet, but these remaining errors need to be fixed by someone with the raw data in hand.