Capital Trials at the Old Bailey

Revised the file bailey_text_to_fk.xsl based on the revised structure of the incoming text files. Tested that with the 1790-1799 data file and it seems to be working to produce a file called fkTextToInt.xml. The structure of the output is identical to the input except the jurytype and jurysubtype fields in former are collapsed into jury_combinedtype in latter. All the lookups to related tables that I can turn into ints I have.

That file is the source file for a further transform (bailey_add_id.xsl), which adds ids for links between trials, crimes, rec-reps etc.

Final tweaks on rng and data files to accommodate revisions to normalized respites and crimes. Involved a whack of search and replaces on the data files for content as well as for structural entities in addition to modifying the rng files.

Will write one XSLT file that turns all the texts of normalized values into integers corresponding to the id of that text value in the appropriate table. End result will be an XML structure identical to the one that went in, but with all the normalized, and group values modified.

I'll also migrate the code that modifies the data from the convention used in the XML to what the SQL requires.

That leaves another XSLT transform file which will take the output generated above and render SQL statements. Then I'll need to modify some javascript code to take all the id's generated by the XSLT (which are in the form X3Y124) and render them as integers. Result of that should be an sql file full of statements to populate the relevant tables with the data for a decade.

Have begun on the first XSLT file, got the crime-groups and judge-names working, leaves crime-normalized, respite-normalized, outcome-normalized, and probably jury-type and jury-subtype and a couple of others to do.

In the bailey_for_sql.rng and bailey_proofing.rng files for 1790s and 1800s, I updated set of normalized respites to include both respites of duration and respites of sentencing. Original list had been just respites of sentencing and first update had been just respites of duration.
<choice>
<value>Pleasure</value>
<value>PledBelly</value>
<value>JudgesConsideration</value>
<value>Within24HoursOfExecution</value>
<value>OneWeek</value>
<value>TwoWeeks</value>
<value>OtherSpecifiedTime</value>
<value>Transport</value>
<value>ToDie</value>
<value>Prison</value>
<value>Military</value>
<value>Free</value>
<value>None</value>
</choice>
Validation now finds no errors in the respite_normalized field in my copy of the 1790-1799 data and 1800-1809 data.

Also found all instances of "No respite" in the respite_text field and replaced them with empty string, as that's what was in the original record.

Modified the four rng files to allow an empty string in the respite_text field. Previously I had required at least one character.

Revised the data structure and the values in some of the normalized and list elements in the 1790s and 1800s data so they would validate against the new rng files. They are valid. One possible problem issue is that one of the old normalized crimes was larceny, which mapped to the burglary crime group. Under the new crime groupings a "larceny" instance can be mapped to any one of at least three crime groups. That's the only situation where the strict hierarchical relationships between text, normalized text and group elements is violated. (i.e. a given text is always normalized to the same normalized text and a given normalized text is always a member of the same group).

Revised the proofing and for_sql rng files in the 1790s and the 1800s folders based on the discussions with SD. Added some new elements: respite_duration, respite_group, mercy_appeal_why_group. Revised lists of legal values in a number of the normalized lists and group lists.

Did not modify any of the regexp that process the raw data file into the raw xml file. Those will have to be heavily modified before the next decade's worth of data can be processed.

Created a php version of the visualization. It now looks at the state of the year, sort and hide/show controls and can generate a URL reflecting those. The page can also parse the get array to recreate the state. Of the hide/show controls, only gender has been coded so far.

Had meeting with SD:
- restructure crime groupings
- repurpose respite to actual deferrals of decision rather than recommended pardons
- restructure outcome groupings

Updated the rng files for proofing and sql for 1790s and 1800s, but none of the other code (i.e. the regexps that generate the raw xml files, nor the numbered rng files used in preparation of the raw xml files).

Unable to get Marri's most recent data files to update them with appropriate new structural elements and to run a bunch of regexps to reflect the new groupings of crimes and outcomes. Will do that next week.

SD requested adding the following fields to the visualization:

mode of execution - that information is not currently captured, so we'll have to add a new field to the db then it's pretty straightforward to add the data and grab that for the visualization

condition of pardon - SD sent a list of 9, which is very similar to the existing setOfOutcomeGroups list. We'll likely revise that set and then have someone go through and update the values as needed in the records.

respites - SD sent a list which has "defer" in addition to the items on the sefOfNormalizedRespites, so we'll likely revise the set to include that new value and have someone go through and update the data as needed.

Started work on an object-based version of the data structure. I hope to have complex enough objects that the data structure containing them is simple, and that each trial object can modify the svg object associated with it. I've got a test set of 20 objects appearing by year in default colours. The main part of the work will be rewriting the interactive code so that when a user clicks on filter checkbox, the appropriate values are modified in the grid of trial objects and then each of those modifies the svg object associated.

Prior to that, tidied up a few minor errors in the all-array version of the site and sent email to SD updating him on improvements I've made and getting input from him on exactly what data fields and values to use in the remaining categories he wishes to sort and hide/show by.