Capital Trials at the Old Bailey

Met with SD and Maria (?sp) to get her started on editing the XML output from my series of regexp processing of the rtf raw data provided by SD. He'll give me list of judges to add to the judges field. Next week we'll set up times for her to come down and start. Primarily she'll be taking records with content in the Trial Notes field and manually placing that into appropriate structures (creating those structures if needbe) in that record.

Met with SD to confirm his categorization. He will manually change instances of "capital theft" and "theft" to "St in dwelling" or "Robbery in Dwelling" as needed. Any remaining instances of "capital theft" or "theft" will go into the MISCELLANEOUS category. Instances of "extortion" will go into the CURRENCY category

Arson : MISCELLANEOUS
Burglary : BURGLARY
Cattle : ANIMAL THEFT
Coining : CURRENCY
Extortion : CURRENCY
Forgery : CURRENCY
Horse : ANIMAL THEFT
Housebreak : BURGLARY
Impersonation : CURRENCY
Murder : MURDER
Pickpocket : MISCELLANEOUS
Rape : SEXUAL ASSAULT
Returning : MISCELLANEOUS
Riot : MISCELLANEOUS
Rob in Dwelling : ROBBERY
Robbery : ROBBERY
Sheep : ANIMAL THEFT
Shoplift : ST IN SHOP
Shooting : MISCELLANEOUS
Sodomy : SEXUAL ASSAULT
St in Dwelling : ST In DWELLING
St in Mail : MISCELLANEOUS
St on Thames : MISCELLANEOUS
[other values] : NOTKNOWN - to be manually edited by proofer

confirmed status of my work with SD. My crime categories were not correct, so he sent me the correct ones. I'll modify the regular expression code before using it again to reflect the new categorization. I also need to add an element to contain the leftover bits of text from the processing and use the trial_notes element only for the proofer to manually put bits of text into.

Over past couple of weeks, I've been doing further tests with three types of visualizations of the Bailey data:
1) Frequency (each trial gets a box, x-axis is years, y-axis is number of trials, user can colour or hide/show trials by a range of attributes)
2) Motion (trials aggregated by year, primary key is one attribute (e.g. crime), rest of fields are numeric counts or percentages (e.g. %male, %age_under_19)) then put into display box where trends seen by position and movement of circles representing each of primary attribute values.
3) Clumping (similar data as for motion, but instead of axes being %age_under_19 and %male, one axis would be age with points for under_20, 20to40, 40+ and the other axis would be gender with one point for male and one for female. The area of the circle at each intersection would correspond to the number of trials in that year with the appropriate values for the two axes.

#1 will be the primary visualization as it provides both immediate access to individual trials and indicator of trends over the set. #2 will be secondary visualization and may appeal to people whose learning style is more synthetic/right brain and less analytic/left brain. #3 doesn't get me anything I can't get from #2 and would be a huge amount of work to create the display engine.

Created a tab-delimited file of the records I've been using for my frequency visualization. Each of those has a unique id in the first field, so of course no trends could be presented by the software, but I could get a snapshot of the distribution of trials on the selected variables at each year.

Then decided to focus on crimes, so created a table with 1 of the 8 types of crimes in the first field of each record, then repeat for the ten years in the sample set. Then I had to aggregate the number of e.g. murders in 1790, and then work out the percentage of that number which had male for the suspect's gender, L for the jury code, age<20, age21-40, age40+ etc.

So, you can no longer get at each individual trial, but only at the group of trial which match the up to 4 attributes (2 axes, colour, size of ball) that you can filter by. That may be 1 or it may be more than 1 in a given year. On the other hand, you can now see trends e.g. for each crime, what the gender breakdown is for each year over time and how that's trending.

I'm not sure if the kind of aggregating required to get the trends will make the data useless or not yet.

Gapminder Motion (a google gadget based on Hans Rosling's Trendalyzer) requires the following in a tab delimited table: first record: a string for name of field first field: a string for primary key variable value (e.g. province if you're comparing stats across provinces) second field: a number for date (I used 4-digit years, not sure what else it supports) third and subsequent fields: string values will appear in the colour drop-down and the size drop-down but not in the two axis drop-downs; numeric values appear in all drop downs. The set of primary keys is repeated for each year (or other date value). That's how the software can show the trend over time for that primary key in the variables that have been chosen in the dropdowns. I'm going to see if there is any way to manipulate the Bailey crimes data into something that is amenable to the demands of this visualization tool, and if the numbers are useful having been so massaged.

Got the Google Gadget based on Hans Rosling's Trendalyzer. It's called Gapfinder Motion. There is absolutely no supporting documentation. It took me quite some time and a trip through Google's visualization api to find that the URL that the gadget needs requires "tq" as an argument rather than the "ccc" provided in the URL of the page itself. For example to address this URL successfully from the gadget: http://spreadsheets.google.com/ccc?key=pKaktXm-iVRpJG7BHoeNdEg&hl=en (for now this page is accessible only to me) I must enter this into the URL field in the gadget http://spreadsheets.google.com/tq?key=pKaktXm-iVRpJG7BHoeNdEg&hl=en Here's the URL for the page that allows you to specify the necessary arguments, and then generates the block of javascript code that you put into the page (I've added linebreaks to make the URL a little more manageable in the blog viewer): http://k5r309h2-a.gmodules.com/ig/creator?synd=open&; url=http%3A//www.google.com/ig/modules/motionchart.xml& pt=%26context%3Dd%26synd%3Dopen%26lang%3Den%26.lang%3Den%26country%3Dus%26.country%3Dus%26cat%3Dall%26num%3D1%26start%3D-1%26cols%3D1%26objs%3D6Ya8& sn=6Ya8&lang=en

I need to develop an implementation plan based around his funding schedule, with proposals for how much of what kind of labour is needed (data prep, database design, client-server interaction, user interface, visualization interface), costs and likely sources for that labour.

In advance of the meeting with Simon I went through the files and got Martin to send me the XSLT which takes my xml input and generates ids for the recorder's reports and the trials. (Bailey_add_id.xsl and Bailey_ids_added.txt) Those ids are hexadecimal and there doesn't seem to be a way to turn those into decimal integers in xpath, so the output file will need to be processed.

Met with SD to confirm details of SSHRC grant and have initial discussion.

SD has $87,000 over 3 years (28k, 28k, 31k) to fund research (including travel for him to England) of the data, preparation of the data and creation of a website containing a searchable database and a graphical representation of trials in the dataset.

State of data:
1689 - 1730 limited or no data at present (end of year 3)
1730 - 1810 data needs minor correction and normalization (end of year 1)
1810 - 1837 data needs major work (end of year 2)
"respited larcenies" will likely be included, Simon to consider and decide

Simon will want to add a couple of fields to accommodate links to external sources (images of the original documents etc.)