HCMC Blogs

Five of the engravings contain unusual glyphs used for abbreviation, and now the same has appeared in one of the regular texts, so I've documented the markup process for these glyphs, and passed the document on to LSPW so she can merge it into her growing markup documentation.

Vol 19 is to be published as a book, and has to be completely reformatted to produce the camera-ready copy for the publisher. These are the basic requirements:

Left and Right margins = 2”
Top margin = 1.5”
Bottom margin = 2.25”
Footer = 2”  [Page number centered at the bottom]
Line spacing = 16 pt. exactly
Space after = 6 pt.
Font size= Times New Roman 12 pt.
Footnotes= Times New Roman 10 pt.
Text alignment = Justified, including footnotes, endnotes, bibliography

I've made considerable progress with this:

• Made fresh copies of the XML source, XSLT transformation, and other ancillary files for this particular rendering, because all of them may need to be edited to fit the new layout.
• Changed all units from cm to in, because the publisher is US and uses imperial.
• Changed all page sizes, margins, borders, spaces etc. to conform with the publisher's requirements.
• Changed most font sizes to accommodate the shift from a para-level font size of 10pt to 12pt.
• Changed almost all Gentium fonts to Times (keeping Gentium for e.g. Greek and IPA).

The resulting volume is over 800 pages, but there are still lots of problems:

• The normal page content area seems too small (or perhaps widow/orphan settings are causing whole paragraphs to be shifted to the next page wrongly).
• Either the running titles are too high up, or the space between the running title and the head of the page is too big.
• The page numbers are still showing at the top of the page; I need to move them to the footer.
• Once pages are all stable, all the images will need to be re-positioned for optimal text flow.

According to the HTML5 spec, this is valid code:

<video controls="controls" width="320" height="256" id="gbua1"> 
                            <source src="http://lettuce.tapor.uvic.ca/~taprlans/media/gbua1.m4v" type="video/mp4; codecs="avc1, mp4a""></source> 
                            <source src="http://lettuce.tapor.uvic.ca/~taprlans/media/gbua1.ogv" type="video/ogg; codecs="theora, vorbis""></source> 
                            <p> 
                                <a href="http://lettuce.tapor.uvic.ca/~taprlans/media/gbua1.m4v">Download the video</a>.</p> 
                        </video>

According to the W3 validator, however, it's invalid. Removing one of the source elements gets rid of the validation error. This doesn't affect the performance of the video. So, it's just a case of waiting for the validator to catch up to the standard.

My work on this site can be put into three different categories:

• Adding the 1911 table and views to the database, and modifying the views for the other census years to take into account any new fields introduced in 1911
• Adding the new 1911 census data
• Modifying the 'loader' application to be more robust, and to accept the CSV data files
• Making mostly minor changes to the actual website, such as textual changes and some additions to the advanced census search

My Workflow

I did all work, including importing data, on my local machine first, on which I had a complete development environment. After importing the data and testing all changes, I then updated the development server, using phppgadmin.uvic.ca for the database changes and Subversion for the PHP changes.

The Database

In addition to making the census_1911 database, I also had to make new views for the 1911 census, since the website only interacts with the views for searching and displaying, rather than the tables. I also had to modify the older views and put in some new fields (which are NULL on those older views). The search functions on the site use a UNION query to join all of the views, so they all need to have the same columns for the search to work properly, even if some of those columns are NULL in the various views. This process is all fully documented in David's excellent manual. I didn't change the methods or the way anything works; I just added the new data.

The Data

Each sub-district is in its own CSV (and XLS – Patrick usually sent me the data in both formats). Thus, each sub-district was imported separately. This is documented in full on the blog, but a short summary of the steps:

1. Add a new row to the location table, which becomes the location_id of the rows to be imported
2. Make a new 'mapping' file in the map directory in the loader, following the conventions of the other mapping files
3. Add any new rows to the auxiliary tables as necessary (occupations, nationalities, etc.) - Patrick supplied these when sending the sub-districts
4. Import the data with the loader, check for errors, delete and re-import as necessary
5. Create a dump file and then import into the development database on tapor

The Loader

Although it had cursory support for CSV files, the loader application was really only built to handle the old Access database format, and then only on a Windows machine. So, I modified it to accept CSV files. I also beefed up the available mapping functions, and changed some of the queries to be more organized. The application isn't 100% complete, but does the job.

The loader.php script itself was a mammoth, so I created a class Loader in inc/loader.php to do some of the heavy lifting and abstract some of the functionality. David had written a short manual for the loader which explains how the mapper works. I filled out the manual with the new mapping functions that I wrote. The basic steps for importing data are:

1. On the 'Configure' page, putting in the full path to the CSV data file in the field 'CSV File (for CSV imports) - absolute path'
2. Choosing census_1911 for the table name on the main page and ensuring that “empty table before import” is not checked
3. For the 'field map' file , choosing the map file made for the sub-district
4. Cross fingers and import! There are often incorrect foreign keys which cause SQL errors, so it wasn't uncommon for me to delete the new data a few times and re-import to account for new foreign keys

The loader tends to time out and/or run out of memory when processing larger CSVs. If this happens, then it's fine just to re-import because I extended the loader with the option to skip previously entered records, which can be defined in the mapping files (and is documented in the loader doc file).

The loader does not live on the development website; all loading was done on my machine (parsnip).

The Website

Almost all of the changes I made to the website itself (i.e. the .inc pages) were textual changes given to me by PD. I did have to modify some of the search functionality, most significantly re-organizing the boxes and adding some new fields for the 1911 data. These changes were relatively minor and are documented in the blog.

[SA 120116 made changes affecting what's talked about in this post, see http://hcmc.uvic.ca/blogs/index.php?blog=36&p=9010&more=1&c=1&tb=1&pb=1]

Met with SD yesterday primarily to discuss two things: how to link together criminals from different trials that are the same person, and how to link trials in different trial_files to each other (i.e. for a bunch of criminals tried for the same crime). We determined that the simplest, quickest, and therefore most prudent way to do this is to use an "id" attribute to link together the elements as required. So, if two <criminal> elements both describe the same person, then each <criminal> element in question should look like this:

<criminal id="myuniqueidentifier">
(stuff)
</criminal>

where "myuniqueidentifir" can be any string as long as it's the same for each criminal that should be treated as the same person. <criminal> elements that don't correspond to any other <criminal> element do <strong>not</strong> need an id attribute.

Similarly, for <trial> elements that refer to the same trial (but a different criminal):

<trial rel="5">
(stuff)
</trial>

In the case of trials, 'rel' does not have to be absolutely unique, just unique within one court session (i.e the same trial_file_start_date and trial_file_end_date values). SD requested this to make it easier to keep track of the numbers.

I will be able to alter the import script to account for these new attributes, but I'm not sure if SD will have a chance to go over all of the data before my contract is up.

Long lunch on Wednesday for JN and SA.

Tracked down a bunch of bugs, which EGB fixed, and the output now validates, after coming through our two-stage transform. Yay! From 406 errors to zero in a few days.

Met with RS and EGW about the Professional Writing site, which we'll move into Cascade. We're meeting to do nav planning next week.

I've written an intermediary stylesheet to split out <list> elements containing <pb> and <fw> children into separate lists with the <pb>s and <fw>s outside them. This is processed prior to the main transformation, and has solved the problem for lists. I may revisit it for lg/l at some point. For the record, here's the template (the only non-pass-through template in an identity transform):

<!-- Split out interrupted lists into a sequence of lists with the interruptions

      between them. -->

  <xsl:template match="list[child::fw or child::pb]">

<!-- Stash a reference to the current list so we can easily refer to it in later nested code. -->

    <xsl:variable name="thisList" select="."/>

    <xsl:for-each-group select="child::item" group-adjacent="count(preceding-sibling::fw) + count(preceding-sibling::pb)">

        <list>

          <xsl:for-each select="$thisList/@*">

            <xsl:copy/>

          </xsl:for-each><xsl:text>

</xsl:text>

          <xsl:for-each select="current-group()">

            <xsl:copy-of select="."/><xsl:text>

</xsl:text>

          </xsl:for-each>

        </list><xsl:text>

</xsl:text>

      <xsl:variable name="lastItem" select="current-group()[position() = last()]"/>

          <xsl:for-each select="$lastItem/following-sibling::fw[preceding-sibling::item[1] = $lastItem] | $lastItem/following-sibling::pb[preceding-sibling::item[1] = $lastItem]">

            <xsl:copy-of select="."/><xsl:text>

</xsl:text>

          </xsl:for-each><xsl:text>

</xsl:text>

      </xsl:for-each-group>     

      

    

  </xsl:template>

For a little while now, the Stylesheets task on teijenkins.hcmc.uvic.ca has been showing as unstable, because of this warning:

Warning: XML resolver not found; external catalogs will be ignored

SPQR found the solution: my /var/lib/jenkins/hudson-log-parse-rules file was out of date. The most recent version in SVN has a line suppressing that warning. This seems to be the fix:

cd /var/lib/jenkins
sudo svn export https://tei.svn.sourceforge.net/svnroot/tei/trunk/P5/Utilities/hudson-log-parse-rules
sudo chown jenkins hudson-log-parse-rules

Running a build right now to see if it's fixed.