Map Of London

I've been looking at how we might do variant spelling searches on the fly, using our plethora of name attributes, and this is as far as I've got. The code below searches for the search term in name elements, and then for each name element where it finds a hit, it gets the @target attribute. It then gets all the text from all the name elements with the same @target attributes as the hits, tokenizes them, and returns a list of the distinct values (and, in the code below, the hits themselves):

xquery version "1.0";

declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace tei="http://www.tei-c.org/ns/1.0";
declare namespace ft="http://exist-db.org/xquery/lucene";
import module namespace kwic="http://exist-db.org/xquery/kwic";
declare namespace util="http://exist-db.org/xquery/util";
declare namespace exist="http://exist.sourceforge.net/NS/exist";
declare namespace xh="http://www.w3.org/1999/xhtml";


let $hits := //ref[ft:query(., 'abchurch')],
$targs := distinct-values($hits//@target),
$targString := concat(string-join($targs, ','), ','),
$refs := //ref[@target][contains($targString, concat(@target, ','))],
$strings := (for $r in $refs return tokenize($r/text(), '[\s]+')),
$dist := distinct-values($strings)
return ($dist, $refs)

This might be further refined by using a fuzzy-match between the search term and each of the distinct values, so that you could reject any that are completely different (e.g. "Mary"), and end up with a list of those which are similar (such as "Vpchurch").

There don't seem to be any really good examples of how to search a document collection and display kwic summaries of hits grouped by document, so it took me a little puzzling to figure this one out. Most of the examples show processing only of the first hit in a document. Here's my sample code:

xquery version "1.0";

declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace tei="http://www.tei-c.org/ns/1.0";
declare namespace ft="http://exist-db.org/xquery/lucene";
import module namespace kwic="http://exist-db.org/xquery/kwic";
declare namespace util="http://exist-db.org/xquery/util";
declare namespace exist="http://exist.sourceforge.net/NS/exist";
declare namespace xh="http://www.w3.org/1999/xhtml";

for $hit in //TEI/text[ft:query(., 'henslowe')]
let $expanded := kwic:expand($hit),
$config := <config xmlns="" width="40"/>,
$t := $hit/ancestor::TEI
order by ft:score($hit) descending
return
<xh:div>
<xh:h3><xh:a href="{xs:string($t/@xml:id)}.htm">{$t/teiHeader/fileDesc/titleStmt/title[1]/text()}</xh:a></xh:h3>
{
for $h in $expanded//exist:match
return
kwic:get-summary($expanded, $h, $config)
}
</xh:div>

I've done an analysis of the search engine on the current site, with a view to identifying aspects of it that should be maintained, and possible missing features or limitations that we might be able to overcome.

Key features:

• Long s substitution. The search engine does a simultaneous search for the search term itself, and a version of it in which s has been replaced by ſ. However, see limitations below.
• Wild-card searches using * and ?. These are provided by the original eXist search engine.
• Special handling for variants in name elements. A cron job that runs every day analyzes all the name elements in the system, and creates a separate entry for every word token, linking it to the @xml:id of the target element, as well every distinct complete content element. This index is then searched, and all other tokens which are in name elements pointing to the same @xml:id are added to the search list. This is quite effective.

Limitations:

• Long s substitution is crude and does not create the variations that are likely to occur. For instance, if you search for "sinfulness", the search will also look for "ſinfulneſſ". However, such a form is unlikely to occur anywhere, since long s is not typically used at the end of a word; more likely is "ſinfulneſs". A more realistic substitution algorithm can be created, which avoids replacing final s with long s, although that still may not allow for all the possible variants that may occur.
• The nightly cron job for building the variant spelling index is a bit fragile, and also means that changes to the db are not reflected in the variant spelling index for several hours. It might be easier to build Lucene indexes that will enable us to do the same sort of search on-the-fly.
• No keyword-in-context results are returned; all that comes back is the title of the document as a link. In the case of the long documents which are split into sections for display, clicking on this only shows you a TOC or an introduction, which may not be the part of the document in which the hit appears. This problem would be obviated by my plan to stop breaking these documents up, but JJ hasn't yet responded to that.

I've completed all of the major accessible site pages. In some cases (such as the topic listing and the library listing) the pages are generated by parsing the content; in others, they're converted from the hard-coded PHP files.

I'm still waiting for JJ's OK to go ahead and refactor some of the long documents in the topics subcollection, which are a bit of a mess. I've also discovered one new annoyance: the use of <name type="book_title"> instead of <title level="m">. There are 474 instances of this. I can probably fix them using regex search-and-replace.

There are several other PHP files which are not linked from the menus or other pages on the site; these I'll convert when I have the chance, and place in /data/info, but I will check with JJ to see if any of them are obsolete first.

I've implemented a system whereby captions and other boilerplate text content is abstracted from the XHTML and/or TEI code and stored in /db/data/boilerplate/boilerplate.xml. This is how it works:

• /db/data/boilerplate/boilerplate.xml contains individual <div>s, <seg>s etc. which are identified by @xml:id and contain the text snippets. The bulk of the items, which are very short, are stored in <seg> elements which look like this:

  <seg xml:id="sshrcFundedCaption">This research was supported by the Social Sciences and Humanities Research Council.</seg>
  

• The includes.xql module retrieves these, and makes them available in two forms: one as a sequence of items that can be accessed from XQuery (where XQuery itself is constructing the output and requires the captions), and one in the form of <param> elements which can be passed on to XSLT transformations (where XSLT will require the snippets to use when constructing XHTML output). This is the code from includes.xql:

  declare variable $inc:captions := doc('/db/data/boilerplate/boilerplate.xml')//tei:div[@xml:id='captions'];
     
  declare variable $inc:captionParams := 
  for $inc:c in $inc:captions//tei:seg
  return <param name="{$inc:c/@xml:id}" value="{$inc:c/text()}"/>;
  

• XQuery makes use of these items like this:

  <p>{$inc:captions//tei:seg[@xml:id='sshrcFundedCaption']/text()}</p>
  

• The main XSLT library, general.xsl, includes a module called captions.xsl, which defines parameters (along with default values); when the XQuery engine calls the transformer, it passes values for these parameters in. For instance:

  <xsl:param name="sshrcFundedCaption">This research was supported by the Social Sciences and Humanities Research Council.</xsl:param>
  

  The XSLT can then use the params like this:

  <xsl:value-of select="$mapCaption"/>
  

There are two significant advantages to this, which make it worth the work. First, the boilerplate.xml file lives in the /db/data collection, which makes it editable by the data editors, so e.g. JJ can modify captions etc. if necessary. Second, if the site ever needed to be translated into French or another language, the text components are already centralized, so that would be easier.

Made a lot of progress today. The sub-index pages are all working, and the literary references pages now display correctly. Pages for people now include not only contributions from contemporary people, but mentions of historical, fictional and mythical people. Many problems with underlying XML have been fixed, and neigbourhoods and wards are now all correctly identified; names of wards have been changed on JJ's instructions to include "Ward" at the end, to distinguish them from sites or streets with the same name.

This is the list of items in addition to wards which lost their types in the conversion. I'm fixing them manually now. Wards have been done already. Some seem to lack types at all; those I'll investigate.

CAMO2   street
CURT2   site
CUST1   site
HART4   street
HOLY4   
STLE2   church
STMA20   church
STMA40   
STMA43   church
STNI4   church
SUGA1   street
SUGA2   street
THEA2   site
TOTT2   site
TOWE6   site
TRIG2   site
WALL2   site
WATE6   site
WATE7   site
WRES1   site

I've been working on reproducing the Index page and its subpages -- it's basically an index of indexes, and the subpages are sometimes hard-coded and sometimes generated automatically from the db. I've set up individual pages in /data/info for each of the subpages, which allows them to have their own introductory text, but the rendering of all, including the main index page, is handled by index_site.xql, which gets the content from the info page and merges it with data from the db where appropriate.

I've hit one stumbling-block of my own making. There were a lot of entries in the INDEX1.xml file which were defined as @type="ward" (the list is below). When these were converted over to new files in the locations subcollection, they were not provided with <place> elements because they had no location data in them (no <section> or <coords>). That means they don't have their @type attribute. I'll have to add this info manually to the 26 records, and I should also check for other records which have lost a type attribute. There are no other type attributes which are completely missing, but there are 18 street, site and church items items which didn't have proper locations, so lost their types.

ALDE2
ALDG2
BASI1
BILL2
BISH1
BREA3
BRID3
BRID4
BROA3
CAND2
CAST2
CHEA1
COLE2
CORD1
CORN1
CRIP2
DOWN1
FARR1
FARR2
LANG1
LIME1
PORT1
QUEE3
TOWE4
VINT2
WALB2

Just an additional note to my previous post: I've been switching the somewhat random and varied @rend values over to pure CSS in @rend attributes, although there are some which can't easily be switched until I know exactly what they mean. However, most @rend values are CSS, and that will be the policy in future. It gives the encoder much more control, and can more easily be validated when the encoder has to introduce new styles to the markup.

I've continued working on trying to fix up the XML in the existing documents while in the UK. The more I looked at them, the worse they turned out to be; it seems unlikely that any of them would have validated, unless they were using a heavily-modified schema which redefined the way some core tags should be used. I've pursued a strategy of fixing one-off or two-off errors manually whenever I encounter them, and turning the other fixes into a single large XSLT transformation, which I include below because it amounts to documentation of the fixes required. The result of this transformation, which I think is finished as of this morning, is that all documents validate against tei_all, with the exception of two, which have peculiarities relating to mapping the routes of pageants; those will have to be entirely rewritten manually; and the credits.xml file, which uses XInclude to pull in data from another document, so can't actually validate because oXygen can't handle the xpointer syntax to pull in the relevant content (although eXist can). Following this, I'm going to convert all of the remaining PHP pages on the site to XML (that has to be done manually), and then we'll have a good data set and I can start coding things like the search engine.

That said, there are many documents which still have problems, despite being valid, and those will have to be fixed manually. Most commonly, there are documents which have internal sections, which should be handled by dividing them into nested divs, each with a head element; however, they've been done with special paragraph or seg elements for inline headings, which validates but is clearly incorrect. Those can be dealt with in time, by an RA, once we have good detailed markup documentation.

Here's the XSLT:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:xd="http://www.oxygenxml.com/ns/doc/xsl"
    xpath-default-namespace="http://www.tei-c.org/ns/1.0"
    xmlns="http://www.tei-c.org/ns/1.0"
    exclude-result-prefixes="xs xd"
    version="2.0">
    <xd:doc scope="stylesheet">
        <xd:desc>
            <xd:p><xd:b>Created on:</xd:b> Oct 1, 2011</xd:p>
            <xd:p><xd:b>Author:</xd:b> mholmes</xd:p>
            <xd:p>This is just a utility identity transform for any purposes required.</xd:p>
        </xd:desc>
    </xd:doc>
    
<!-- This template finds instances where there is a group element
     with only one child text element, and removes the group and 
     the child text. -->
    
    <xsl:template match="group[count(child::text) lt 2]">
        <xsl:apply-templates select="child::text/*"/>
    </xsl:template>
    
<!-- body elements should have their content in a div, but usually they don't.   -->
    <xsl:template match="body[not(child::div)]">
        <body><xsl:text>
        </xsl:text>
            <div><xsl:text>
        </xsl:text>
                <xsl:if test="not(child::head)">
                    <head><xsl:copy-of select="//teiHeader/fileDesc/titleStmt/title[1]/text()"/></head><xsl:text>
        </xsl:text>
                </xsl:if>
                <xsl:apply-templates select="*"/><xsl:text>
        </xsl:text>
            </div><xsl:text>
        </xsl:text>
        </body><xsl:text>
        </xsl:text>
    </xsl:template>
    
<!-- This template adds a generic revision change element, to capture the range
of different changes I've made over the past couple of weeks. -->
    <xsl:template match="revisionDesc">
        <revisionDesc><xsl:text>
            </xsl:text>
        <xsl:apply-templates select="change"/>
            <change when="2011-11" who="mol:HOLM1">Various updates and fixes made through XSLT, to standardize and normalize encoding practices.</change><xsl:text>
            </xsl:text>
        </revisionDesc>
    </xsl:template>
    
<!-- At the same time, we might as well remove useless <change> elements
     which are empty, and also wrong, derived from a template. -->
    <xsl:template match="change[string-length(normalize-space(.)) lt 2]"></xsl:template>
    
<!-- Lots of <change> elements contain <l> and <p> elements, which are invalid. These are used to
     separate multiple changes made during the same session. The simplest thing is 
     to replace them with line-break tags, I think. Many have no content,
     so those can be suppressed. -->
    <xsl:template match="change/l">
        <xsl:if test="string-length(normalize-space(.)) ge 2">
            <xsl:value-of select="."/>
            <xsl:if test="following-sibling::l[string-length(normalize-space(.)) ge 2]">
                <lb/>
            </xsl:if>
        </xsl:if>
    </xsl:template>
    <xsl:template match="change/p">
        <xsl:if test="string-length(normalize-space(.)) ge 2">
            <xsl:value-of select="."/>
            <xsl:if test="following-sibling::l[string-length(normalize-space(.)) ge 2]">
                <lb/>
            </xsl:if>
        </xsl:if>
    </xsl:template>
    
    
<!-- Normalize previously-inserted change elements.   -->
    <xsl:template match="change[contains(@who, 'staff.php#martin')]">
        <change who="mol:HOLM1">
            <xsl:copy-of select="@when"/>
            <xsl:apply-templates select="* | text()"/>
        </change>
    </xsl:template>
    <xsl:template match="change[child::name[@ref='mol:MDH']]">
        <change who="mol:HOLM1" when="2011-09">
            <xsl:value-of select="text()"/>
        </change>
    </xsl:template>
    
    
<!-- Some files have no revisionDesc, so we'll insert it.   -->
    <xsl:template match="encodingDesc[not(following-sibling::revisionDesc)]">
        <encodingDesc>
            <xsl:apply-templates></xsl:apply-templates>
        </encodingDesc>
        <revisionDesc><xsl:text>
            </xsl:text>
            <change when="2011-11" who="mol:HOLM1">Various updates and fixes made through XSLT, to standardize and normalize encoding practices.</change><xsl:text>
        </xsl:text>
        </revisionDesc>
    </xsl:template>
    
<!-- We need to handle previously-transformed internal links like this:
<ref type="internal" target="/licenced.htm"> -->
    <xsl:template match="ref[@type='internal'][matches(@target, '/[a-zA-Z0-9_]*.htm')]">
        <ref target="mol:{substring-before(substring-after(@target, '/'), '.htm')}"><xsl:value-of select="."/></ref>
    </xsl:template>
    
    
<!-- @rend attributes are very problematic. Here are the elements which carry them: 

emph (bold)
graphic (center, imgmap) Don't know what to do about this.
group (multi ... WTF?)
head (italics)
hi (sup)
l (various: see below)
lg (center)
name (book_title, no_index. The former fixed below; the latter mysterious, and left alone.)
p (center; center italics; center, italics; copyright; indent)
ref (noindex, no_index)
reg (fixed manually)
seg (see below)
title (fixed manually)

-->
    
    
    
<!--  These are the distinct values of seg/@rend. They include both inline and block formatting 
      commands. 
      
      bold
      italics
      superscript
      bold italics
      center
      heading center
      heading
      center italics
      italics center
      italics indent_10
      
      Each will have to be handled slightly differently, depending on its context and usage.
      
-->
<!-- There's only one of these, and it's just flat-out wrong.   -->
    <xsl:template match="name[@rend='book_title']">
        <title level="m"><xsl:apply-templates/></title>
    </xsl:template>
    
<!-- There are two distinct usages of ref/@rend regarding "no indexing": 
no_index and noindex. We'll normalize to the former. -->
    <xsl:template match="ref[contains(@rend, 'index')]">
        <ref rend="no_index">
            <xsl:copy-of select="@*[not(local-name() = 'rend')]"></xsl:copy-of>
            <xsl:copy-of select="* | text()"/>
        </ref>
    </xsl:template>
    
<!-- These are the simple ones, which are all inline. <seg rend="bold">, for instance, occurs in <p>, <q>, <l>, and <item>; in all those contexts it can be replaced by <hi>.   -->
    
    <xsl:template match="seg[@rend='bold']">
        <hi rend="font-weight: bold;">
            <xsl:apply-templates/>
        </hi>
    </xsl:template>
    
<!-- <emph> is only used for bold, and should be kept as <emph>.   -->
    <xsl:template match="emph[@rend='bold']">
        <emph rend="font-weight: bold;">
            <xsl:apply-templates/>
        </emph>
    </xsl:template>
    
    <xsl:template match="seg[@rend='italics']">
        <xsl:choose>
            <xsl:when test="contains(., 'MoEML') or contains(., 'Map of Early Modern London')">
                <title level="m"><xsl:apply-templates select="* | text()"/></title>
            </xsl:when>
            <xsl:otherwise>
                <hi rend="font-style: italics;">
            <xsl:apply-templates/>
                </hi>
            </xsl:otherwise>
        </xsl:choose>
        
    </xsl:template>
    
    <xsl:template match="seg[@rend='bold italics']">
        <hi rend="font-weight: bold; font-style: italics;">
            <xsl:apply-templates/>
        </hi>
    </xsl:template>
    
    
    <xsl:template match="seg[@rend='superscript'] | hi[@rend='sup']">
        <hi rend="vertical-align: super;">
            <xsl:apply-templates/>
        </hi>
    </xsl:template>
    
    <!-- These are the more complicated ones. They require that the style be realized on the containing element, not the <seg>. Many are caused by abuse of <lg> and <l> to create a heading. In the right
        context, these can simply be converted to <head> elements. -->
    <xsl:template match="lg[preceding-sibling::*[1][local-name() = 'head']][count(//l) = 1]">
        <head>
            <xsl:attribute name="rend"> text-align: center; <xsl:if test="contains(l/@rend, 'italics')">text-style: italic;</xsl:if>
            </xsl:attribute>
            <xsl:apply-templates select="l/* | l/text()"/>
        </head>
    </xsl:template>
    
    
<!--  Normal <lg>s and <l>s. There's a lot of redundancy in these (rend="center" on both 
<lg> and its child <l>) Only "center" exists.-->
    <xsl:template match="lg[@rend]">
        <lg rend="text-align: center;">
            <xsl:apply-templates/>
        </lg>
    </xsl:template>
    
<!-- <l> has a wide range of values for @rend. 
     We need to handle each of them correctly.
-->
    <xsl:template match="l[@rend]">
        <xsl:variable name="style">
            <xsl:if test="contains(@rend, 'center')">text-align: center;</xsl:if>
            <xsl:if test="contains(@rend, 'bold')">font-weight: bold;</xsl:if>
            <xsl:if test="contains(@rend, 'italic')">font-style: italic;</xsl:if>
            <xsl:if test="contains(@rend, 'align_right')">text-align: right;</xsl:if>
<!-- Indents are of the form indent_3, where 3 = number of spaces to indent by. We will use
     em as the value for this. In all cases, the indent value appears at the end of the 
     @rend. -->
            <xsl:if test="contains(@rend, 'indent_')">text-indent: <xsl:value-of select=" normalize-space(substring-after(@rend, 'indent_'))"/>em;</xsl:if>
        </xsl:variable>
        <l>
            <xsl:attribute name="rend"><xsl:value-of select="normalize-space($style)"/></xsl:attribute>
            <xsl:apply-templates select="* | text()"/>
        </l>
    </xsl:template>
    
<!-- <p> elements have some values which can be converted to css, but others 
that we'll just leave in the rend attribute. -->
    <xsl:template match="p[contains(@rend, 'italics') or contains(@rend, 'center')]">
        <xsl:variable name="style">
            <xsl:if test="contains(@rend, 'center')">text-align: center;</xsl:if>
            <xsl:if test="contains(@rend, 'italic')">font-style: italic;</xsl:if>
        </xsl:variable>
        <p rend="{normalize-space($style)}">
            <xsl:apply-templates/>
        </p>
    </xsl:template>
    
<!-- There are several instances of <p> elements which contain only a seg with @rend 
     = "heading" or "heading center". These can't be turned into <head> elements 
     because they appear right in the middle of divs. They SHOULD be handled by 
     sub-divs, but that's impossible to do algorithmically. -->
    
    <xsl:template match="p[child::seg[contains(@rend, 'heading')]]">
        <xsl:comment>The following p should NOT be a p tag; it should be 
                     a heading, using a head tag, and the text should be subdivided
                     to provide a nested div at this point.
        </xsl:comment>
        <p rend="heading">
            <xsl:if test="contains(seg/@rend, 'center')">
                <xsl:attribute name="rend">text-align: center;</xsl:attribute>
            </xsl:if>
            <xsl:apply-templates select="seg/* | seg/text()"/>
        </p>
    </xsl:template>
    
    <!-- old internal references
    <ref type="internal" target="render_page.php?id=HENS2&n=16&searchterm=manuscript"> -->
    
    <xsl:template match="ref[@type='internal'][starts-with(@target, 'render_page.php?id=')]">
        <xsl:choose>
            <xsl:when test="contains(@target, '&')">
<!-- There are other parameters which we need to retain for the moment. -->
                <ref>
                    <xsl:copy-of select="@*[not(local-name() = 'target')]"/>
                    <xsl:attribute name="target">mol:<xsl:value-of select="substring-after(@target, 'id=')"/>&<xsl:value-of select="substring-after(@target, '&')"/></xsl:attribute>
                </ref>
            </xsl:when>
            <xsl:otherwise>
                <ref>
                    <xsl:copy-of select="@*[not(local-name() = 'target')]"/>
                    <xsl:attribute name="target">mol:<xsl:value-of select="substring-after(@target, 'id=')"/></xsl:attribute>
                </ref>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:template>
    
    
    
    <!-- <date calendar="Julian" value="1613">   -->
    <xsl:template match="date[@value]">
        <date when="{@value}">
            <xsl:apply-templates select="@*[not(local-name() = 'value')] | * | text()"/>
        </date>
    </xsl:template>
    
    
    
<!--  
    <text>s in multiple groups often have this: <text corresp="THRE3">.
    They point at e.g. locations, and should look like this:
    <text corresp="mol:THRE3">
-->
    <xsl:template match="text[@corresp]">
        <text>
            <xsl:copy-of select="@n"/>
            <xsl:attribute name="corresp" select="concat('mol:', @corresp)"/>
            <xsl:apply-templates/>
        </text>
    </xsl:template>
    
    <!-- Copy everything else as-is. -->
    <xsl:template match="node()|@*" priority="-1">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*"/>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>