Map Of London

I have my simplification process almost complete, with all links being handled quite straightforwardly without reference to their type. The XSLT still needs to be cleaned up a bit, because it was written to handle the old stuff, but it's not a big job. I'm also continually refining the XSLT to handle the wide range of different document structures which have proliferated over the years; all of these will have to be fixed in the end, but in the meantime I have to allow for them. I think I'll write an identity transform to fix the most egregious structural errors (such as <group> elements with only a single <text> in them, which are common).

I've also eliminated the <milestone/> tags I was using to generate hr tags in the output, and also the <hr/> tags; I'm achieving the same effect with borders on <h2> and <h3>.

Leaving for the UK tomorrow, but taking the project with me so I can keep working on it.

The issue of linking inside and outside the site is something that really needs to be dealt with. Right now, there's a proliferation of different types of link, specified by the @type attribute on the <ref> element; this is the range of instances currently in the XML files (some obviously typos, and some errors):

internal
bibl
detail
external
'detail
pers
name
CORN1
library
special
topics
detal
SIRT1
URI

I propose the following:

1. The @type attribute should just be dropped, and deleted from existing <ref> elements.
2. There should be two types of reference, distinguished by what their @target attribute looks like:
   • <ref target="mol:ADAM4">
   • <ref target="http://some.url/">
   The system should be able to distinguish between them by itself.
3. When the output (XSLT) system sees @target starting with mol:, it knows it has an internal reference. In that case, it constructs a simple link consisting of the relevant @xml:id (which is substring-after(@ref, 'mol:')) followed by .htm.
4. The controller.xql and index.xql files are able to figure out where the relevant @xml:id is, and process it accordingly. So, for instance, if it finds a TEI file in /db/data/ which has that @xml:id attribute on its root element, it processes that; if it finds a person element in PERS1.xml which has that @xml:id, it processes accordingly.
5. When the output (XSLT) system sees @target which does not start with mol:, it assumes an external link, and creates an anchor accordingly.
6. In the case of name tags, which also give rise to links to people, I have already converted all @key="ADAM4" instances to @ref="mol:ADAM4", and links are being created accordingly.

This means that all pages are being served out of the root of the site, and as long as all @xml:ids are unique, everything is accessible in that way. There may still be a couple of details to work out relating to @type on <name> (which is possibly worth keeping, although I don't think it's being used for anything useful at the moment), and the various uses of <bibl> (which I think in some cases ought to be <ref>s).

Wrote a simple table sort library in JavaScript, that does something similar to what the Adaptive DB table sort does, but which is much slimmer and faster because it simply decides whether a column is numeric or textual, and because (unlike the Adaptive DB tables) rows are sorted individually rather than in pairs. This should be a plug-and-play utility -- add the JS file to the header, and add the class "sortable" to any table, and it does the rest. Right now it's working on the list of ids and the contributor list, and I'll probably use it on the "index of the site" page, which has lots of little subsets of items.

Today I hit a major problem with WebDav: when I tried to get good working copies of my XML files out of the db, by mounting it as a share through WebDav, most of the XML documents came out with dozens of trailing #0 characters, which of course are very hard to deal with; they make the file invalid, but you can't easily search-and-replace them. I had to unmount and remount several times, and copy-paste the files in small batches, to get a good copy of my data. It took me ages to clean it up and confirm it was clean. I need to edit externally because of the XInclude issue mentioned in my last post, so I'm now going to abandon the data explorer approach, and work in a more conventional way:

• All data in SVN (not set up yet).
• Checkout / edit / commit.
• Upload to db when ready.

I'll make sure the SVN permissions are set so that the site data can't be changed by the data editors.

Processing of people from PERS1.xml is now working, and the Credits page uses this XInclude to include the contemporary ones to whom credit is due:

<xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="/db/data/PERS1.xml" xpointer="xpointer(//person[persName[@type='cont']])">
  <xi:fallback><p>Unable to find people in the PERS1.xml file.</p></xi:fallback>
</xi:include>

Use of XInclude precludes editing "live" inside the db via the oXygen data explorer, because the XIncludes are expanded on serialization, so you end up with a static page with the included data whenever you open the file. However, there are lots of other reasons not to use the data explorer (see my next post).

I've also basically finished the id listing page, although I have to add some JavaScript to make the table sortable. It gives a lot more info than the previous one, and links to the relevant data.

This document has proved quite complicated to turn into XML, because it contains a bibliography which should be drawn from items in the main bibliography, but several of the items are missing or slightly different. I've had to develop the following conventions for referencing and linking:

• Normally, when a document is processed, a <ref type="bibl"> will result in the importation of a biblio item from the main bibliography file into a special <div type="biblReferences"> at the end of the TEI file; that will then be processed into a full bibliography list. However:
• When a structure like this appears:

                    <listBibl>
                      <bibl type="replace"><ref type="bibl" target="OGIL1"/></bibl>
                      <bibl type="replace"><ref type="bibl" target="ATOZ2"/></bibl>
                      <bibl type="replace"><ref type="bibl" target="HORW1"/></bibl>
                      <bibl type="replace"><ref type="bibl" target="BACO1"/></bibl>
                      <bibl type="replace"><ref type="bibl" target="SAUN1"/></bibl>
                      
                    </listBibl>
  

  the XSLT assumes that we're constructing a bibliography in-place from items in the main bibliography, so it does not produce a list at the end of the document; instead it replaces each of the above <bibl> elements with the one referred to, and creates a bibliography in place.
• When an @xml:id attribute is placed on one of many key elements (<div> etc.), it will be turned into an <@id> attribute in the output XHTML only if there is also a <ref type="internal" target="#[theId]"> in the same document. The <ref> element will then be turned into an internal link to the XHTML element concerned. This is how internal linking within a document is accomplished.
• Other types of ref element include <ref type="detail" target="[an id]">, which I've inherited but which is wrong (the id should be a URI); this will be fixed eventually, but both before and after fixing, the element is turned into a link to [the id].htm (a "detail" page, from the locations subcollection or similar). There is also <ref type="external">, which is a normal external link, and (for instance) <ref type="internal" target="/page.htm">, which is a link to an ordinary page in the root of the site (which is where all pages are served from).

The Guidelines doc is dispersed across various doc and pdf files, and should be a single XML document which will appear as HTML on the site. I've been working to convert it, but it's slow going. I've done one other site info document, and created a template for them. The need to fix XML issues and convert existing documents which should be XML but aren't is going to add about a week to this job, unfortunately.

Configured some indexes as well.

The "original" map is now functional. I've also changed the way I'm handling pages and folders so that it's much simpler all round, and there's only one XSLT file. This is how it works, basically:

• All pages are now served from the root, with [xml:id].htm file names.
• All pages apart from the map pages are handled by index.xql.
• All pages apart from the map pages (which produce XHMTML directly) are passed through general.xsl.
• All links are simplified because all pages are served from the same directory.
• Bibliographic sources are working well.

Next steps:

• Literary references are supposed to work the same way as bibliographic references.
• BIBL1.xml should be in a subdirectory.
• The selected menu item should be highlighted when you're on one of the root pages.
• Remaining PHP pages need to be converted.
• List-of-ids page needs to be created (as a sortable table with lots of info).
• Site layout needs to be redesigned to be expandable and to use actual text.

I've done more updates and fixes to the location files, and transferred some data from /topics to /locations.

I've also got the basic setup for the "original" map working: the image map showing all the sections, and then the individual section display with navigation between sections. What remains to be done is the display of the locations as little stars. That's tomorrow's job.

I discovered today that backups from the db come down with all indents removed. That makes them not really useful as backups, or as a way to bring down a local copy of the files, to do search and replace etc.

Advice from the eXist list pointed me at webdav, which I haven't used before. This is partially useful. You can mount the db like this:

sudo mount -t davfs "http://localhost:8080/mapoflondon/webdav/db" "/home/mholmes/WorkData/english/map_of_london/webdav"

This gives you a folder which is readable, but of course not writable because it was mounted as root. You could edit as root, or copy the files to another location and edit them there; then you could upload them into the db in the normal way.

All of this roundaboutitude is convincing me we really need to be using SVN for our data. I'll speak to JJ about this tomorrow.