EMLS

I'm presuming I've completed this, because the code is working on all the current browsers...

James C. on the TEI list suggested that clicking again on a note anchor should hide the note that was shown; look at implementing that. Also, the code uses offsetTop to position the note; that's probably a bit too loose, so we need to implement the solution from the Devonshire project, using the findPos() function at the end of this file.

I've gradually evolved a policy for handling XHTML and its mime type. We've determined that there's no reliable way to get good results if you serve XHTML as application/xml+xhtml, so unfortunately the policy for the moment is to serve them as text/html using the xhtml11_compat serializer. See the following posts for more background:

• This post
• This post
• This post

I need to check that EMLS is doing that for all pages, and check that the search works OK on all supported browsers.

The link to the blog was pointing at the old location instead of the current blog URL.

The search page was failing to show results in Safari. Running a search would result in "DOM exception 7", which is "NO_MODIFICATION_ALLOWED_ERR", whenever I the JavaScript tried to set the innerHTML of an element.

This flummoxed me for ages, until I finally figured out that it's caused by the response header of the page. If the page is server as application/xhtml+xml then Safari presumably considers innerHTML inappropriate, since the page is not HTML. If the page is served as text/html, then it works. This is logical, if rather infuriating. For now, I've set the search page to serve as text/html using the xhtml11_compat serializer. I'll now go and investigate ways to use proper DOM methods rather than innerHTML, which is a non-standard hack in any case (although it's supported by all browsers).

Just posting minutes spent looking at what is completed, and discussing what should be done.

Karin's old stylesheet is still on the site, and is now available as an option in the browser View menu.

Implemented these changes requested by Ray:

1. Remove the View column, in favour of...
2. The title of the article should click through to the XHTML version.
3. Give the XML and Text links a class, so they can be styled small and icon-like.
4. The Vol/Issue column can simply read Issue.

Spent a bit of time making minor tweaks for Ray. Added a new task for some stuff which will take a bit longer, and so have to wait till I'm free.

In response to Karin's request (comment on previous post), I've distinguished each type of link that I can distinguish (based on what's in the target attribute) using these classes:

• NoteLink
• ImageLink
• ExternalLink

The original RefLink class no longer exists, so its properties (cursor, mouseover etc.) will have to be moved to the new classes, and modified as appropriate.

As Karin begins to look at the output and compare it with the original HTML documents, a new issue is emerging: in the process of converting HTML to SGML in 1999, it appears that all anchor (<a>) tags in the HTML were converted into simple <emph> tags. Although the original HTML anchor tags were included in the SGML in the form of comments, they were not subsequently carried over when the SGML was converted over to P4 XML (presumably in 2000, although the revisionDesc refers only to "SGML" even in 2000), even the comments were lost.

Preliminary analysis suggests that there are 1125 instances of these links-which-are-no-longer-links. They could only conceivably be recovered manually, either through working on the XML side-by-side with the original SGML, or by working from the online HTML versions of the documents on the purl site.

Karin had a number of requests through the day for changes and tweaks in the XSLT code to support the display features she wants to use. Also, we spent a fair amount of time finding and diagnosing errors in the XML; the three large XML files which were previously unavailable due to having a broken eXist are now available in the temporary new deployment Greg has provided for us on port 9090.

After much hacking around I came up with a solution for identifying when a match in the page should be ignored for linking:

    <xsl:template match="exist:match">
        <xsl:element name="span">
            <xsl:attribute name="class">SearchHit</xsl:attribute>
            <xsl:variable name="ThisId" select="generate-id(.)"></xsl:variable>
            <xsl:for-each select="//exist:match[not(name(preceding::*[1]) = 'exist:match')]">
                <xsl:variable name="ThisHitNum"><xsl:value-of select="position()" /></xsl:variable>
                <xsl:if test="generate-id(.) = $ThisId">
                   <xsl:attribute name="id">hit<xsl:value-of select="$ThisHitNum" /></xsl:attribute> 
                </xsl:if>         
            </xsl:for-each>
            <xsl:apply-templates />
        </xsl:element>
    </xsl:template>

This works for all reasonable cases; there are some occasions where it seems to fail, but they're extreme (such as searching for the word "text", which occurs dozens of times in a document). I'm going to stop at this point, because for most cases it works well -- better than planned, because you can jump directly to the match in the document -- and because getting it to 100% reliable might well be impossible, given the limitations of eXist's KWIC functions.

Michael Beddowes and Ron Van den Branden had a couple of suggestions on the eXist list for handling the searches, and it looks as though Ron's idea could be made to work. This is an outline of what we might do:

• For each hit in the KWIC, get the @exist:id of the parent node, and construct the link with that.
• For each hit in the page display, get the @exist:id of the parent node, and construct an id (not sure yet how to differentiate multiple hits with the same parent).

Still thinking...

Karin has started work on the CSS code for the site, so we're now working together, in between my hacking at the search functionality, to make the site look and work correctly. Spent a couple of hours working on problems with the TOC and underlying bibl elements: the titles of reviews are marked up in such a way that they contain more info than the TOC should show, but aren't sufficiently marked up to enable us to discard extraneous info, so after some attempts to solve the problem with XSLT we decided that it would be best to edit the original XML to make the titles show the way they ought to. This data was added by Gerry Watson anyway, so it's not original source material.

Also added handlers for inline quotes, block quotes, unordered lists, and a couple of other bits and pieces. We're still struggling with the broken eXist db on Lettuce, which now removes all the contents of a collection every time you upload a single file into it.

After much hacking around, posted this request for suggestions on the eXist list:

I'm working on kwic-display searches, and specifically on the problem of giving each of my list of matches a link into the exact location in the document where the match occurs. What I'm basically doing is to make number each match (hit1, hit2 etc.) in the kwic display to create a link into the document itself, pointing at doc.htm#hit1, doc.htm#hit2 and so on.

In the target document rendering code, I do the same search in the XQuery, and then in the XSLT I convert exist:match tags to tags, at the same time giving them numbered ids (id="hit1" etc.) so the URL will jump to the correct location in the document.

drawback, however, is the problem described by Michael Beddowes in <news://news.gmane.org:119/68bc01c6a727$cf8aa190$0200a8c0@michaelnt2k>. The KWIC display call is triggered for each node which contains one or more hits, whereas the XSLT numbering system is triggered for every individual hit. Therefore if a node contains two hits, I get only one KWIC entry (which is OK), but in the page I get two hits, and therefore the numbering schemes get out of sync.

I'm attempting to get around this by trying to detect, in the document rendering XSLT, a situation which parallels the single-node-with-multiple-hits scenario in the KWIC code. If the XSLT can detect when two hits are in the same node, then I can only id the first node, and my counting won't be disrupted. The second hit will still be highlighted, but it won't have an id.

I thought that this code should work:

    <xsl:template match="exist:match">
         <xsl:element name="span">
         <xsl:variable name="ThisId" select="generate-id(.)"></xsl:variable>
         <xsl:for-each 
select="//exist:match[not(preceding-sibling::exist:match)]">
             <xsl:if test="generate-id(.) = $ThisId">
                <xsl:attribute name="id">hit<xsl:value-of 
select="position()" /></xsl:attribute>
             </xsl:if>
             <xsl:attribute name="class">SearchHit</xsl:attribute>
         </xsl:for-each>
             <xsl:apply-templates />
         </xsl:element>
     </xsl:template>

In other words, when an exist:match tag has a preceding sibling exist:match, we should be able to assume that the KWIC code would have triggered only once (returned only one node), so I don't need to give an id to it. If it doesn't have a preceding sibling, it will be the match that shows up in the KWIC display, and so it will be a link on the search page and the element in the document needs an id.

However, this doesn't always work; the KWIC code works on the level of a text node, so a matches may be preceding siblings without being part of the same KWIC hit if an intervening node exists. My next thought was that matches would be in the same KWIC hit if they were siblings with only an intervening text-node between them. So I tried this pattern:

<xsl:for-each select="//exist:match[not(name(preceding-sibling[2])= 'exist:match')]">

That doesn't quite seem to work either, and I'm not sure why. Has anyone worked on this (Ron??), and if so, do you have any hints or suggestions?

I started working on the code to make search hits in the search page link directly into the same location in the target document, so you can jump not only to the document but to the exact location in the document containing that specific hit.

The early stages went well. I set up the article xquery to do the search based on a searchString parameter in the URL, and I got the XSL to highlight all the matches resulting from that. I had one slight wrinkle with the URL hash in a Cocoon matcher: the hash has to follow the search parameters rather than precede them, otherwise Cocoon fails to match the URL. This is probably my fault; I'd always thought of the has as coming before the search, but perhaps not (or perhaps it's variable).

The next problem was more serious: the number of matches in the target article turns out not to be the same as the number of matches found by the search page. This is because of a peculiarity about the way eXist processes and returns kwic displays. In the XQuery code (find.xq), I'm processing each node which contains a hit in order to be able to pass that node to the kwic-display function. However, if a node contains two hits, then the node is only passed once into the function (the second hit is ignored). In the article.xq page, I'm processing the whole article so I can only highlight all matches. This means that linking to a specific match is problematic because I'm doing it by counting the hits, and the number of hits is not the same in each scenario. This needs some thought...

After a lot of hassle, I finally got the search working in a way that seems satisfactory (tested on FF, IE7 and Opera). There were a number of AJAX and Cocoon hurdles to overcome on the way, well worth documenting:

• Firefox was gacking on a number of empty elements being rendered by Cocoon as (for instance) <span/>. When the browser hits an element like that, which should or can have content, it disrupts the internal DOM. Remember to fill empty elements with &#160; entities to make sure they stay open. You can't use &nbsp; in a Cocoon/XML context because it's not an XML entity (it needs the XHTML DTD to be resolved).
• Once that was solved, I hit a problem with the AJAX request response. When Cocoon serializes XML, it prepends the XML declaration. When Firefox receives a block of XML in the XHTML namespace, it appears to try to make it valid (in this case by wrapping it in <html> and <body> elements), which then enclose the XML declaration. My solution was to create a new XML serializer in Cocoon, in the EMLS sitemap, like this:

  <map:serializer logger="sitemap.serializer.xml" mime-type="text/xml" name="utf8xhtmlnoheader" src="org.apache.cocoon.serialization.XMLSerializer"> <omit-xml-declaration>yes</omit-xml-declaration> <encoding>UTF-8</encoding> </map:serializer>

  thus explicitly omitting the xml declaration. Then FF receives the XHTML fragment OK, and accepts it. Then, when it inserts it into the page, it appears to discard the wrapper tags -- presumably these are only used for internal well-formedness or validation checks -- so everything works OK.
• I need to be able to provide links from the search hits to their locations in the article text, and the way I intend to do that is to give each hit link a number in its url, so that a url from a search hit link looks like this:

  ...article.htm#hit2?id=blah&searchString=blah&hit=2

  meaning this is the second hit from that search string in the article. Then I can update the article rendering code so that it does the searching (in article.xq), and higlights the hits (in XSLT), giving each hit an id based on its hit number.

The next stage is to rewrite the article display code to add this functionality. Added a task for this.

Despite the ongoing problem with the eXist db on Lettuce, which Greg is currently working on, I continued working on the search page. I decided to learn how to use the kwic functionality built into eXist, which took a while because the documentation is sparse; in the end, I found a sample file posted by Wolfgang on the eXist newsgroup, kwic.xq, and that got me started. Managed to get the XML part of the system working (in other words, a query to find.xml returns a nicely arranged TEI XML document containing the search hits organized appropriately, with kwic hit strings).

Doing the actual search from the page still doesn't work, though; an HTML rendering of the data (serialized as XML) is being returned, but the JavaScript is gacking on it so it's not inserted into the page. I think this is because it has an XML header, and also because it's peppered with unnecessary namespace attributes. More on thie tomorrow.

Greg noticed an error generated during the text output pipeline. Tested and debugged, and determined this:

• When you serialize text, the serializer still expects XML as input; it simply strips out the tags.
• I was giving it text as input (from my text_out.xsl), so the root node wasn't found.
• Added a text node wrapping the XSLT output, which got rid of that error.
• Noticed that the XSLT wasn't actually doing what I wanted it to do (hiding the teiHeader data, adding linebreaks, removing extraneous space).
• Fixed this by adding the namespace prefixing throughout the XSLT, and then by adding normalize-space() to the output from the text() matcher.

Everything now seems to work as intended.

Made some progress:

• Created the search page itself (search.xq), based on the ScanCan page.
• Rewrote the XQuery to lift appropriate details from the P5 EMLS code as opposed to ScanCan, and started doing the extra customization which is needed to take account of the fact that EMLS files are collections (issues) containing multiple articles, while ScanCan has each article in a separate file.
• Worked on the find.xq retrieval system; got the basic stuff working (retrieving articles according to the search selections in the dropdown lists), but before I was able to get the text search incorporated, I hit the problem with eXist described in the Maintenance blog.

I'll get back to this later. Very frustrating to be interrupted in the middle.

Puzzling error that turned out to be caused by a combination of a Cocoon bug and a Firefox bug, worth documenting:

• Cocoon serializes an empty element such as div with no content as a self-closing element. It should not do this; elements which can contain content should never be self-closing.
• When Firefox encounters a self-closed div which should not be self-closed, it fails to close it, placing all subsequent content inside that div.
• As a result, my note popup was not showing up in FF because it was parsed as part of a preceding div which was display: none.

Links from the contents page to the HTML view of the complete issues are now working (had some struggles getting fn:doc to work with this -- remember in future that fn:doc returns a node(), not an element()!). There are no XML links as yet, but XML urls work (e.g. this).

The HTML output is only partially rendered at the moment; there needs to be proper rendering of the TOC, and footnotes don't seem to be working in the new location -- this needs fixing for both individual articles and issues.

The EMLS project now has its own account, which includes Greg and myself in its group; Karin will be added later so that we can all log on as ourselves and edit materials in the emls account (no-one should need to log on as emls). I've posted the site materials there, and everything created so far appears to be working (except for the links to full issue display, which are not finished). The new, hopefully permanent, URL for the project is:

http://lettuce.tapor.uvic.ca/cocoon/journals/emls/

Right now we're providing links to individual articles from the TOC, but we should also be making issues available as single documents (presumably), because that's the granularity of the actual XML. Ongoing work on Lettuce has rendered it non-functional, making it pretty much impossible to continue work on the Search system, so I diverted to working on the problem of issues:

• Created an XQuery file for retrieving them by vol and issue. This should be very simple, but I'm not sure of the actual results of wrapping a complete TEI document in a pointless wrapper tag just in order to make the XQuery well-formed; I'll test that once the server is working.
• Added new code to the TOC XSLT to create a link to an XHTML rendering of the issue.
• Added appropriate sitemap configurations for the XML of the issue and the XHTML rendering.

Gave some consideration to how we should actually implement this. It probably makes little sense to spew out the entire issue in one page; perhaps it's better to render the TOC for the issue, with links to the articles. Then one would be able to get an overview of the contents of the issue, and access its articles, undistracted by the contents of other issues (as you would be on the TOC page). The issue TOC could have a link to the complete XML file (XML needn't be chunked), but the individual article links would access the articles in the same way as the main site TOC links do.

This would mean the issue TOC is similar to the main site TOC, except that it would also include lots of metadata and other info from the teiHeader of the issue XML file.

I think this is the right approach, so I'll continue with it, but I'm inviting comments from Ray and Karin about this, in case they see things differently.

Spent some time analysing the problem of square-bracketed non-linking URLs, and wrote some regexp code to turn them into plain ref tags. Had to run the regexp code JEdit because oXygen cannot match on multi-line expressions, and many of these things have return characters in them. For future reference, this is the regexp:

Search for: \[([\s]*)(http://.*[\s]*)\]

Replace with: <ref target="$2">$2</ref>

This worked OK except for one file, 4-2, in which there were several instances of nested square brackets, and the match was greedy so extra stuff got into the URL and the ref tag content. Fixed these manually.

The result is that we now have working links, consisting of the URL as both target and content, which are converted into working links on the site. However, the worst part of the original problem remains: the linked text doesn't form the link, it follows it, and there's no way except manually to fix this. I estimate up to four hours to do this; it'll have to be done painstakingly based on the original HTML versions of the articles. I'm waiting for Scott's response on whether I should do this myself, or it should be handed over to ETCL staff.

Added code for handling internal links and simple tables.

Further to the posting on image handling, another similar problem is the large number of URLs included as text, not links, in square brackets; they look like this:

[ http://kartoserver.frw.ruu.nl/HTML/STAFF/krogt/ct_eng.htm]

These can't be turned into links using XSLT because they're not elements. This is rather bizarre, because in the original HTML version they're clearly links (see, for example, this page). I'm coming to the conclusion that the original conversion from HTML to XML was done very badly. Looking at the changelogs, all I can see is a list of edits to SGML ("SGML edited by Kathryn Harvey"), with no mention of XML conversion, but the respStmt says "Conversion from HTML to TEI.2 conformant markup: Kathryn Harvey". This is undated. Clearly, in the process of doing this, the identification of links as links was lost.

It's impossible to know whether the original conversion was made between HTML and SGML, and the linking was lost at that point, or whether the HTML was directly converted to TEI.2, and the links were discarded then. However, since it's clear that the XML produced by Kathryn Harvey does not adequately represent the source SGML or HTML of the documents, and consequently none of its descendant formats do either, I think we have a strong case for further editing of the XML. I don't know if this should be my job or not; it's tedious but not difficult, so it could be assigned to anyone. It needs to be done because the current XML is historically inaccurate as a representation of the original publication.

This page clearly illustrates the problem.

The handling of figures (images) included/referenced in articles has been very difficult to manage, because of profound inconsistencies in the original markup. Figures are linked from their positions in the text using a <ref> element with a target attribute pointing to the xml:id of the figure, but the xml:id referenced appears in at least three different places:

1. On the <tei:figure> element itself.
2. On a <tei:head> descendant of the <tei:figure> element.
3. On a <tei:item> ancestor of the <tei:figure> element (part of a list, when figures are enumerated in a list).

More variations may emerge later as I work through the code, but these three are now handled, with images showing up correctly when ref elements are clicked on, and they provide patterns which can easily be adapted to handle other variations.

This has raised one interesting issue, though; in my opinion, the inconsistency in markup is unjustifiable, and results from errors by the person or persons who did the original markup (or perhaps errors in the way the original HTML was written). However, since this is in essence a historical preservation/publication project, it's not appropriate for me to amend the markup (other than to convert it algorithmically to a P5 format, which I've done). Therefore the XSLT must jump through a lot of hoops to ensure that the content is displayed consistently. I have no problem with that, but it is time-consuming, of course.

Worked all day on this:

1. Got files uploaded into Lettuce eXist db.
2. Got TOC working here
3. Got XML output working (see links from TOC).
4. Got quite a way with XHTML output (see XHTML links from TOC).
5. Created a barebones CSS file for basic HTML display.
6. Wrote JavaScript for showing footnotes as sidebar popups.
7. Began writing code to handle display of figures. Problems here because original code appears to be a bit of a mess; xml:id attribute is sometimes on the figure element, and sometimes on its child head element. Curses.

Masih belum ada Cocoon and eXist on Lettuce, so Greg set up a sandbox environment for me to work with on his machine. Began debugging and fixing the code written in the last couple of days. Got the following working:

• Table of Contents as XML (teiCorpus document).
• Table of Contents as HTML; with working JavaScript-based sort-by-clicking-on-headers: sortable by title, author, vol/issue and type.
• XML output of individual articles (TEI documents), accessed through TOC links.
• Framework for HTML output of articles (no templates for anything yet, just the basic page with title and site includes).

This should port directly to Lettuce as soon as it's ready.

Posting minutes spent creating the previous tasks in the blog, which plot the project progress into April.

Finally got the admin client to connect and create a db, then uploaded the files. Can't test XQuery yet...

The remaining issues with P5 conversion turned out to be due to the fact that the schema was missing the figures modules (which handles figure and table elements and sub-elements). All files now validate as P5.

1. Structure of tables has changed. This may take some figuring out.
2. <figure> element is different. This will need a little research.
3. <xref> element needs to be effectively converted to <ref>. Fixed that (although we may have to watch for oddities in the "to" attribute).
4. Attribute "lang" not allowed. Should be xml:lang. Already added code to fix this.
5. There are some empty n elements in the source. These are errors in the original, so I fixed them there (by deleting the empty attribute).
6. Some date tags have value="yyyy-mm-00"; this is illegal, so added some code to strip off the last component.

1. Attribute "corresp" is in linking module, so added that module and all was OK. In the process, noted that Roma is currently slightly broken; it does not save the title and description you give to your customization in the ODD file.
2. Added an ident attribute based on the guess that only English and French appear as content of the language tag. That may or may not prove true with other files, but works for vol 1.1.
3. Restructured <change> element significantly. It seems to have been much simplified in P5.

Began working on the P5 conversion code, building on what was already done for ScanCan. Discovered that only Xalan correctly handles the default namespace assignment (and presumably it's Xalan that's doing it on Cocoon). First attempt shows the following problems:

1. Attribute "corresp" not allowed on bibl. This is important, because it's how the index links to the articles, but it may be solved by adding the linking module into the schema (see Roma and Guidelines).
2. Element "language" requires attribute "ident". This may be a simple fix trapping for the relevant language names and adding the appropriate values.
3. Element "respStmt" is structured wrongly. My code for handling respStmt was based on the ScanCan practices, so it will need modification here.

This is the current status of EMLS:

Before we can set up the eXist db for EMLS, we need the 1.1.x release of a build of Cocoon with eXist built as a block. Wolfgang has promised that it will be released any minute now.

Meanwhile, the P4 format of the existing EMLS files has become obsolete; they should really be converted to P5 before we go forward with a site. I've been working on XSLT to convert the ScanCan code to P5 this week, and that is now working; I'm going to move next to the ACH abstracts, elaborating the XSLT as I go, and finally I'll deploy the same code against EMLS so that we can at least start the project with good P5 XML. That should be done (I hope) around the same time that the new eXist version is running on Lettuce, at which time we'll be able to start work properly.

If you'd rather leave the code as P4 and work with that, then let me know and I'll start writing stylesheets for the Web interface. I don't really mind either way. I think P5 has more long-term stability, but it will take a little longer to do the conversion (an additional ten hours or so, conservatively).

To build the entire Web interface, I would estimate something like:

1. Setting up the DB: 4 hours.

2. Index/TOC page: 4 hours.

3. AJAX search page: 8 hours.

4. Article display (XQuery/XSLT/XHTML): 20 hours.

5. Site styling/CSS: 20+ hours (depends largely on what Ray wants -- apparently simple requests for particular display features can eat up days with CSS).

6. Peripheral pages (About, Contact, etc.): 5 hours (assuming content is readily available).

7. Debugging, cross-browser testing, fixing and tweaking: 10 hours (very roughly).

8. PDF output (if required): 20 hours.

That's a total of just over 100 hours, give or take.