Mariage

Today:

• ol.Feature objects are being created with the outline colours from the XML files.
• Mousing over them lights them up temporarily.
• Clicking on them selects them (with the annoying OL click delay getting in the way, of course).
• The annotation is shown when the zone is clicked on.
• The page layout, metadata and heading are better organized to save space.

Found a lot of changes to OpenLayers, some bad (apparent bugs with setting properties in constructors), some confusing (ol.dom has disappeared), and some just a question of getting used to new ways of doing things. I'm using OL 3.9, and it's clear that MoEML would have to be rewritten considerably to work with it.

Much progress today:

• Added tileset-building to the build process. After various false starts, this ended up being done with vips called directly from ant using exec, so we have another dependency there.
• Added JSON generation from zones in image markup files.
• Added some extra python to figure out the zoom levels created by vips. One more dependency.
• Added OpenLayers (latest version) code to the repo.
• Tweaked layout to get the metadata out of the way.
• Implemented JavaScript and JSON rendering as part of the HTML build process, so we now have working zoomable images for the IMT files.

The next step is to write the JS that creates the zones and the navigation panel for the image.

Back to working on the build, working towards the OpenLayers image output:

• Implemented HTML validation as part of the build.
• Fixed a whole pile of errors showing up as part of this, including the need to split lists across page boundaries.
• Added JSON generation of categories for image markup files.
• Fixed XSLT handling for rendition elements in image markup files.
• Added display drop-down for metadata from teiHeader for all files.
• Researched and implemented a way of retrieving image dimensions using ant. These will be needed for generation of tilesets.

Fixed a huge number of errors in the source XML, and added some pre-processing to fix many of them. Set up three levels of validation for the incoming XML and the generated XML as part of the build process, and then started hacking away at the remaining errors in the HTML output. What's left, as far as I can see, is just the problem of milestone elements showing up as direct children of e.g. list elements. That's on the table to be fixed next week.

After some investigation, I found that the best option for validating XHTML5 is not the RNC schemas from whattf, which depend on datatype definitions in their own namespace that I can't seem to find a jar implementation of; instead, the VNU project provides a single JAR file which can do it. I've integrated that into the project and I'm now doing validation of XHTML5 pages produced by the build.

Two sets of errors were easily fixed: first, the encoders seem to have been making up language codes as they went along, so there were hundreds of invented ones. I've fixed all those. Second, the files are not in Unicode NFC. I have XSLT to fix that in the XML, but I haven't run it yet; instead, I built normalization into the pre-processing of the XML, which solves the problem for the XHTML output, and also protects against bad data coming in in the future. But I will fix the normalization in the source XML files soon.

What's left is about 1100 errors of the expected type (divs inside spans and the like), each of which will have to be looked at in the hope that generic fixes that cover lots of them can be found. Meanwhile, validation of XHTML5 is now a solved problem, very usefully, and I'll be able to port this fix to other contexts. I don't know if I can find a way to make it validate all the fragment files, though. One option there would be to build them all into a single file, validate that, and then delete it.

We now have a working job on our HCMC Jenkins box which builds the static Mariage site, thanks to anonymous read-only checkout from SVN.

This is part of the cleanup of XML prior to the new site building.

Over the weekend and today I've got the transformation working so that it now builds a completely updated version of the XML which complies with the current P5 (and some upcoming changes/deprecations), as well as abstracting all the styles into rendition elements. I've build an Ant build file which generates all this stuff automatically from the original XML, and then uses that to start building the web output. That means I can focus on website generation work without having to update the existing site code to take account of changes in XML, which is a relief. I'm building this so that it will run outside of Oxygen, in anticipation of its running on a build server we plan for HCMC.

I now have primary source text documents rendering, although in many cases the results are not pretty. There are some fixes to be made to the original documents, I think. We may also decide on putting menus at the top rather than down the side, which will involve some tweaks (but not much, really, because the new layout is designed to detach the site structure from the document display as much as possible). Tested the build on the laptop and it works great.

I've started the process of re-building the Mariage schema, XML and file structures. I have a "pre-process_xml.xsl" stylesheet which reworks the XML, moving everything from @style to @rendition/rendition elements in the header, and this is partially tested; it has a couple of bugs still to work out, in that it generates duplicate @xml:id elements for a handful of fw elements. The ODD file itself has been partly changed, but is not yet generating a fully-working schema (@rendition does not seem to be available), so I'm working on that now. Once the schema is working, it will disallow various old habits such as biblScope/@type in favour of @unit.

Todo:

• Fix the bug in xml:id generation for fw elements.
• Fix the ODD file so a schema allowing @rendition everywhere is created.
• Test the transformation and make sure it's producing fully-valid and unbroken XML files from the source documents.
• Update all XSL that looks at biblScope/@type on the site so it can handle @unit alongside it.
• Test-upload some transformed documents and ensure that they are displaying correctly in the interface. Fix any bugs arising out of that.
• Commit all converted files in one batch, after checking with CC for a good time to do it.
• Warn CC about changes.
• Upload everything into eXist and test extensively.

By the end of today:

• All wrinkles with articles worked out.
• Banner and menu in place.
• AJAX retrieval and display working.
• Footnotes working.
• An ant task to build the entire set is done.