A new edition of the Map of Early Modern London site, edition 6.5, was published on 15th September 2020. This is the first edition to include a fully-tagged transcription of the 1598 edition of John Stow's A Survey of London.
Many other changes are listed in the Release Notes.
A new edition of the Map of Early Modern London site, edition 6.4, was published on 26th June 2020. This edition is the first completely static edition; previous editions depended on the eXist XML database for search functionality, but this edition uses the Project Endings staticSearch engine, and requires no back-end database of any kind.
Many other new features, new texts, and changes are listed in the Release Notes.
I've done a bunch of work over the last few days debugging and fixing what turned out to be two distinct problems with the Agas JavaScript. One was a simple failure to deal properly with an unexpected return value from a calculation; all I needed to do was put in place a default value. The other was a bit more complicated; it turned out that it was actually caused by the existence of some places which had Agas map locations specified, but which didn't have appropriate subcategories assigned from the mdtEncyclopediaLocations parent category. Attempts to display any of these places on the map (when, for example, you click-dragged a box to highlight all the places, and caught one of these within the box) would generate an error which make the map stop responding. I've fixed the problem data, and added Schematron to prevent it from happening again. 240 minutes.
Wrote XSLT to generate library metadata for PDFs and EPUBs in the form of a spreadsheet; this has triggered some additions and rewrites of content in those files. Still need to add the abstract field once it's in the data. 180 minutes.
The encoding team raised the issue of needing to break various elements around page-breaks so that the fw elements don't inherit styling from their ancestor paragraph (or similar). I've written some usefully-repurposable XSLT in /db/data/utilities/break_elements_at_pbs.xsl, and debugged/tested it with TEH's help, to recursively split elements up through the hierarchy and knit them together with @next and @previous.
This was conceptually quite tricky, although the solution is relatively straightforward once written. Recording details here because one day I'll need to reorganize or restructure a document hierarchy in a similar way, and I don't want to have to reinvent this particular wheel. 120 minutes.
Trying to get validation working on Windows, JT and I have succeeded, but at the cost of breaking the regular diagnostics build on Jenkins. I think there's only one remaining problem to solve, though. 90 minutes.
CSS floats are extensively used in our primary source encoding, and they're now no longer really necessary in many cases (such as forme works). They also cause problems for the PDF generation. I've started a process to remove them, processing away all the right floats used for catchwords. So far the results are pretty good. I'm next going to look at running headers and page numbers. The code was a bit tricky because in some situations I needed to flip the order of forme works, and in others not. 180 minutes.
Worked with TEH a lot today on the FO rendering, and found a workaround for the problem of floats in forme works. I've tested a better encoding method for the situation where catchwords appear on the same line as signatures, using negative margins instead of floats, and it works well, so if JT can't see any reason against it, we'll implement that.
I also changed thousands of list type="bulleted" to list rend="bulleted", to conform with current TEI -- need to look out for fallout from that -- and put the redirects file into the XML processing pipeline so that it's available for building the id listing page, so that people don't accidentally re-use an already-obsolete id.
180 minutes.
The search box in the banner disappears when the page width is too small; I've coded a workaround which makes a new menu item appear to replace it, and I'm now waiting for that to percolate through the build to see if it works OK. 15 minutes.
Met with TEH and worked through some issues relating to the title page background, font embedding, and FOP configuration.
Met with TEH and got a run-down on the current situation; she's down to the last ten or so rather thorny rendering issues, in particular problems with whitespace and tables. We fixed a couple of things relating to block vs inline, and discussed some others. Good progress.
Yesterday and today, TEH and I got the code I originally started in September brushed up and working, so there is a functional PDF generation pipeline; she's now beginning to fill in some template details and getting familiar with how attribute-sets work. There's quite a lot of work to do here, but it's all going to be quite satisfying from an Endings point of view, and there's a lot to learn for TEH.
All TEH's work of course.
Helped TEH with some ePub generation debugging.
The new Schematron rules were being mysteriously triggered and failing the build at the Original XML stage; I eventually figured out that this was caused by the generated statistics.xml file, which only exists as a result of the build, so is not caught till build time. Fixed it. Did a couple of other tweaks to help with ePub rendering at TEH's request, and we talked about the CSS issues, which she's moving forward with.
Made a schema tweak arising out of CT's work. Worked with TEH on her normalization of the div structure in born-digital documents, which is now complete.
Having written it, we're now writing it properly. :-) TEH is very patiently starting back at the beginning to give us a well-organized and documented process for the epub generation. Did some basic planning and worked on the early stages today.
Received computer and spent some time tonight setting it up. Just a summary of some of what I have done:
With TEH, got a basic XSLT set up to convert the XHTML static output to something simpler for ePub. She's now working on that conversion, and meanwhile I'm adding templates which create all the additional files needed for the ePub creation. I've also spun out the ant tasks for this into a separate file, which will eventually handle all the ePub work.
TEH's experiments with ePub have shown that one source of invalidities is the remaining scattering of vendor prefixes throughout the CSS, all of which are probably now completely unnecessary, so I've removed them all (with the exception of those in the Open Layers CSS, which is better left alone IMHO). This should clean things up a bit and help with getting decent ePubs. Checked with PS before removing them.
With TEH, worked on the ePub project. She's building a test file to see what will work and what won't from our current files. I've added automated download/update of the epubcheck jar file to our static build process, and tested that. I can't yet see a way to pass a batch of files to epubcheck, so it may be the usual one-at-a-time process, but with luck it shouldn't be that onerous because (I think) ant will allow us to avoid instantiating a new vm every time.
Now that Jenkins is able to run the MoEML build again, I've been working on the fallout from the imposition of the new Schematron rules which aim to reduce the number of redundant hi elements. I've had to exclude Stow documents from those rules, because there are too many existing instances awaiting attention from editors who will re-encode them anyway. Fixed a few dozen errors spread across the rest of the collection.
Discussed some issues of CSS with LS who is making lots of progress on the mayoral pageant CSS; we really ought to get some of the common problems/issues documented in Praxis.
Also checked over CH's work in putting xml:ids on the divs in 1633, which looks good. Now we have to do some bigger regularization steps, including adding line beginnings where we can and some other small clean up stuff.
TE's work on teiHeader has highlighted the impoverished content model of projectDesc, so she's submitted a feature request to enrich it. Both JT and I helped work on it.
Finished the first draft of this presentation. I would like to get the image upload feature working in BreezeMap before doing the presentation, but that's not essential.
Met with MDH today to discuss the plan for putting Stow back together. Most of the big concerns were regarding rendition elements and pointers. I've been able to implement the first bit of the plan today: getting the renditions rationalized. It's a multi-step process like so:
I'm currently doing this after the original build, but I don't know if that's the right time to do it; the issue is that it breaks if you try to do in quick and subset (since there are now no documents that have to be resolved); this isn't great if you're trying to rebuild the reconstituted Stow in a smaller build process. I'll have to investigate that.
The now-fragmented Stow document needs to be reconstituted in XML form, even though we don't want or need a complete version in HTML. After JT's successful work splitting out the source files, we planned out in detail the steps necessary to reconstitute a complete XML version for the "original" set, from which the other XML versions will be generated. It ain't simple. It took two hours.
TE is working on the elimination of unnecessary hi elements, as a first (although pretty hard) XSLT project. Spent some time discussing approaches, and she had it working by the end of the day. Fixed the calendarDesc issue in simplePrint output (using settingDesc -- ugly but meh). Also fixed all instances of style attributes lacking a trailing semicolon, and implemented Schematron to prevent further instances. Lots of discussion of how to rework the teiHeader and split Stow.
Since most of our work is now being documented in Asana rather than here, I'm just posting hours spent over the past couple of days of version 6.4 changes, discussions of changes to the teiHeader, build process rationalization, and a variety of other things.
Following a successful release earlier in the week, we're now able to undertake more significant structural changes to the repo and the build process. The first one, done today, was to split out the news page into separate articles, deal with the fallout, and add documentation.
Bug found in modern map functionality, and new errors introduced into Stow and PERS1 by the checking process itself. Now postponed to Friday.
Release day didn't happen due to some last-minute glitches, so it's now been rescheduled, and a comprehensive release plan has been put together in Asana to avoid the stumbles that happened today. We all learned a lot, and it'll go well when we come to do it again in a couple of weeks. Fixed a lot of minor issues and polished off a couple of tickets.
Fixing a bunch of typos and slight annoyances; building out some changes to the home page per PS.
Like MH, spent most of the day fixing CSS syntax errors. I decided to write a process that goes at the beginning of the validation build that creates a bunch of small CSS files for validation against the VNU; each separate node becomes a CSS stylesheet, with the node's XPath is embedded within the filename. The errors are then parsed by an XSLT that also retrieves the node that contains the error and then outputs a message and breaks the build.
Plan of attack going forward, I think, will be to create a new validation process that will be the automatic transformation scenario for each TEI file. It will:
On Friday we deployed a new version of the VNU validator, and the fallout was significant, since it now appears to be checking all CSS thoroughly. This is great, but it found hundreds of errors, and I've spent the day fixing them and attempting to implement Schematron to prevent as many of them as possible.
Spent most of the day working on the HTML diagnostics, which I think are now done, at least in the XSLT 2.0 version. I've split the build into multiple targets and it runs quite smoothly. While I originally used regexes to see if a file exists locally, I've decided to just do it by sequence checking (i.e. $fileSequence[.=$thisFile]). After a certain point the regex becomes too slow whereas this is consistently quick.
Ran this on MoEML (15 minutes) and Graves (1.5 minutes) and the results are interesting. MoEML has a ton of broken links, particularly in the Ajax pages, that MH and I will need to discuss; Graves also has some broken links, but only on the gallery pages. From what I can tell, the titles are being processed by templates, but no appendix content is being created; we could decide to either just take the text, apply templates to the title in a different mode, or add an appendix to these pages.
I think there's some kind of bug in the way that successive animations (particularly if the last of them is ol.View.fit rather than ol.View.animate) are run in the current version of OL, such that the starting point of the final animation is not the ending point of the preceding one, but the previous zoom level. So the effect of zooming out and zooming back in again is to zoom out, but then jump back in to a close zoom, then pan to fit the target extent. That's jarring. I've tried every solution I can imagine, including precompose and postcompose scenarios, setTimeout to postpone the final animation, and many other permutations, but nothing works nicely. So instead I've included a little overview map that shows you where in the main map you are. That's the best I can do for now. When OL has gone through another couple of releases, we'll see if the animations chain better.
Tweaked the rendering of a box-selected area so that it zooms gently into place, rather than snaps as before. Then tried three or four more approaches to the zoom-out-zoom-in functionality without success. I still have one idea...
Lots of work prepping for the release:
Also chatted with JJ about plans going forward as well as MAPS1, which raised a lot of questions about the document. Did some investigation and wrote a long explanation with possible problems and solutions to JJ and AR
Mostly working on the Bills of Mortality finding aid; cross-referenced it with the TCP's huge JSON and added links to TCP and ESTC.
Today:
Ported over the XSL schematron pipeline to MoEML, which is working nicely. What's nice about this is that we now have the TEI schematron embedded into MoEML's own schematron, which is quite nice.
I've changed the main build (london/build.xml) to use the XSLT created by the schematron process and everything seems to be fine. It is very slightly slower than the schematron ant task, but it much nicer since the error messages fail the build immediately, rather than waiting for Jenkins to parse the "ERROR" output.
Met with JJ and MK to discuss mayoral pageants plans. JJ and MK are looking into facsimiles and may have found a potential one, which is exciting. Discussed plans going forward and how and if we should do the embedded transcription. MK is going to experiment with FAME2 to see if it's worthwhile while I continue tinkering with some code to turn our semi-diplomatic transcriptions into surfaces, zones, and lines.
One issue that still needs to be resolved, however, is file naming practices. Is it right to keep the diplomatic transcription as, for example, FAME2 or should it be FAME2_diplomatic and FAME2_modern? And if users go to FAME2.htm, what should they get? It seems intuitive for them to get FAME2_index, but we'll need to discuss that further.
Working more closely on the map rendering thing, and I still haven't figured out a solution. It's as if my zoom-out code is being optimized away, whatever approach I try. However, I found and fixed a bunch of old bugs in the code, and cleaned some things up.
Using (for the first time) ol.Overlay, I've got the popups working for source elements on the OL KML maps for locations. This was quite straightforward. The only remaining issue is the main Agas map zoom movement, where I want to pan out from the existing location, then zoom into the new one; I don't have that working yet. Once that's done, we're ready to go live with the new version.
Fixed the issue of the missing icon for points; it was a CORSS problem. A relative link fixes it.
Started work on making the source info for placemarks show up in a popup. It's basically working, but I have to figure out how to make the popup show where I want it to.
More work on the text to sourceDoc transformation. Tested across all of the published mayoral pageants and spent a good amount of time re-working the pb/fw extraction code, which is convoluted but I think is now working well. There are also some issue with hi elements and line elements to do with styling, but those should be resolvable soon.
Spent today experimenting on a text to sourceDoc transformation for the new Mayoral pageant protocol. I wanted to see if it was feasible to take a mayoral text that is encoded in MoEML's current standards and transform it into a sourceDoc version. It's a tricky thing, since the hierarchies are radically different. That said, I think I have it working; the hardest, and most time consuming part, was getting it so it would break the a paragraph/div at a pb element. Tested on MONU1 and CHRU1--works okay, but still more to do.
Today I replaced the Google Map with an OpenLayers map in the site pages. The map is based on KML rather than JSON, because that's what we have, but ultimately I'll probably switch it to JSON. It also needs functionality for showing the popup info about sources, and the OL KML parser currently fails to pick up the rose icon for points, but we'll get there on that one. I'm now a little less worried about being overtaken by the Google problem, but I'll need at least another couple of days to get everything properly shaken down to the point where we can publish.
I now have a fully working Agas Map based on Open Layers. There's more that could be done, and I will revisit the pan/zoom functionality later, but the next job is the urgent replacement of the Google Maps feature with OSM. I've successfully tested an implementation of an OL map using the KML that we're currently generating, and I think that will be the way to approach it. It'll take some work. At present, the Icon style setting doesn't appear to be supported, although it is being retrieved. I think we'll need to override the styles from the KML, which will actually be nice because OL gives us more flexibility there. The other thing not yet working is the annotation popups, and that may have to be done in a different way.
Further to my last post, I have now moved the regnal calendar out of the spreadsheet and into a TEI document; I just took the latest one built in the Jenkins build and went from there. It's should now be rendering properly on the Jenkins site; I had to change the structure of the document slightly, which caused a chain-reaction of errors throughout the static build, mostly to do with add events to the standalone header and with rendering the date popup. These should now be resolved.
The regnal calendar spreadsheet was modified so that it had an unexpected column on the left; this broke the regnal calendar processing, which relies heavily on the placement and order of the table rows and columns.
We need to update the Agas Map code so we can use the latest OpenLayers for GIS maps as well. It turns out that the changes in OL since the original code was written have been significant, so I made a start on this today. I have the basic functionality working (although I still haven't managed to successfully reproduce the pan/zoom functionality -- needs more work), but more important is the drawing code, which is only partly working; drawing itself works, but no feature is available at the end of the process, so the TEI output is not being created in the text box. We can't switch to this version till I've fixed this.
The controller didn't have anything set up to show the XSLT files which we provide as part of the documentation. JT reported it, and I fixed it.
Tiny bug in the TEI Simple XSLT broke the build: LS used the @resp attribute on <soCalled>, which is a nice idea and makes perfect sense, but which the simple code wasn't expecting. Took a while to narrow it down, but it was an easy fix.
MoEML training day today, so spent the day prepping and training. Went well, especially the hands-on stuff.
Put in ticket for @where in the TEI and debated on that thread
Since the static site is now deployed, we can finally convert all pointers to "mol:julian" to "mol:julianSic"; we'll have to change these on a case by case basis. These have now all been changed and the schema updated so that "mol:julian" is no longer a valid value. Luckily, I didn't need to change any of the static code, since it could handle all four cases (julian, julianSic, julianJan, julianMar). In the future, we might want to clean up the code so that it doesn't have to handle "julian," but, for now, it should be fine.
The more pressing concern, however, is the documentation. I've updated encoding_dates.xml to reflect new encoding practices and changed the wording where the difference were obvious. However, the documentation will need to be updated so that it includes a larger discussion of why we are disambiguating julianJan/julianMar/julianSic with evidence to support why we need to do this.
Martin, Tye, and I made a presentation on 2015-09-02 about this, which ought to be turned into the documentation. There's some good evidence in the slideshow that I had completely forgot about (the reason we did this in the first place is that the STC regularized to January 1 without doing the necessary arithmetic to get the date into the Gregorian equivalent).
Worked with MK and JJ over the last two days working on various mayoral pageant encoding. Each playbook gets two documents (as well as paratexts/critical introductions etc etc):
Created an intro to SVN using the training repo, and delivered it during the meeting.
JT pointed out that the redirect links are not working in the current MoEML, and on investigation it appears that we never implemented them at all. I did so, hastily:
Took a while to figure out how to get the first part to be valid; it turns out that the single quotes that used to be normal around the URL are now invalid, and you need to provide this formulation:
<meta http-equiv="refresh" content="0; URL={$targId}.htm" />
Fixed a number of issues with the now-live site, including the removal of the old stats.htm page in favour of the new statistics.htm, and some tweaks and updates to documentation to take account of changes.
Encoding of the Pantzer document was causing MoEML ids to show up as variant spellings for places, while missing the variant spellings of Pantzer itself; corrected this with some utility XSLT.
This is good to see, and it's also a pilot for the moving of the existing Jetty apps to the new web cluster. So far all looks good...
Built the eXist XAR and pushed it to the server, and then requested that the subdomain be pointed at the new site.
We really want the build to fail rapidly when there are duplicate ids, so I've made the duplicate id check terminate the XSLT and that process fail the build, so we should get the info very quickly and not have to wait for Jenkins to do the full build.
One of our team had not quite understood how to figure out what the next id should be, so created a stack of new documents with ids duplicating existing ones; it took a while to sort this out, but I think it might be done now (Jenkins will tell me soon). One reason might have been the alpha-sorting of items by id in the A-Z Index, so I've done two things: rewritten the initial sort, and rewritten the JavaScript version that runs when you click on headers, so that MoEML ids are treated by the insertion of leading zeroes before sorting, to get a more intuitive order (so 11 sorts after 9, for instance, instead of before).
Troubleshooting a content issue sent in by a user, I discovered that in fixing a previous bug I'd introduced a new one; it was a straightforward fix once I'd figured it out. Links in popups are now working again.
The build process throws up validity problems that are hard to catch before we get to HTML validation, such as the presence of spaces in URLs. I've had a shot at catching spaces within (rather than between) URIs in a new Schematron rule, and fixed a bunch of errors with AR.
Picking up work on the RDF project (which I will document on here). I've spent some time looking through the TEI listServ and issues to see what the best solution is for creating RDF from TEI. As far as I can tell, there's no consensus from the TEI community on how to approach RDF.
Also researching RDF and GeoNames, which might be the best ontology for the project. Trying to map over correspondences and see what looks good. Lots of work still to do, though.
You can now clone an existing feature and edit it; also I've cleaned up a lot more of the GUI stuff and interface strings, and started work on pastable GeoJSON.
Rather tricky work to enable new features to be created from drawn geometries, and their geometries preserved and shown as descs in the popups. This is now working. I've also fixed a bug related to the drawing of complex Polygons with cutouts; these now work correctly, and are distinct from MultiPolygons. Finally, I extracted a bunch of caption strings into a more flexible object which allows for addition of new languages and selects language based on the root lang attribute.
As part of moving all our GIS data over to clear GeoJSON, I've implemented @type="GeoJSON" for the <location> element, and I've fixed the five outstanding MultiPolygon problem locations by re-capturing that data from the original KML files from Locating London's Past, converting them to GeoJSON, which results in well-formed GeoJSON geometries. Where such geometries exist in location files, they are used in preference to converting the existing GIS coord lists when building the GeoJSON version of the Gazetteer. There will probably be some fallout in terms of rendering the Google Maps for the short term, but that will be fixable when we move to OpenStreetMaps. For the moment, we're in limbo between two systems.
At the same time, having rebuilt the schema, I found that @precision has been removed from the <location>, <birth>, <death> and <floruit> elements, making some of our code invalid. This is only in the dev version. I've reconfigured my own schema-building process so that it's pointing at the current release version; meanwhile I've reopened the ticket that caused the problem because it does need a deprecation period anyway.
MoEML now needs a better Vertexer that can do things right according to the right-hand rule, so I'm working on one as one of the test projects for BreezeMap. Today I rewrote the TEI/GeoJSON relationship so that TEI files now contain GeoJSON geometry objects right in their geo elements; XSLT to generate GeoJSON was updated to support both old and new formats, and I'm now working on a system for storing newly-drawn shapes in a new category on the location panel. After that, the plan is to integrate Nominatim searching; then we'll convert the MoEML geo-handling to use the new format and OSM, and convert all the existing files over.
I think I have the GeoJSON basically working; I just have to do the right-hand rule stuff, and a bit of cleanup.
I'm currently building GeoJSON output from our Gazetteer data for the Pelagios project, and using the opportunity to figure out how best to convert our <geo> elements into GeoJSON data. The basic output is done, but I'm now working on figuring out what type of GeoJSON geometry a given set of geo elements contains, and then how to output it correctly, while enforcing the right-hand rule. Much more to do here.
Our actual map coordinates are not GeoJSON, and polygons don't follow the right-hand rule. We can look at this implementation for an algorithm for the right-hand rule test/fix, and we can convert the geo elements to GeoJSON geometries.
Couldn't figure out why the MoEML build was suddenly broken, but it turned out that there was an empty <death/> element (no content, no attributes) which was tripping up the date processing. Added a Schematron rule to prevent it.
Spent a couple of hours fruitlessly working on a rendering bug in the static build: refs from abstracts going into AJAX files should not be local pointers as they are in their source file, since the local targets will not be there in the AJAX popup. But nothing I did had any effect. Very odd, must be missing something obvious.
The number of trivial errors had built up a bit, so I spent some time working on the most straightforward of them, and getting my head back into MoEML space.
Following work described in detail today on the Mariage blog, I've updated the MoEML build process validation of XHTML to use the explicit XHTML flag for vnu. Everything validated without change, unlike the situation in Mariage. I'm a bit surprised by that, and I'll take a deeper look at some point.
Put an hour into knocking off some of the growing list of minor issues in the diagnostics. More to do, especially on 301s.
No additional tasks for me arising out of this, but we will be making some long-term decisions about prefixDefs for dates.
MoEML pages were not correctly getting their svn info in the footer, which turned out to be because the version of svn set as the default in the Jenkins management interface was 1.5; I set it to 1.8 and all seems to be well. It's a global setting.
I've removed the MoEML Static build from Jenkins and added that process as a second build step in the MoEML job. That simplifies things a bit. There is now some duplication between the builds (validation and so on), so we may now be able to simplify further. The original build also creates some products which were intended for the old site, and those steps can be removed now.
I've also reworked the way the gazetteer JSON file is being produced; it's now placed in the /js/ folder, and a zipped version of it is created in the same place. The controller now serves zip files appropriately, and I've added a link to it in the gazetteer_about page.