LEMDO

Following up after 2020-06-15 mtg with MH and RM about design:

I've changed my way of thinking a bit about the LEMDO anthology.
I think we need five Jenkins builds:

1. lemdo-dev on Jenkins where we can see everything, regardless of status (roughest of rough drafts all the way to done).
2. A lemdo anthology on Jenkins where we can see what we are going to push to the public site (lemdo.uvic.ca).
3. And the three anthologies on Jenkins: DRE, QME, ISE

Then we push LEMDO, DRE, QME, and ISE to lemdo.uvic.ca, lemdo.uvic.ca/dre, lemdo.uvic.ca/qme, and lemdo.uvic.ca/ise (unless we can secure dre.uvic.ca, qme.uvic.ca, or ise.uvic.ca for these projects).

I have an idea for the menus for LEMDO-dev. I think it should have all the menus for the live site + a menu for "Drafts". Basically, we can publish everything but the stuff under that menu (which will be governed by docTypes).

Other possible top-level menu items are:
About Anthologies All Texts How To Documentation

After the team mtg, met with MH, TEH, and JJ and came up with a plan for encoding the QME performances. Since performances must be understood as equal to the text (and textual artifacts), we are encoding the performances using a `<facsimile>` element, with `<surface>`s representing individual scenes; these surfaces will then be keyed to a timeline, at which divs can point using `@corresp`.

I'm going to test out this method with FV and report back.

After reading through Hinman to determine how TLNs are actually counted, JJ and I realized that we should be using the terminology turnover and turnunder for what we've called hungwords in the past; this is a more granular way of describing where precisely these things exist in relation to the preceding/following lines. To encode these, we will use:


<seg type="turnover|turnunder">(word</seg>


The last thing to figure out is how we should encode the bracket that usually precedes the turned word; we may want to regularize these at some future point so we might decide to tag the preceding bracket with a pc element so that we remove it from indexers et cetera.

Met with JJ to discuss glyph encoding and came up with a fairly robust course of action for encoding. While most glyphs are fairly simple (long S) there are instances of combining characters (y + (e|u|t) above) and we assume there will be similarly difficult glyphs.

We divided the problem into four parts:

1. How to encode in the charDecl

   • JT will discuss with MH and SA about the best strategy, based off of some of the work MH + SA have done for LOI.

2. How to encode in the text itself (i.e. what should the @xml:id of the glyph be)

   • We think yUAbove, yTAbove et cetera would suffice and is easily readable by encoders etc

3. What to put in the glyph tag

   • There are a few options: 1) Nothing; 2) the literal character [which is what we've been doing so far, but may not suffice for characters in future]; 3) a normalized version of the string (i.e. yu); 4) A regularized version of what the character represents (thou).
   • JJ and JT think (4) is the best approach as it will help for future searching and indexing; it is basically a short form of using choice and friends.

4. How to process and render:

   • This will come later, but basically, if we decide to use a regularized string, we can either replace the string with its regularized format (or make that an option) or display the character as best we can with the full-featured font that we use.

To do: JT will discuss some of the various encoding solutions with MH and SA; once decisions are made, JT will implement.

Fall 2019 LEMDO Project Plan

Team: RAs at UVic (Ashley Howard, Kate LeBere, Chris Horne); RA at UNR (Sarah Johnson); Programmer (Joey Takeda); Director (Janelle Jenstad)

Project Title: Remediation and Documentation

Outcomes

1. Some ISE editions will be fully remediated and available on LEMDO (by RAs).
2. One DRE edition will be fully remediated and available on LEMDO (by JJ).
3. One QME edition will be fully remediated and available on LEMDO so that we can resolve performance notes encoding and video embedding for the LEMDO platform (by JT). 

Objectives

1. Clean up six editions.
2. Clean up the metadata for all the files in those five editions.
3. Get the digital surrogates for those five editions (and metadata for the digital surrogates) into UVic ContentDM.
4. Republish those six editions on LEMDO.
5. Finish the documentation for all the parts of the edition.

Deliverables

See Asana LEMDO-Tech space for details.

Tasks

See Asana LEMDO-Tech space for details.

Texts

1. JJ does 1HW
2. JT does FV
3. AH does AYL
4. CH does H5
5. KLB does Oth
6. SJ does TBA

Process

1. We all remediate one or more OS texts. Pick a facsimile from the ones available, document the choice in the metadata, and make sure the Library adds each one to ContentDM.
2. JJ and AH finish the documentation for OS texts.
3. We all remediate one or more M texts.
4. JJ and AH finish the documentation for M texts.
5. We all remediate an annotation file.
6. JJ, AH, and JT finish the documentation for annotations.
7. We all remediate a collation file.
8. JJ, AH, and JT finish the documentation for collations.
9. We all remediate one or more critical paratext files (secondary). This task will entail cleaning up bibliographies and adding entries to BIBL1.
10. JJ, AH, and JT finish the documentation for critical paratexts.
11. We all remediate one or more supplementary texts (hybrid primary/secondary).
12. JJ, AH, and JT finish the documentation for supplementary texts.

Necessary Information

Types and Origins of Texts to Be Remediated

1. OS: Old-spelling transcriptions of primary texts (from IML or EEBO-TCP)
2. M: Modern-spelling texts of primary texts (from IML)
3. Critical paratexts (from xWiki or from the old ISE, QME, and DRE sites)
4. Annotations (XML files written in an IML-XML hybrid)
5. Collations (XML files written in an IML-XML hybrid)
6. Supplementary materials (from xWiki or from the old ISE, QME, and DRE sites. These texts were treated as critical paratexts in the ISE Platform; LEMDO treats them as primary texts.)
7. Title pages from QME (already in good TEI; need LEMDO teiHeader)

Remediation Processes

There are four ways in which we convert texts for LEMDO.

1. IML to LEMDO TEI-XML: OS and M texts.

IML files are stored as .txt files.

JT and/or MH run multiple conversion/cleanup cycles to produce rough LEMDO TEI.

RAs clean up the TEI and check the accuracy of the transcription simultaneously.

1. EEBO-TCP TEI-XML P5 texts to LEMDO TEI-XML: OS texts only.

We get EEBO-TCP files from GitHub. They are already encoded in TEI P5.

JT runs a conversion process that produces relatively good LEMDO TEI.

RAs clean up the TEI and check the accuracy of the transcription simultaneously.

xWiki custom HTML to LEMDO TEI: Secondary texts, About pages, Paratextual materials

RAs clean up the files, with varying amounts of copyediting and checking. Generally, the editor of the edition and/or the Anthology Lead will be involved with checking.

RAs encode the files using the LEMDO tagset for born-digital files.

RAs clean up the files, with varying amounts of copyediting and checking. Generally, the editor of the edition and/or the Anthology Lead will be involved with checking.

Some texts were programmatically converted. 

We might have to copy and paste the text into our own XML files. 

1. XML to TEI-XML conversion: Annotations and Collations

These files are stored as .xml files. They are encoded in a boutique IML-XML language.

JT does a conversion that embeds pointers to these annotations and collations (about 60% successful with the best texts).

RAs check the pointers. 

RAs manually add pointers for annotations and collations that could not be resolved programmatically.

RAs clean up the encoding of the annotations/collations.

Got myself set up in the office and met with JJ and KL to discuss the broad plan for fall and, more immediately, the plan for the team meeting tomorrow. Also discussed layout issues with PS and came up with a tentative plan for the complicated annotation/collation display: readability is a primary concern here since the annotations can range from two words to large paragraphs and block quotations. He's going to look into technical implications and see if our plan is feasible.  

Chatted with PS about the LEMDO site design, which is looking great. Discussed some of the odd features that crop up in EM drama and some of the workarounds we might need to do in terms of popups etc. Things are coming along nicely.

Met with JJ and MH to talk about the LEMDO infrastructure, including build jobs, shared resources, and the potential for ODD chaining. Will do some more research into ODD chaining and think about cost/benefit.

Met with JJ and MH to discuss LEMDO framework, plugin, and the anthology overall.

For the LEMDO anthology, we will include everything, even if it is in draft et cetera, and will provide a "two-step" link as per MoEML (i.e. click to view draft, and very obvious THIS IS A DRAFT watermark background). Projects can decide whether or not they want to have that link available on their site or they can hide it using CSS simply (i.e. #showDraftLink {display:none;}). People can still find that page if they know how to construct the URL, or they can even view on the LEMDO site. We may decide to split the LEMDO listing so that you can choose between peer-reviewed, already anthologized, not peer-reviewed, and not yet anthologized documents. 

Met with JT to thrash out plans for processing ODDs.

Met with JJ to discuss priorities, which was very helpful. She laid out some ideas for documentation and structure; she's going to get working on some texts to help flesh out some real life encoding.

To do so, she needs some stuff from the TCP, which we've done before. Since we've needed to do this a few times, I've decided to automate the process slightly:

• First, I downloaded the JSON version of the TCP catalogue, and converted it to a much smaller XML file, deleting fields that we don't need
• Then, wrote another XSLT that resolves an STC number to a TCP number, if available, retrieves that TCP document and then passes it through the (existing) set of templates for converting TCP to LEMDO

I needed to regenerate the schema, but the TEI outage was causing issues. A simple fix for this was simply to switch the defaultTEIServer parameter and the defaultTEIVersion parameter in the XSLT. I've now codified this in the ANT as a set of conditions, so that, by default, we use the TEI server, but if that's down for whatever reason, then use the TEI Jenkins.

Extensive mtg with MH about LEMDO; mostly just catching up and came up with a plan for various copies (for lack of a better term) in collations, and sent an email about that. Also started to work through the issue 328 on Github, but ran into some issues with the git pull stuff (which I'll figure out myself)

Then, met with JM twice, to work through encoding challenges and to resolve some issues; fixed some of the code template stuff in the process and he updated his oXygen (he was running on 16). Worked through some encoding problems and some errors, including some pretty printing problems; we really should see if we ca disable that by default in a project.

JM sent a bundle of files for KNDW edition--including a selection of annotated primary sources, an OS text, a modern text, and an essay--for conversion, which I've now done. It was an interesting experiment: most of these documents were in older forms of IML, which we didn't necessarily know how to handle. So I've modified the IML conversion slightly to allow for an AB element, so we can fix hierarchy issues more easily.

Over email, discuss Stylesheet work with MH; he has created a branch and we're going to do a dive into the odd2lite stuff. This will be particularly useful for LEMDO, but will be broadly useful for any of the Endings projects

Biweekly meeting with JM and worked through a number of issues. Came up with plan for the next two weeks

• JM's SVN access isn't working at the moment, so he's getting me to merge his Ado_M [DONE]
• Rom_Q2M was out of date; reconvert [DONE]
• Convert Err for editors and add linebreaks to the OS as test case for bare display
• Start thinking about metadata display (discuss with MH and PS)
• Add schematron to use em-dash instead of double dash [DONE]

Added some new schematron for common problems that arise after conversion that have to be fixed by hand, including when there is embedded verse in prose (which could be done programatically but is safer to do by hand to confirm lineation, etc) and other common errors (unescaped angle brackets et cetera). The idea here is that if the texts pass the basic TEI conversion (which they basically all do), the converter must then check the text and ensure that they are valid against the LEMDO schema; if not, then the texts must be modified slightly by hand to fix things that require human intervention. It's a bit more involved, but it stops absent minded conversion (which I am guilty of).

Fixed a ton of small issues (mostly embedded verse, but some other small things) and converted Mac and 1H4 as per JM.

Long meeting with JM to discuss progress; discussed plans and priorities going forward. He also asked for a better display of overlapping annotations, which I agreed with; he wants everything that crosses multiple lines to be displayed as a line on the side rather than multiple types of underline. This is, of course, a rendering issue primarily, but we do need to get the algorithm in place to get the annotations and collations embedded in the HTML so that rendering decisions are easily handled later.

So, reworked the code so that if something goes over two lines, the behavior is slightly different. One way to resolve this would have been to create container divs, but decided against that since it's a lot of manipulation of the HTML; instead, we'll just add an onclick() event to the div itself and a left-border, and make sure the events don't propogate. Seems to be working alright; more work will likely need to be done, but it's mainly aesthetic at this point

I realized that the annotation matching code was a bit too generous with its selection of nodes, due to the preceding:: and following:: axis. I've now reworked it into something much more efficient and cleaner. The problem is that there are a number of different ways in which an editor might select a bit to annotation; the best way would be to point directly to an element (say an l or an lg), but most often editors want to gloss particular words or phrases using an anchor, so we have to handle both (and both in combination: say something starts at a line but ends halfway through another). So the solution to this is to first generate all the nodes that are between the two boundary points (using the incredibly fast << and >> selectors, which I've just learnt about) and then group them based off of ancestor nodes that were in that original selection of nodes. (That's not a very elegant way to describe it, but it's all documented heavily in the XSLT.)

This new process shored up a number of issues with the annotations. Many of them are aesthetic, some are functional, and some are processual. We'll have to have a set of diagnostics to check whether or not something that was annotated with anchors could be better annotated by pointing to a line or a line group. That won't be trivial, so that'll take some time

After corresponding with JM, determined list of priorities for the conversion. I decided to do H5 first since it is one of the most densely annotated/collated and the editorial process should be swift.

Converted H5_FM and Q1M and got annotations working; there were a few bugs in the HTML code, which I fixed. Also discovered the issue with part lines; the IML usually flagged the part lines with an empty line preceding the inferred line (i.e. via one of the LNs), so re-wrote bits of the conversion to handle that. Also had to make various fixes to the apparatus linking code to handle supplied tags.

Discussed plans and evolved a new idea for an ODD for ODD/Documentation with its own processing, to be part of the Endings project. More coming on this.

I've been trying to figure out the best way to retrieve all nodes between two anchors; I'm not changing the document hierarchy (i.e. I'm not tagging them), but I want them for labelling purposes etc. But doing following:: and preceding:: or using the following/preceding operators (>>) operators with intersect isn't very efficient.

It's nice because we have @xml:ids on these things, so it's easy enough to know when to stop. It's not generalized for non-id'd things (although, I bet you could use generate-id(.)).

<xsl:function name="hcmc:betweenTwoAnchors" as="node()*"> <xsl:param name="anchor1" as="node()"/> <xsl:param name="anchor2" as="element(anchor)"/> <xsl:variable name="f1" select="$anchor1/following::node()[1]" as="node()"/> <xsl:variable name="f2" select="$f1/following::node()[1]" as="node()?"/> <xsl:copy-of select="$f1"/> <xsl:choose> <xsl:when test="($f2/self::anchor and $f2/@xml:id = $anchor2/@xml:id) or empty($f2)"/> <xsl:otherwise> <xsl:copy-of select="hcmc:betweenTwoAnchors($f1,$anchor2)"/> </xsl:otherwise> </xsl:choose> </xsl:function> <

Working on the ODD to allow teiCorpus (which was trivial), but then spent a long time trying to get a schematron rule in place to check the attributes of a processing-instruction. Note that XSL variables do work in schematron patterns, but functions and other bits (at least with the current stylesheets) do not.

The second issue was that the schematron building code (that I lifted from MH's Keats code) does not automatically process "foreign elements" (i.e. XSLT variables) in the final XSLT step (iso-to-svrl-2.xsl). You have to add an "allow-foreign=true" parameter to the XSLT for it to work as expected.

Following MoEML, I added the egXML and CSS validation steps to the buildValidate.xml build. Had to modify it slightly, since MoEML often uses collections whereas the builds currently rely more on filesets. It simplified things slightly.

Had to deal with quite a bit of fallout, since our egXMLs were either invalid or our schema wasn't handling things properly.

Phone and personal meetings with three people re the latest developments; slight reshuffling of plans.

Meetings today with MC, JJ and JT, with lots of planning and the writing of a sitrep to pass to MB.

Met with JJ and MH to discuss plans moving forward, particularly documentation and plans. We're going to make sure our documentation is easily navigable from the outset, with maps and decision trees giving multiple pathways to get to a particular bit of documentation.

Also made some headway on the build processes themselves; I've been able to modularize the builds so that the master build (lemdo/build.xml) calls a number of other builds (all with suffix '_master.xml') that themselves call either XSLTs or other modular build processes. Each of the smaller master builds can be run on its own (it also includes the globals module) or they can be run from the master one

Working on the XWiki conversion; oddly, the converter tool that XWiki uses to export the texts is inconsistent. If you use the "texts" space, it gives you a fragment with a content element (in no namespace), if you don't, it gives you the full XHTML page in the XHTML namespace. So if you want to download all of the texts associated with a text (i.e. all the things in edition 1HW), then you use the text space and convert the content element; if you want to get something like a paratext page, then you have to convert the body in the XHTML namespace.

I now have a conversion working that first turns the former situation into the latter and then converts it all the same way; I also worked a lot on the linking, file naming, and other parts the HTML conversion. The conversion looks good and gives us an excellent starting point; there will need to be lots of edits to the pages, but otherwise, they are good. We can cut off XWiki anytime now.

Spent some time implementing what MDH, JJ, and I discussed regarding @rendition and @rend. We won't use @rend and we will always use @rendition or @style; we would have our own prefix ('rnd'), which are initially based on simplePrint.

The next challenge will be a way to say: either choose from this list OR use one defined within the file. I don't think you can do that in Pure ODD, so I'll have to figure out some schematron to constrain that.

Lots of work on the DRE framework. I have a very quick and dirty imitation of the current site (but using grid, so it flexes nicely); it's not perfect, but it's helpful for visualizing the content. Made a test build to see how the original XML looks when moving in the HTML and it's all looking fine.

Started thinking seriously about the site page framework. MDH created a blank HTML site page, which has all the semantic bits that have to be filled in. I like this idea a lot, especially if we combine it with a flex+grid approach in CSS. Did some reading up on that, played around with grids, and then converted some of the About pages into TEI.

Mostly working on supplementary documents and corresponding with JL and CW. These documents are coming along nicely.

Converted 3LL, 3L3LL, OWT, and TTR3 from DRE. These were fairly straightforward and look fine in the output. The rest of the QME texts have specific issues, which I've emailed JJ about.

Converted FV, FBFB, and Leir from the QME site; the conversion seems to be working well still, but every text throws a new wrench. These look good, though, but they are the already published ones; the next ones might be a bit trickier.

Lots and lots of work on the conversion code and converting the DRE texts. Tried three (1HW, 2HW, AHDM) and they only required a few small tweaks to get them working. The annotations and XWiki documents are also working, which was a nice surprise. The code had to be tweaked a fair bit and it certainly requires some better documentation, but it's coming along

Also created a new build in conversion/buildEverything.xml. This code calls the three other builds (buildSgml, buildApparatus, buildXWiki) so that we can build a number of documents fully. There is now a parameter for works, which is a list of works (AYL, H5, MND) and the build iterates over the list, doing the full conversion for each work. Worked well for DRE. (To note: each individual build works by itself and does not depend on buildEverything in any way.)

Pop up dialog boxes created from embedded Javascript in Ant were appearing behind oXygen, which wasn't very user friendly. I posted a ticket on the Oxygen forum (https://www.oxygenxml.com/forum/topic16346.html) and the solution suggested worked. Basically, I had to create an empty JFrame and then place the other dialog boxes on top of them. It slows down the process slightly, but I'd rather sacrifice speed for usability. Code below posted for future reference:

Got the Jenkins job working with the LEMDO name and a new build up and running after a request for the id page from JL. I've split the id list generation code out from diagnostics.

I've also combined the username and password getting Javascript into one dialog box, which is much nicer for the user--it's a much more familiar look and feel.

Editor packages are working well now on my machine and the instruction documents are starting to come together. We need to start testing cross-platform, which is the next step.

Lots of work done today on editor packages in oXygen. I've now put the schematron within the ODD file, have it building out in the proper files (thanks to MH's code), and have a nice validation suite working. In data/tools/, there's a file called "validate.html"; editors can open it up, press the play button, and all of their files are validated.

I've modified MH's process schematron code slightly so that it doesn't fail immediately if it finds an error; it compiles all the errors first and prints out the message, and then fails. I like this a bit more, since I would likely use this code when debugging/testing the ODD, etc, and it's nice to see all the errors all at once.

The setup and updating builds are both working nicely; still not cross-OS compatible, but otherwise they're nearly ready to be used, I think. However, the committing stuff is melting my brain. I've looked at both svnant (http://subclipse.tigris.org/svnant.html) and svntask, and tried nearly everything I could find online, but nothing seems to work. More work to be done there.

Edit: I was misspelling the variable (I was passing ${password} and not ${pwd}). A very silly mistake, which I will chalk up to the combination of travel, conferencing, and sun over the last few days.

The LEMDO repository is now restructured and renamed. Save for a likely few errant strings, everything should now be "emd"(early modern drama) or "lemdo". It came together quite quickly and enabled me to fix some things in the code. We'll still have to fix the conversion code (esp. in regards to paths), but that should be fairly straightforward.

Also started to work on the oXygen/ANT utilities for editors. oXygen editor variables (even the password ones) are not right; they are passed as parameters to the ant build, which show up in the ANT text stream at the bottom of oXygen, which is no good. The ant input element doesn't work either, since there is no way for ANT to work through the oXygen IDE (as far as I can tell, at least). After some research, I found an okay solutioon using javascript; although I struggled with the whole Nashorn/Rhino thing. I did this
load("nashorn:mozilla_compat.js"); which I got from stackexchange here: https://stackoverflow.com/questions/14835882/apache-ant-gets-frozen-while-running-this-script and here: https://stackoverflow.com/questions/22503100/java-8-javascript-engine-backwards-compatibility. I'll need to consult with others though, since I'm not all too familiar with this stuff. It works, however, and seems secure. Lots more work to do.

Attempting to work with author mode, CSS, and ANT. It only works if the ANT file isn't prefixed by "build" (i.e. not associated with ANT in oXygen) and you have to work around the lack of flexibility in the ANT's XML. You can more or less force it by putting an @el='blah' on the description element; you don't get hierarchical XML, but it might be enough for a simple ANT file with some documentation.

Met with JJ and MH to discuss the repository structure for the LEMDO project. There was confusion to be hashed out, but we ultimately came up with a structure that I think works well. It is being forwarded to the major stakeholders to be approved.

We've also come up with the short form "emd" as a prefix

JT and I talked at length about the work he's now doing renaming everything tagged with "ISE" that now needs to be LEMDO; and we devised a cunning plan for editor projects:

• When an editor is getting started we first send them out an Ant build file which they open in Oxygen to set up their project.
• When it runs, that file creates directory, checks out the svn repo, and then runs a second ant task that does this:
• svn update to get any changes to core files
• svn export to get local copies of schemas (avoids externals)
• That ant file is the one you open and press the red triangle before you start work.
• Another ant file is the after-work version, which does svn update and svn commit.

So: no need for svn externals; no problem if connectivity is temporarily down; no need to learn svn; everything happens in Oxygen.

...in keeping with our forward planning.

..for the benefit of other people who will be working on related texts. Results were valid, no errors.

I'm beginning the work on customizing the ISE3's ODD so that we can have a "standoff" element to store all of the database-like stuff that the rest of the Endings projects have been putting in the teiHeader. It is based off of the stdf proposal, but is less concerned about linguistic annotation.

Basically, the standoff element contains model.listLike and listBibl (which is part of model.biblLike) and spanGrp (for annotations).

Note to self: Adding the standoff element (or any custom namespace element) between the teiHeader and the TEI element requires adding that namespace to the @defaultExceptions attribute in the schemaSpec element.

Combined the two build files that we had in the ISE3 repo (ise3/diagnostics.xml and ise3/site/build.xml) and their associated ant_globals.xml files. It's now one build process, which by default goes through and:

1. Validates the TEI in ise3/data
2. Runs diagnostics on the TEI in ise3/data
3. Then begins the static build process

Working on the creation of the Standard XML, which for now means resolving pointers. Since the ISE has decided to use more granular prefixDefs (i.e. 'doc' for documents, 'pers' for person) instead of using general ones (like MoEML), prefix resolution can be more generalized. There's a template that matches every TEI attribute that has a pointer data-type and, like the Endings diagnostics code, resolves the pointer based off of prefixDefs. Seems to be working well.

Wrote a utility function for retrieving data from the taxonomies; this will be needed to complete the Dublin Core metadata in the pages.

I've decided we should build the HTML pages from a genuine template, so that anyone who knows HTML can easily edit such things as the menu items and the boilerplate content. I've set one up, and given it a basic flex-based CSS layout that shouldn't be too hard for later styling. I'm thinking about building in the small-format device rulesets from the beginning, so they don't end up being grafted on later. The basic process would be to load the template, and process it through XSLT templates, with the source XML document passed as a tunnelled parameter; that should mean we can pull anything we like from the source XML fairly easily, and meanwhile most of the boilerplate stuff will just fall through in an identity transform. XML will be processed under a distinct mode.

JM needed documentation for subversion so got a start on writing that. We already had some stuff in there, but a lot of it was unedited stuff from MoEML's. Rewrote it significantly with code blocks and clear instructions. It's a bit less discursive, but it should do the job for now, since JM needs it right away. Used the oXygen TEI P5 --> HTML conversion and then saved as a PDF.

Lots of work on the apparatus conversion. I've included MH's character code into the XSLT, which seems to be working well. Most of the plays are being handled well. One small issue is that all of the other annotations from the XWiki docs are being included as well, but those have already been converted to be inline on the documents. One solution would be to create a list of all the documents that don't need to be brought over.

Also began refactoring the process for attaching the standoff annotations to the texts. It's complicated business, since there's a lot of attempting to find the right documents to attach the annotations to. Currently, the process runs like so:

1. Create a list of documents and their associated annotations
2. Then iterate through that list
3. Tokenize the base text and add ids to each character
4. Attempt to match the apparatus files to the base text using character ids
5. Then, add anchors in the base text where the apparatus ought to attach
6. Finally, untokenize the text and just leave the anchors

A better and more flexible process might be to fork on type of text using the ISE document types. If the document is a primary source, then tokenize; otherwise, leave it. Then, for any apparatus documents, see which document it is attempting to match (encoded in its relatedItem in the header) and then look for the tokenized version. It will take longer in the long run, but it is simpler than nested for-each lists in ANT.

Regardless, the match_lemma module was (as MH rightly noticed) complicated and difficult to debug. I've refactored it now into multiple functions and added a "verbose" switch for very detailed bug reports. There's still lots of fine-tuned error checking and documenting to be done, but it makes more sense than it did before.

I've been doing some modularization of the ant build processes in the ISE3 repo, and taking the opportunity to get familiar with JT's work so far. I'm going to start work on the HTML output next, leaving the annotation/collation stuff to him.

MT did a lot of work encoding the facsimiles using feature structures, and I've now merged that into the repo. It was messy because of the horrible tangle of "externals" we have, which are not really external; they're just local relative links. The external pointing from data/sch to sch did not update itself automatically; I had to delete the files in that folder and svn up to get it to refresh them. Annoying.

Met with JJ and JT. Results: I've overwritten the current data/text versions of the test plays with my own generated versions, which now have glyph encoding; in the case of H5, I actually ran the glyph processing on the existing versions since JM confirms he's been editing those directly (although not since July). JT has brought back some of the collation/annotation offset-checking code from GitHub so we can integrate that. We've decided that we should not hard-convert editor-specified offsets using string matches into anchors until rendering time, because it's easier for editors to work with them if they're text-based, but we'll provide easy checking for editors.

Found some more minor validity issues in the output from the conversion; dropped Titus (not ready for prime time), and worked on the remaining six until all were valid. We're now ready to look at annotations and collations.

The character work I'd done and tested actually wasn't getting called in the build process because of a pre-existing set of templates that were converting some of the entities. Took out the old conversion templates and did a bit more work on mine, and finally H5 was correctly converting. Added in four more of the test plays, did some more tweaking and fixed some errors in the original IML, and then added in Timon and Twelfth Night to make a set of seven plays. These are now not only validating against tei_all but also against the ISE3 schema. We're making progress. Next is Titus, after which we'll move on to annotations and collations.

Ironed out the last of the bugs, and tested with the file pilot works; they threw up a couple of other bugs in the conversion, which I've also fixed. I think we're now ready to move forward.

There are some very weird things in the IML. I have thirty-odd weird entities left to figure out, but they're getting stranger and stranger. At some point we'll reach diminishing returns, so I'll just put something in the output that requires human intervention.

I've worked through most of the curly-braced entities and built on the taxonomy JT created for glyphs, splitting it into two (ligatures and single chars), just to aid in clarity; I've created equivalences in the form of choice elements in a list in a TEI file, and that can be plugged into any transformation. The idea would be to order the items by length descending, so the long ones are done first, and have a template that matches text() and runs all the equivalent replace things using analyse-string. We may have to build the analyze-string element mechanically. Alternatively, we could just run a scripted cli replace thing.

Came up with a detailed plan for identifiers, for consideration by JT and JJ; then started looking at the stuff that's been done to the original transformation process I had running. The changes are mostly mysterious and undocumented.

Crucial for the RAs to work is the list of ids produced by Jenkins, so I'm working on that. I think I'm done for the moment, but there are lots of issues with the actual structure of ids to deal with.

Finally got back to this project, and started moving forward. Met with JT on Hangouts and set up a basic plan and some tasks. Cleaned up the repo a bit, and made a couple of changes on Jenkins (more to do there).

LONG meeting, with decisions in notes and in Asana.

Skype meeting on SFO project. Outcomes documented in email.

Priorities per meeting discussion:

• Make all XML valid so build process is useful again (JT).
• Call a freeze on XWiki work (JJ) and convert XWiki pages (JT).
• Fix errors in biblio code from XWiki sources (an RA working for one of us).
• Finish documentation of conversion of texts, and iron out a few issues, then convert the texts (MT).

Spent some time reviewing JT's encoding proposal for SFO; it's coming together.

They have prepared ontology/taxonomy spreadsheets, and JT will turn those into TEI taxonomies before we start mapping them onto elements and attributes.

Met with JJ and MT today for first walk-through of initial work, which went well. All of the ANT builds are currently working on Windows, which is great. Also finished up some templates (Decided to move from the regular project templates to ANT builds, which gives us a bit more control of where the documents end up).

We also decided today to hold off on doing the editor specific ODD; it's all a bit tricky with potentially different frameworks for TEI because of different versions of oXygen. We can almost do the same thing in schematron anyway.

Been busy trying to get everything finished before vacation, so I've been forgetting to blog. Summary of minutes below.

Editor packages should be good to go. Everyone has schemas, xprs, and tools in their project files, with the tools doing fairly decent lemma checking. I'm fairly confident that the ODD is good enough to start working with. I was able to get annotation and collation templates ready as well. I also added DJ's files from a few years ago to the repo and started chipping away at them; they're good and interesting TEI experiments that need a fair bit of wrangling to get into ISE3 TEI. A good exercise, though, since it's contributing greatly to the ODD and schematron.

Minutes summary: 23: 330min; 27: 360min; 28: 540min (360 in office)

• ident="doc" | matchPattern="(.+)(#.+)?| replacementPattern="http://ise3.uvic.ca/$1$2". This gets us around TLNs not yet having explicit xml:ids (which we also decided will only be local to the document, not project-wide)
• ident="tln" | matchPattern="(.+)" | replacementPattern="iseH5_FM.xml#tln-$1". This will only be used in the context of @to and @from in the apparatus files. The prefix will be defined for each apparatus/collation document so that the link will refer to a specific TLN in the text file (i.e iseH5_FM will change).
• And simple ones to refer to people, document types, and glyphs

Posting time spent discussing lemma matching and milestone insertion over the past two days.

• TLNs, QLNs, L, and MS tags are all milestone units and not end of line breaks
• Editors can make up their own types of milestones with their own numbering systems if they choose
• White space line breaks (\n) are significant and represent the linebreaks for that particular edition

Met with MT and JT. We established coding and documentation standards for XSLT, Schematron, and Ant (to some degree), and figured out how the ISE2, ISE3 and GitHub repos will relate to each other and to the Jenkins build. Documentation in Github repo.