teiJournal

Created a set of documents covering all the major document types outlined here, drawn from these sources:

• ACH/ALLC 2005 Abstracts (by XSLT conversion). We now have a pretty good start on doing a full P5 conversion of these documents.
• ScanCan, by XSLT conversion.
• EMLS (for types such as Note, Editorial etc.), by manual markup.
• Pure invention (for types such as Announcement, Advertisement and Corrigendum).
• New markup of existing texts (for Fiction and Poetry).

We now have a document set that can be used to refine the detailed instructions for markup. Added a task for the next stage.

The document set is here, and the schema is here.

As part of the work to plan a generic teiJournal Web application, I began the process of creating some sample documents we can use to test and refine our schema and XSLT.

I started from a random document selected from the ACH/ALLC 2005 Abstracts project, and adapted some XSLT from the ScanCan P5 conversion. I generated a schema from ROMA, keeping it quite limited and eliminating a number of elements from some of the core modules. The results is (currently) only 300KB; I think it could be refined further, especially by eliminating some elements from the teiHeader which I'm not actually using, but I want to look more closely at them first; it seems that better options than <classCode> exist for categorizing different types of journal contribution (such as <category>).

Next, I worked on the XSLT. It's XSLT 1.0 (for maximum portability), and I found that making the TEI namespace the default ns for both XSLT files, and switching to Xalan for the transformer, eliminated all namespace problems (such as empty xmlns attributes). I got the transformation working for the first sample file, with only one significant exception. Bibliographical references for online resources were encoded using an <xptr> element, and its crdate attribute was used to encode the "last accessed" date, but crdate seems to have been eliminated from P5. Posted a message to TEI-L about this.

Next, I'll work on a larger group of documents from ACH, including images, tables etc. There is some significant value in doing this anyway; the ACH/ALLC Abstracts project will need to be ported from Mustard to Lettuce when Mustard is rebuilt, and in the process it will have to be updated to work with the new Cocoon/eXist stack. In the process, we might as well do P5 conversions for archive purposes, and make P5 the default XML view of the data. So work on this XSLT would have to be done at some point anyway, irrespective of teiJournal.

The plan for teiJournal proceeds. Today I've researched and written two documents:

• Textual features for teiJournal, based on an analysis of the tags used in the ACH Abstracts and ScanCan projects, as well as the DHQ Tag Zoo document.
• Components of a journal volume, which enumerates and describes the separate sections of a print volume, and considers how the information to appear in them might be encoded in the master document, or harvested from XIncluded constituents (the articles, reviews etc.).

This gives us a sound basis on which to proceed with more detailed planning. Next is the creation of sample documents, one of each type, then the description of output formats.

For the first half of the "Document types and features" task, I created a taxonomy of document types here. This is based on an analysis of a range of different journal types from the Current Periodicals section of the UVic library, and the Oxford Journals online site.

The next phase is to work out the divisions and components of a typical journal, and of each of the document types.

The first part of this task (enumeration and description of output formats) was completed on March 6. The second part has been spun off into a new task.

This is part of the detailed project plan process for the proposed teiJournal project.

teiJournal will produce a range of different output formats, including P5 XML (individual documents), P5 teiCorpus docs (volumes and the entire collection), PDFs (individual articles and volumes), and XHTML (abstracts, articles, TOCs, indexes, search interfaces etc.).

This task is to enumerate and describe in detail each of these formats, including (for instance) the default format of bibliographical info (Chicago?), layout of each format (e.g. running titles), the content of each format, and the abstracted and user-configurable aspects of each output type (font, style, page size and other choices, especially for PDF).

For each document type identified in this task, create a sample document. In at least one of the documents, include all the document features we want to cover, in mock-up form.

For the teiJournal project, make lists of:

• Major document types we need to cover (preface, article, review, review article, etc.)
• Document features we need to cover (lists, images, tables, notes, etc.)
• Major document divisions (header, front, body, back) and what will be included in each.
• Components of a journal volume (title page, publication info, toc, index, etc.)

Use ACH, ScanCan and also the DHQ Tag Zoo document as source material.

A tabular listing of all tags used, and all non-default attributes and non-arbitrary value sets is now complete, viewable as HTML and XML . This is a good start.

The next stage is to create a sample document including all of these features, but realized in P5. This should include a complete bibliography with examples of all the main item types. We should also look closely at the DHQ Tag Zoo document to see if we can identify weaknesses in our P5 document type, and any document features we haven't covered.

I'll create tasks for this.

The first phase of planning is to get a good idea of the range of tags, attributes and attribute values used in the two projects we've already done which will form the basis for teiJournal: the ACH Abstracts, and the ScanCan project.

The problem is that both projects use a relatively tightly-constrained subset of the TEI P4 DTD, but the project DTDs are large and inclusive because they were generated at the beginning of the process when it wasn't yet clear what elements and attributes would be used. In addition, many attribute values which are simply open string values in the DTD are used in a more constrained and methodical way (with a fixed set of values) in the actual projects. Therefore we need a way to get a clear overview of what elements and attributes are used, and particularly what values the latter actually have. Once we have this, we'll be able to map these directly to P5, and create a restricted P5 schema using ROMA to recreate the same basic document format. Following that, we can convert the ACH docs to P5 (which we need to do anyway), and we can write some detailed instructions for marking up articles, as the basis for all the transformations and other dependent code.

I spent this morning trying a variety of ways to get a useful report out of the old eXist db on Mustard, showing all the data we need. Because of the requirement to retrieve all and only distinct values for element and attribute names, and attribute values (very expensive in processing time), and because of the need to search across two different collections, this is a slow job.

I ended up with nearly 200 lines of XQuery and a query which is takes several minutes to execute, but the results look promising. I need to write some XSLT to produce useful output now, filtering out what's not necessary (id attributes, for instance) and presenting the results in a table.

This task has been set back a bit by other priorities, but I'm also breaking it down into a series of individual, more specific tasks.

Much back-and-forth by phone and email with Peter and Scott, culminating in a cautious decision that I should go ahead and write a formal project plan, including estimated timelines, with no final decision yet on whether the project will go ahead in the end. I'll build into the plan at least two or more stages where, if we should stop the project, we'd still have something useful.

Entering this as a task with a deadline next month.