Contact Us on 0800 123 4567 or info@example.com
For the PKP conference in July, Karin will have to deliver my presentation on teiJournal, because I'll be away. I started adapting the presentation from Vancouver a few weeks ago to make it more detailed and clear. Did some research on all the existing journal e-publication tools I could find, looking for any others which use XML in the way teiJournal does, but couldn't find any.
There's another day or so of work to do on this presentation.
The book "Programming Firefox" arrived, and the timing was good; I'm considering the possibility of using a XUL-based application for the editors to access the database, because it would allow more flexibility in terms of local file access and storage. I started working through the early chapters, and reading around the later ones to figure out how practical it is to deploy a XUL-based application right now. It seems quite feasible, although the standalone XULRunner app is still a developer preview at this point. I'll keep working through the book, because I can see lots of possibilities for an effective GUI using Firefox/Gecko.
Also spent some time looking for ways to batch-convert my growing collection of SVG GUI icons to PNGs for GUI building in Java and XUL. ImageMagick looked promising, but it does a lousy job of converting SVGs; they come out without colours. The GIMP does a good job of conversion, but isn't batch-scriptable. This needs more investigation.
XSLT 2.0 brings a lot of extra functionality to the table, and teiJournal will take advantage of it. One obvious step forward is the ability to define your own functions. In many contexts, this is much better than the old method of doing things using named templates, because you can call a function from right inside an XPath expression. We've recently needed some string-manipulation functions, and I'm anticipating the need for many more, so I've started building a utilities.xsl library, which will be released under MPL 1.1 if it ever gets released. Today I coded and tested these functions:
mdh:lastIndexOf, a simple but necessary routine that's missing from XPath 2.0 for some reason.mdh:truncate, a function for truncating a string to a particular maximum length and adding an ellipsis; this also has the an option to trim it back to the last preceding space, so that words are not broken in the middle.mdh:stripLeadingArticle, a function for use in sorting and presenting document titles which have leading articles. For instance, this title:TitleSortComparator I wrote yesterday; that works fine, but this provides another option in contexts where we might not be able to install our own Java classes under Cocoon. It can also be used for formatting the titles for display, which the sort comparator can't do.There's a lot to learn in XSLT 2.0, but this is coming along nicely. I still have to trawl through all my other XSLT projects to identify other utility code that could be encapsulated in this way.
I now have the basic system working in pilot form. This is what I did today:
xsl_attribute_sets_to_css.xsl file.
<map:match pattern="*.css">
<map:generate src="xmldb:exist:///db/teiJournal/styles/test/{1}.xsl" />
<map:transform type="saxon" src="xmldb:exist:///db/teiJournal/xsl_trans/css/attribute_sets_to_css.xsl" />
<map:serialize type="text" />
</map:match>
This is good progress. Now I have to revisit the database structure again, and figure out what needs to be stored where.
My plan is to store the data for CSS files in the eXist db in the form of xsl xsl:attribute-sets. This presents a number of challenges, and I got to work on one of them today.
This approach is useful because it enables us to store CSS data in a highly-structured format, so that we can read and write individual properties and values in the database; thus we can allow the user to customize the layout and appearance of documents through an browser-based GUI, and use the results to supply CSS for the site.
The first problem we have is that a we have very limited possibilities for storing the details of the CSS selector, and selectors can be quite complicated. All we have, really, is the attribute-set's name attribute, which is a QName, and for our purposes, is actually an NCName. (We could consider using the namespace prefix as another place to store information, but strictly speaking that would be abuse). Therefore we need to find a way to encode a complex CSS selector in the form of an NCName.
At the very least, we need to handle element names, spaces which separate them in a descendant selector, commas which separate them in a multiple selector, class names, and the periods that separate class names from element names. It would also be good to be able to encode the right-angle-bracket used for child-of. Ideally, we would be able to use use all of the characters allowed in CSS.
Since we know the name of XHTML elements, and we ourselves have control over the naming of classes etc. in our project, we don't need to worry about naming collisions as long as we're careful. All QNames must start with a letter or an underscore; this slight limitation suggests that we should use a known prefix for all of them, so that we can strip that off, and therefore be limited only by the restrictions in characters which apply to NCName NameChars. NameChars consist of:
Letter | Digit | '.' | '-' | '_' | ':' | CombiningChar | Extender
Obvious decisions are to use a period for a period, and an underscore for a space; we should also avoid the colon, because of possible confusion with the namespace separator. Digits probably make a poor choice for encoding anything except digits, because they can occur in various positions in CSS selectors. First, I considered using the extenders as substitute characters. Extenders are:
#x00B7 | #x02D0 | #x02D1 | #x0387 | #x0640 | #x0E46 | #x0EC6 | #x3005 | [#x3031-#x3035] | [#x309D-#x309E] | [#x30FC-#x30FE] \u00b7, middle dot \u02d0, modifier letter triangular colon \u02d1, modifier letter half triangular colon \u0387, Greek ano teleia \u0640, Arabic tatweel \u0E46 : THAI CHARACTER MAIYAMOK
... and a bunch of Japanese and Chinese characters. These are not so useful, really. The first three might be, but the first and third would be hard to distinguish anyway.
So I fell back on using non-English letters for substitutions.
This is the list of non-letter characters we need to cover (based on the CSS2 and CSS3 selectors from
http://www.w3.org/TR/REC-CSS2/selector.html and
http://www.w3.org/TR/css3-selectors/
respectively):
* [ ] = " ~ $ ^ | - ( ) : . # [space] > + ,
The period and the dash are acceptable in an NCName; only the following will need to be substituted. These are suggested substitutions. They're meaningless, and not human-readable, but there's not much we can do about that.
* ø U+00F8 : LATIN SMALL LETTER O WITH STROKE [ Ƹ U+01B8 : LATIN CAPITAL LETTER EZH REVERSED ] Ʒ U+01B7 : LATIN CAPITAL LETTER EZH = ŧ U+0167 : LATIN SMALL LETTER T WITH STROKE " ü U+00FC : LATIN SMALL LETTER U WITH DIAERESIS ~ ñ U+00F1 : LATIN SMALL LETTER N WITH TILDE $ ß U+00DF : LATIN SMALL LETTER SHARP S ^ ê U+00EA : LATIN SMALL LETTER E WITH CIRCUMFLEX | İ U+0130 : LATIN CAPITAL LETTER I WITH DOT ABOVE ( ʃ U+0283 : LATIN SMALL LETTER ESH ) ʅ U+0285 : LATIN SMALL LETTER SQUAT REVERSED ESH : ʘ U+0298 : LATIN LETTER BILABIAL CLICK # Ħ U+0126 : LATIN CAPITAL LETTER H WITH STROKE [space] _ regular underscore > ʌ U+028C : LATIN SMALL LETTER TURNED V + Ɨ U+0197 : LATIN CAPITAL LETTER I WITH STROKE , ɹ U+0279 : LATIN SMALL LETTER TURNED R
This is the sequence:
øƸƷŧüñßêİʃʅʘĦ_ʌƗɹ
I wrote templates which function as converters between the name and selector forms, and wrote a test package to make sure they work. Then I wrote a template for outputting an <xsl:attribute-set> node in the form of a CSS ruleset. This also seems to work fine, according to my testing. The next stage is to try executing all of the tests on the server under Cocoon, with the data in eXist.
There's no need for a template converting a CSS ruleset to an attribute-set, because the user will edit the properties and values in a GUI, and XUpdate will be used to make changes to the documents in the eXist db.
Recent work on the Mariage project has shown the most effective way to approach user-switching between styles on the site: we should use Cocoon's built-in cookie selector to set the path to XSLT and CSS files used. The exact details of this need to be worked out, though; given the limitations we have in doing includes in XSLT files (see previous postings), we won't have complete freedom to mix and match components; we may have to depend on relative paths to do that for us, so XSLT stylesheets may have to be organized into subcollections.
It turns out that the simplest way to do this is with XInclude. This is how it works:
Both XSLT files are uploaded into the database (with "Expand XIncludes turned OFF in the client -- this is important for reasons that will be clear later). The root XSLT file has an XInclude inside it, pointing to the file that should be included:
<xi:include href="globals.xsl#xpointer(//xsl:stylesheet/*)"></xi:include>
Note: this XInclude code is actually technically incorrect, but it looks this way because the XInclude implementation on the version of eXist we're running right now is non-conformant. This has been fixed in the eXist SVN, and when we upgrade to the next stable eXist version, we should be able to use the correct code, which would look like this:
<xi:include href="globals.xsl" xpointer="xpointer(//xsl:stylesheet/*)"></xi:include>
Now we invoke the transformation in the pipeline like this:
<map:match pattern="text/*.txt">
<map:generate src="xml/{1}.xml" />
<map:transform type="xslt" src="xmldb:exist:///db/teiJournal/xsl_trans/text/text_out.xsl" />
<map:serialize type="text" />
</map:match>
What happens is that Cocoon retrieves the XSLT from the database, and as it does that, eXist expands the XInclude on the fly, and a single stylesheet is created as input to the transformation. Thus the problem of one stylesheet referring to another is avoided.
In the system we envisage, a base transformation would include components from other stylesheets which are under the editable by the admins and editors of the journal, so the contents of the included file will change. This is why it's important not to expand xincludes when uploading the stylesheet in the first place: if we do that, a static copy of the original (unedited) inclusions will be permanently stored in the root stylesheet, so customizations will not have any effect.
Did some more work on the idea of storing XSLT in the database and still using it for transformation. It seems impractical to store the root XSLT file that will be used for the transformation in the database; things work better from the point of view of Cocoon pipelines if that's in the filesystem. However, that's not a major problem, because the kinds of things we want to store in the db are string variables and attribute sets, so they'd most likely be includes anyway. Therefore I've begun testing various ways to import a stylesheet into another stylesheet from the database. This combination of sitemap pipeline and import command works:
<map:match pattern="db/xsl_trans/**.xsl">
<map:generate src="xmldb:exist:///db/teiJournal/xsl_trans/{1}.xsl"/>
<map:serialize type="xml" />
</map:match>
<xsl:import href="http://localhost:8080/cocoon/projects/mholmes/teiJournal/db/xsl_trans/text/globals.xsl" />
The main stylesheet, accessed in the normal way, imports the second stylesheet through a localhost URL, which calls the pipeline, which then transmits the stylesheet from the db as XML.
The obvious disadvantage here is the hard-coded URL; although it's based on localhost, the remaining path structure is too specific. I'm now going to experiment with every variation of relative URL I can think of, to see if I can come up with a workaround.
Spent most of the day working out the DB design (which is half-documented in a diagram, which I'll post when I've finished it). Before going too far with it, though, I needed to check that it was possible to store XSLT in the database and use it in pipelines. This proved a bit complicated.
You can of course store an XSLT file in the db, because it's just XML. Similarly, you can retrieve it using a <map:generate> or <map:read> element (although the latter is no use for our purposes because it can't be used as input in another pipeline). Another pipeline can reference the XSLT file using the xmldb:/// protocol, and this appears to work OK (at least with the default xslt transformer; using Saxon, things don't quite work, but that's another story). However, any <xsl:include> or <xsl:import> elements in the stylesheet fail, because their relative path is reconstructed using the cocoon:/ protocol, and the XSLT transformer of course knows nothing about that.
We really do need to store XSLT files in the db, because all sorts of user preferences and options will be stored in the form of xslt attribute sets. Perhaps we can use XQuery to compile a complete stylesheet? That actually makes a lot of sense -- that'll be the next thing to figure out.
Spent half an hour playing with it, and found it's not going to be practical for what we want to do. The GUI is too slow -- no AJAX, just form submission, and wait, and eventually get a page back. It's going to be quicker to provide online templates ourselves, I think. Worth checking out EndNote first, though.
We have the OK to go ahead with this project, so I've created a blog for it. I've migrated all previous posts from the Depts blog over to this one (tedious process in phpMyAdmin -- needs a GUI!).
One of the major tasks in this post is to figure out how to link <cit>, <q>, <quote> and <bibl> tags to the xml:id attributes of <biblStruct> elements containing details of the cited work. The key attribute is not available on these tags, so we're left with three options:
ana: indicates one or more elements containing interpretations of the element on which the ana attribute appears.corresp points to elements that correspond to the current element in some way.n gives a number (or other label) for an element, which is not necessarily unique within the document.The first smacks of tag-abuse, because it's really meant to point to an analytical interpretation. The third is a multi-purpose attribute which is available almost everywhere. corresp seems most appropriate, especially for <bibl> tags (it's less suitable for the other tags, but it would do at a pinch).
Adding this as a task, because I need to search the TEI-L archives for any discussion of this before, then lay out my proposed solution (corresp and get comments.
I broke down the initial phase of the project into individual tasks and estimated hours for them, then wrote to Peter to see if he thinks it's worth going ahead with it.
Finished the document on bibliographical markup, made a couple of changes to the implementation document, and then made a start on the detailed timeline. In the process, Greg and I discussed and researched several issues:
Secure logon: this can only be handled by turning on SSL in Tomcat; this enables any page to be accessed through https. If the user does this, we can give them an option to make use of it in the application, whereby sensitive documents would only be rendered if their access URL was using https. The default install will have this turned off, because we can't control or detect the user's Tomcat setup.
Building a WAR file package: Greg has now mastered building Cocoon with eXist as a block, so we can go this route. We can turn on all the blocks we want and exclude all the flaky ones. The only remaining issue is how to inject our website materials into Cocoon, along with the sample database, after building. Perhaps we can just edit the contents of the WAR file (which is just a zip).
Spent most of the day working on the presentation on teiJournal, creating graphics, diagrams and so on, then paring it down to something that will go for 20 minutes. It could be prettier but it's to the point.
Wrote the outline/plan for the presentation. I'll flesh it out with text and graphics tomorrow.
As part of the teiJournal plan, and because it also helps with creating my presentation for Friday's Vancouver trip, I wrote the next part of the project description, the description of output formats. The idea of the user-configured ad-hoc anthology is beginning to get clearer.
One key aspect of teiJournal will be the ability of the editor to configure as much as possible of the presentation information through a user-friendly interface. There is so much to configure that the key problem is going to be organization and storage of this information. One way to think of this is as "style blocks", each of which controls a particular aspect of a particular presentation format.
Typical blocks would be: the default format of bibliographical info (Chicago?), layout of each format (e.g. running titles), the content of each format, and the abstracted and user-configurable aspects of each output type (font, style, page size and other choices, especially for PDF).
Style block code will find its way into two kinds of output: XSL:FO (for PDF), in which it will be inserted inline, and CSS, where it will control the display of predefined XHTML classes. The most convenient way to store this information will be as XSL Attribute Sets. Since these are XML, they can be stored directly into the database, and read easily. The data itself is going to be a combination of CSS and XSL:FO attributes and values; we need to come up with a template-based GUI for editing it.
Finally, there will be some types of style choice that do not translate into CSS or XSL attributes. For instance, the choice of MLA or Chicago formatting for bibliographies or references will be a choice between two blocks of rendering code. There may not be many of these.
Including time spent working this out so far.
Created a set of documents covering all the major document types outlined here, drawn from these sources:
We now have a document set that can be used to refine the detailed instructions for markup. Added a task for the next stage.
As part of the work to plan a generic teiJournal Web application, I began the process of creating some sample documents we can use to test and refine our schema and XSLT.
I started from a random document selected from the ACH/ALLC 2005 Abstracts project, and adapted some XSLT from the ScanCan P5 conversion. I generated a schema from ROMA, keeping it quite limited and eliminating a number of elements from some of the core modules. The results is (currently) only 300KB; I think it could be refined further, especially by eliminating some elements from the teiHeader which I'm not actually using, but I want to look more closely at them first; it seems that better options than <classCode> exist for categorizing different types of journal contribution (such as <category>).
Next, I worked on the XSLT. It's XSLT 1.0 (for maximum portability), and I found that making the TEI namespace the default ns for both XSLT files, and switching to Xalan for the transformer, eliminated all namespace problems (such as empty xmlns attributes). I got the transformation working for the first sample file, with only one significant exception. Bibliographical references for online resources were encoded using an <xptr> element, and its crdate attribute was used to encode the "last accessed" date, but crdate seems to have been eliminated from P5. Posted a message to TEI-L about this.
Next, I'll work on a larger group of documents from ACH, including images, tables etc. There is some significant value in doing this anyway; the ACH/ALLC Abstracts project will need to be ported from Mustard to Lettuce when Mustard is rebuilt, and in the process it will have to be updated to work with the new Cocoon/eXist stack. In the process, we might as well do P5 conversions for archive purposes, and make P5 the default XML view of the data. So work on this XSLT would have to be done at some point anyway, irrespective of teiJournal.
For the first half of the "Document types and features" task, I created a taxonomy of document types here. This is based on an analysis of a range of different journal types from the Current Periodicals section of the UVic library, and the Oxford Journals online site.
The next phase is to work out the divisions and components of a typical journal, and of each of the document types.
The first part of this task (enumeration and description of output formats) was completed on March 6. The second part has been spun off into a new task.
This is part of the detailed project plan process for the proposed teiJournal project.
teiJournal will produce a range of different output formats, including P5 XML (individual documents), P5 teiCorpus docs (volumes and the entire collection), PDFs (individual articles and volumes), and XHTML (abstracts, articles, TOCs, indexes, search interfaces etc.).
This task is to enumerate and describe in detail each of these formats, including (for instance) the default format of bibliographical info (Chicago?), layout of each format (e.g. running titles), the content of each format, and the abstracted and user-configurable aspects of each output type (font, style, page size and other choices, especially for PDF).
A tabular listing of all tags used, and all non-default attributes and non-arbitrary value sets is now complete, viewable as HTML and XML . This is a good start.
The next stage is to create a sample document including all of these features, but realized in P5. This should include a complete bibliography with examples of all the main item types. We should also look closely at the DHQ Tag Zoo document to see if we can identify weaknesses in our P5 document type, and any document features we haven't covered.
I'll create tasks for this.
The first phase of planning is to get a good idea of the range of tags, attributes and attribute values used in the two projects we've already done which will form the basis for teiJournal: the ACH Abstracts, and the ScanCan project.
The problem is that both projects use a relatively tightly-constrained subset of the TEI P4 DTD, but the project DTDs are large and inclusive because they were generated at the beginning of the process when it wasn't yet clear what elements and attributes would be used. In addition, many attribute values which are simply open string values in the DTD are used in a more constrained and methodical way (with a fixed set of values) in the actual projects. Therefore we need a way to get a clear overview of what elements and attributes are used, and particularly what values the latter actually have. Once we have this, we'll be able to map these directly to P5, and create a restricted P5 schema using ROMA to recreate the same basic document format. Following that, we can convert the ACH docs to P5 (which we need to do anyway), and we can write some detailed instructions for marking up articles, as the basis for all the transformations and other dependent code.
I spent this morning trying a variety of ways to get a useful report out of the old eXist db on Mustard, showing all the data we need. Because of the requirement to retrieve all and only distinct values for element and attribute names, and attribute values (very expensive in processing time), and because of the need to search across two different collections, this is a slow job.
I ended up with nearly 200 lines of XQuery and a query which is takes several minutes to execute, but the results look promising. I need to write some XSLT to produce useful output now, filtering out what's not necessary (id attributes, for instance) and presenting the results in a table.
This task has been set back a bit by other priorities, but I'm also breaking it down into a series of individual, more specific tasks.
Much back-and-forth by phone and email with Peter and Scott, culminating in a cautious decision that I should go ahead and write a formal project plan, including estimated timelines, with no final decision yet on whether the project will go ahead in the end. I'll build into the plan at least two or more stages where, if we should stop the project, we'd still have something useful.
Entering this as a task with a deadline next month.
The brief project description of the proposed teiJournal system is now complete. This will be circulated for Peter's and Scott's consideration, and I'll also send it to Barbara Sawhill (IALLT), who is interested in using such a system for the IALLT Journal.
Began work on early specifications for a project I'm calling teiJournal. The first specs doc is here. This needs to be completed by the end of tomorrow, and two more documents need to be added: