Mariage

Worked most of the day on the rendering of editorial articles (modern content). The tricky part has been figuring out how best to re-use a lot of the rendering functionality we already have (such as reference and footnote popups) while still separating out the article rendering so that it can be differentiated where necessary from the anthology content, without duplicating code. I have a new XSLT file called article_p5_to_xhtml.xsl, which is not long, but takes control of the processing at a crucial point, and a new rendering @mode attribute in the XSLT which can be used to fork the processing at various points. All documents still commence their processing with index.xsl, though, and inherit all the main site components at that point.

The basics:

• Articles are retrieved from the URL /article.htm?id=whatever.
• Bibliography items and lists which are fully defined in the original document are rendered visibly in the output, before the footnotes; they're also available as popups from links in the main body.
• Bibliography items which are referred to, but which reside in the main collection bibliography, are retrieved into the document but hidden; they are shown when the reference link is clicked on.
• Footnotes are rendered at the bottom of the document, but also shown as popups from buttons in the text.
• References should work like main-collection biblio items, although there are none in this article anyway.
• Links to anthology documents can go to the document, or to a page in the document (if there are pages with numbers captured by the appropriate tags), or to annotations in image markup documents.

We now have a new way to write reference links:

<ref target="../../back_matter/references/references.xml#aarschot">Aarschot</ref>

This makes your references actually valid in the directory/file tree. What it says is: "To find this item, go up the tree two levels (from texts_original to anthology to mariage) then down the tree to back_matter and then references, then find the file references.xml, and in that file, find the item with the xml:id "aarschot".

You will also be able to make a reference point to multiple locations, just by using a space to separate them, like this:

<ref target="../../back_matter/references/references.xml#montgibel ../../back_matter/references/references.xml#etna">Montgibel (Etna)</ref>

However, it's best not to do that quite yet, because I don't have the rendering code set up to deal with it.

This is what the database structure looks like now:

----mariage
    [mariage.rng, mariage_drama.rng]
    ----anthology
        ----images
        ----texts_original
    ----editorial
        ----articles
        ----intros
    ----back_matter
        ----references
        ----biblio
    ----translations
        ----texts_modern

ALL xml files are at the bottom level (two levels below the root). This means that when you link to RNG files, at the top of your document, you'll need to do it like this:

<?oxygen RNGSchema="../../mariage.rng" type="xml"?>
<?oasis-schema href="../../mariage.rng" type="application/xml"?>

This means "go two directories up the tree to find the file". ../ means "go up one directory", and two of them means go up two. The value of this is that we only have ONE copy of the schemas. This is important because at some point I'll be rebuilding the schemas, and I only want to replace them in one location, instead of dozens. Don't make any copies of the schemas anywhere else. If you try to validate your document and it doesn't find the schema, that means either your document is in the wrong place, or you don't have the right path (as above) in the head of your file.

I'm changing the Lettuce working directories (where the RAs edit) so that they mirror the db structure, and in the process I've been working through existing documents and fixing things.

• The old unfinished folder is now read-only.
• The following files from /satyres_bastards/ have been merged into the main db. I made some basic fixes to these files (header data was wrong, having been copied from existing files and not changed -- xml:id attributes, <idno> tags, titles, page numbers and so on were often wrong. Where the files were usable, they're now on the site. That includes these files: astrologue_cocu, elle_mesme, fantastique_repentir (replacing the old one, although I ported over one of CC's notes), hobreau_cocu, mary_cocu and premiere_nuit.
• One exception was Labyrinthe D'Amour, which seems to be just a copy of Fantastique Repentir. That has been left in place.
• When you're creating a new document, create it in the following place on Lettuce:

  /home1t/mariage/ccarlin/xml/working_files/exist_db_NEW_XML/mariage/anthology/texts_original

  Files in this folder will NOT be pushed up to the website.
• When you're happy with the file and ready to publish it, copy it to this location:

  /home1t/mariage/ccarlin/xml/exist_db/mariage/anthology/texts_original

  Any files in this location WILL be pushed up to the website automatically (every day or two).
• Once you have published a file, edit it in the PUBLISHED location (you may well want to delete it from the working_files tree, so you don't get confused).
• References files are now here:

  /home1t/mariage/ccarlin/xml/exist_db/mariage/back_matter/references

  Edit and save them in that location.
• ONLY XML files appear in the editing locations. Do not put any PDFs, doc files or other ancillary notes and references in those locations. Keep them somewhere in the working_files directory. You'll see that all your old PDFs, docs etc. have already been moved over to the working_files tree.

Made a lot more changes today, through sitemap, XSLT and XQuery:

• All URIs which return XHTML pages on the site now have ".htm" extensions.
• All URIs which return XML documents on the site now have ".xml" extensions.
• All URIs which return text documents on the site now have ".txt" extensions.
• All URIs which return document fragments through AJAX on the site have ".xq" extensions.
• Links between documents are now working (so you can link from an article. intro, note or reference to an anthology document).

This means that old links or bookmarks pointing to documents on the site will no longer work; you'll have to find the document through the menu and TOC, and then create a new link or bookmark.

Markup of the first of CC's articles is now done, and I'm beginning the rendering code for it. The document and footnotes show up just fine already, but the references to external docs aren't working yet; neither are the biblio items, although it looks as though they're all being retrieved.

I now have the retrieval of references by Ajax working. This is the deal:

• Some reference items link to other reference items. When we retrieve a document from the db, we automatically retrieve all references which are linked from the original document, and add them to the output page, so they can be shown as required; but we can't easily parse all the retrieved references to see if they have links to other references, because this kind of recursion is potentially infinite.
• Therefore the JavaScript showReferences() function does this:
  1. Check if the reference exists in the document. If it does, show it as normal.
  2. If it doesn't, then:
     1. Construct a URL to retrieve the reference, and set the targetEl element to the <div> that contains the current list of references. This means the Ajax function will append the retrieved element to the list of references in the document.
     2. Construct a string which calls the showReference() function again, and store that in the onAjaxComplete variable. The Ajax code, when it completes, checks whether this is null or not, and if it's not null, it calls it with eval().
     3. When the showReference() function is called a second time, the item now exists in the document, so it can be shown in the normal way.
     4. There's a trap for when the retrieval fails, to stop infinite loops.

Created a new version of the project (mariage_5) running locally, and did the first conversion of the db format (which differs slightly from the previous plan, to include the modern versions of the texts):

----mariage
    ----anthology
        ----images
        ----texts_original
    ----editorial
        ----articles
        ----intros
    ----back_matter
        ----references
        ----biblio
    ----translations
        ----texts_modern

I've also modified all of the documents so that all their validation links point to a single copy of the main schema in the root mariage folder (one, of course, points at the drama-enhanced version). At the end of this entire process, I'm planning to run oddbyexample and get a really trimmed-down schema.

Next, I normalized all the elements which point to external components (references and biblio items) so that they all have the same format, and don't rely on an @type attribute. Then I confirmed that everything that worked before now also works.

Now, before I go any further, I need to decide once and for all how I'm going to handle the display of this linked content in the site. My instinct goes like this:

• All items (biblio, ref or whatever) which are directly linked from a document are retrieved with that document, and made available for rendering as back matter for that document. Whether they're actually rendered or not will depend on the document type; biblio items in editorial articles should be rendered at the bottom of the page, but there's no point in rendering visibly all the references for e.g. Varin, so they can stay hidden in the output.
• All document links calling out to such items are rendered to call exactly the same function, showReference().
• The showReference() function works like this:
  1. Check if there's an item with the right id already in the document.
  2. If there is, then:
     1. Remove any existing content from the popup box.
     2. Clone the content of the item into the popup box (removing its @xml:id, to avoid duplicate ids).
     3. Show the popup box.
  3. If the item doesn't exist in the document, then it needs to be retrieved by AJAX:
     1. Store the id of the target item.
     2. Empty any existing content from the box. This may delete the item which was clicked-on to initiate the retrieval, so we'll need to be sure there isn't a problem there. We might have to use a timeout to trigger the second part of the process.
     3. Retrieve the item by AJAX and append it to the end of the back div (so there should always be a back <div>, in case we need to do that).
     4. Clone the item into the popup as above, and show the popup.

This allows us to handle situations in which a reference points to another reference, and also avoids retrieving the same content by AJAX more than once.

I already have the same popup code working for both footnotes and references, but I haven't yet finished the code which will retrieve a reference by AJAX.

This is my plan, along with XUpdate code, for transforming the wealth of different reference types we have into a more consistent scheme.

• First, we need to alter the db structure again. Because we want the consistency of encoding all references with relative paths, and we want to be able to use the same structure in any document, we need to rework the db structure itself, from this:

  ----mariage
      ----anthology
          ----images
          ----original_texts
      ----articles
      ----biblio
      ----intros
      ----references
  

  to this:

  ----mariage
      ----anthology
          ----images
          ----original_texts
      ----editorial
          ----articles
          ----intros
      ----back_matter
          ----references
          ----biblio
  

  This places all actual content documents at the same level of the hierarchy, so referencing can always go up two levels and then down two. Links from one ref tag to another inside the same document can maintain the same structure. This makes for consistency (ease for markup folks) and robustness (if files are split or merged at the bottom level, references will still work; just the filename will need to be changed).
• A standard reference popup needs to be converted from this:

  <ref type="reference" target="references.xml#etna">Mont Etna</ref>

  to this:

  <ref target="../../back_matter/references/references.xml#etna">Mont Etna</ref>

  This can be done using the following xupdate code:

  declare default element namespace "http://www.tei-c.org/ns/1.0";
  
  for $r in collection('/db/mariage')//TEI[@xml:id='varin']//ref[@type="reference"]
  let $newRef := <ref target="../../back_matter/references/{$r/@target}">{$r/(*|text())}</ref>
  return <done>
  {update replace $r with $newRef}
  </done>
  

  The corresponding XSLT code also needs to be changed in teiLinking.xsl, from this:

  <xsl:template match="//text//ref[@type='reference']">

  to this:

  <xsl:template match="//text//ref[contains(@target, '/references/')]">

• An external link (<ref target="http://...">) needs no change at all, either to XML or XSLT.
• A bibliographical reference needs to be changed from this:

  <bibl corresp="../biblio/biblio.xml#dejean_2003">

  to this:

  <ref target="../../back_matter/biblio/biblio.xml#dejean_2003">

  These have not yet really been handled in XSLT properly, but the basic rule would be that XQuery would go about retrieving all items from the biblio.xml file and putting them into the back tag of the article XML, alongside any locally-defined biblio items, and they would then be rendered out to the page at the bottom of the article (and made available as popups too). I've already made this change in the only article so far marked up (and not yet finished), and in the toc intro that's also making use of it.
• Multi-targetted round-trip references such as those in the early documents will have to be changed from this:

  <ref xml:id="wifes_lover" target="amant.xml#wifes_lover a_une_femme_mariee.xml#wifes_lover">

  to this:

  <ref xml:id="wifes_lover" target="../../anthology/images/amant.xml#wifes_lover ../../anthology/original_texts/a_une_femme_mariee.xml#wifes_lover">
  

  The @xml:id attribute will obviously have to be preserved here. One wrinkle here is that the components of the target element do not currently specify the subcollection in which the target document is to be found. There are actually only 11 of these items, so they might as well be changed manually. XSLT code which renders them into links will also have to be changed, but see below; this case can be conflated with the next.
• In articles, what I was doing provisionally like this:

  <ref target="xhtml.xq?id=grande_destruction">

  will now be done like this:

  <ref target="../../anthology/grande_destruction.xml">

  These items can now be handled exactly the same way as the items above: first of all, check whether they link has multiple targets; if not, then render a simple link to the XHTML view of the document; if so, create whatever output structure is most elegant for creating multiple links; in both cases, write the @xml:id attribute out as an @id attribute in the XHTML in case incoming links target it.

I think this is a complete plan, and it's basically ready to go; it'll take a day or two to implement it properly, and during that time I need to make sure that no-one is editing existing documents. Next week would be a good -- no-one from the project will be working.

These were getting in the way of note popups, so they need to be closable. Added a little closer, although this is a stopgap measure pending my reorganization of all popups and linking.