HCMC Blogs

Christmas party plus leaving early.

I've re-focused on the task of generating and storing the OAI records in the database, in such a way that they can be updated easily whenever the db contents change. I've written a library called oai_update.xq, which has the original record-generating code from my first attempt, but massaged a bit so that it uses explicit namespace prefixes for TEI; this is necessary because we need to generate the record fragments in no namespace, so it's easier if we don't have a default one. I also fixed a couple of bugs which emerged when I tested my code on the whole 7000+ documents. This is what it does:

• For each record in the correspondence collection, it checks whether there's an OAI record.
• If there isn't, it generates one.
• If there is, it compares the modified date on the OAI record against that of the original correspondence record, and if the former is older, it deletes it and generates a new one.

This is what it's not yet doing:

• Removing OAI documents for any correspondence documents that no longer exist (occasionally we remove a document when we find a duplicate). This will be fairly easy to do.

As I write this, I'm generating a set of OAI records for the whole up-to-date collection on my local copy of the machine. In the new year, I should be able to dump those and upload them into the live db to pre-populate it. Then I can add the feature above, and then write sitemap pipelines for the operations and add them to my set of periodic update operation tasks. Finally, I can then finish the OAI interface, which should be much simpler, since it'susing existing records instead of querying source data and constructing records.

Reminder to self: the OAI docs are here.

find . \( -name '.DS_Store' -o -name '._*' -o -name '*~' \) -delete

Note to self: this is a simple, tested way of storing a document in the db:

declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace  xdb="http://exist-db.org/xquery/xmldb";
declare namespace util="http://exist-db.org/xquery/util";

let $doc := <doc><test>My test doc</test></doc>,
$coll := collection('/db/coldesp/oai/records/')
return xdb:store($coll, 'test.xml', $doc)

This will delete a document:

return xdb:remove('/db/coldesp/oai/records/', 'test.xml')

This snippet will delete a document if it exists, then replace it:

let $remove := 
	if (fn:doc('/db/coldesp/oai/records/test.xml')) then
           xdb:remove('/db/coldesp/oai/records/', 'test.xml')
	else (),
$doc := <doc><test>My test doc 2</test></doc>,
$coll := collection('/db/coldesp/oai/records/')
return
xdb:store($coll, 'test.xml', $doc)

Ran over time trying to get stuff finished before the holidays (timesheets, lab bookings for RAs, Wheelock Latin exercises...)

Units 22-40 have now been rebuilt with macrons and keypads where appropriate, and are being uploaded to the website.

Today I finished the implementation of the GetRecord response, which is very substantial indeed. I then started working on ListIdentifiers, and got to the point where I was able to start testing the execution time of some queries. The results demonstrate that it's going to be entirely impractical to generate this data on-the-fly. We're going to have to generate it in advance and store it, in the OAI record format, and then run the OAI queries against that collection. So this is what I'm now planning to do:

• Create a collection called oai in the database.
• Create two collections inside it, one called meta and one called records.
• Inside meta, store a document called sets.xml, which contains the entire ListSets response.
• Also inside meta, store a document called identify.xml, which contains the entire Identify response.
• Finally, inside meta, store a document called metadataFormats.xml, which contains the entire ListMetadataFormats response.
• Inside records, store a generated full record for every correspondence document, possibly using the same xml:id attribute as on the original document.
• Take code from my existing oai.xq file, and create a new library which does the following:
  1. For each document in the correspondence collection:
  2. If there's no corresponding record document in the oai/records collection, create one;
  3. Otherwise, if there is one already, check the last-modified date of the document against that in the record file, and if it's later, delete the record file and create a new one.
• The initial task of creating records will be time and processor-heavy, so it might be done in the admin client in batches (by year, for instance). However, once it's done, an update pipeline can be created so I can call it on the server on a regular basis to update the metadata records.
• Rewrite my oai.xq library so that it handles all requests using the data in the oai collection.

I've already implemented the three documents inside the meta collection, and simplified my oai.xq accordingly. Now I have to generate the records, before I can start working on the query interface. It's pretty certain I'll have to use the resumptionToken functionality -- I'll perhaps feed out records in sets of (max) 100. I'm going to encode the entire request in the resumptionToken so that I don't have to cache the query or results; that'll be simpler, and will obviate the need to periodically clear out the data from the cache.

I've started on the update script, and I'm going use code like this example to store the documents.

This won't be finished until next year.

On late duty.

Progress so far:

• Implemented default responses for all six verbs, mostly with default info and placeholders.
• Fully implemented the ListSets response, which has quite a range of sets.
• Implemented the core of GetRecord, which calls out to a getDocRecord() function that does the real work.
• Implemented about half of the getDocRecord() function; it can return a header (fulfilling the needs of ListRecords), and I'm just getting started on the Dublin Core metadata output in the <metadata> tag.
• Fully implemented ListMetadataFormats (easy, because we're only supporting oai_dc).
• Fully implemented Identify.

One thing I'm currently undecided on is whether I should bother with the resumptionToken functionality. If I do, that means I'll have to cache the parameters of the request in the db somewhere and retrieve them in response to the token, which is a bit of a pain; I'm more inclined to let the whole thing run, and only worry about the resumption token if it seems likely that the results will be too large to handle.

A further 983 page-images have been added to the manuscript image browser, covering British North America correspondence from 1859 (Individuals). Transcriptions are now being linked into these images. CO 6 vol 30 is close to completion too.