Today I finished the implementation of the GetRecord response, which is very substantial indeed. I then started working on ListIdentifiers, and got to the point where I was able to start testing the execution time of some queries. The results demonstrate that it's going to be entirely impractical to generate this data on-the-fly. We're going to have to generate it in advance and store it, in the OAI record format, and then run the OAI queries against that collection. So this is what I'm now planning to do:

• Create a collection called oai in the database.
• Create two collections inside it, one called meta and one called records.
• Inside meta, store a document called sets.xml, which contains the entire ListSets response.
• Also inside meta, store a document called identify.xml, which contains the entire Identify response.
• Finally, inside meta, store a document called metadataFormats.xml, which contains the entire ListMetadataFormats response.
• Inside records, store a generated full record for every correspondence document, possibly using the same xml:id attribute as on the original document.
• Take code from my existing oai.xq file, and create a new library which does the following:
  1. For each document in the correspondence collection:
  2. If there's no corresponding record document in the oai/records collection, create one;
  3. Otherwise, if there is one already, check the last-modified date of the document against that in the record file, and if it's later, delete the record file and create a new one.
• The initial task of creating records will be time and processor-heavy, so it might be done in the admin client in batches (by year, for instance). However, once it's done, an update pipeline can be created so I can call it on the server on a regular basis to update the metadata records.
• Rewrite my oai.xq library so that it handles all requests using the data in the oai collection.

I've already implemented the three documents inside the meta collection, and simplified my oai.xq accordingly. Now I have to generate the records, before I can start working on the query interface. It's pretty certain I'll have to use the resumptionToken functionality -- I'll perhaps feed out records in sets of (max) 100. I'm going to encode the entire request in the resumptionToken so that I don't have to cache the query or results; that'll be simpler, and will obviate the need to periodically clear out the data from the cache.

I've started on the update script, and I'm going use code like this example to store the documents.

This won't be finished until next year.