I began with the assumption that I could just pull in every linked reference into the XML of a specific document, and thus create complete coherent docs; but this is not the best approach, because references link to each other recursively in this project; one might very easily end up with the entire reference collection embedded in many documents. Therefore I've created an indexing system that builds an index to the 6,000+ items (references, biblio items and tile images) that are not explicitly linked because they're accessed through JS; and I've included an invisible link to that file in the footer of most of the front pages of the site. CD's crawler is now working on this version of the site, and we'll see if it does the job or not.