HCMC Blogs

For text documents, I'm now displaying the content of the bibl tags in the sourceDesc part of the header. These are the "teasers" currently displayed when you mouseover an item in the TOC. They're currently too short to be useful, in most cases, so we have to decide whether to:

• just expand the ones we have, and see them also expand in the TOC; or
• create a new type of teaser which is longer and intended for the head of the document, and differentiate it from the teaser;

We also have to decide whether or not we want to do something similar with the images. That's problematic because space is at a premium where the images are concerned, so we might not want to insert a static block into the page.

Finally, we should decide whether we want to give access to the entire <teiHeader> information, and if so, how. We could add a "Details..." button to the little info box that would expand it and show all that information.

I was intending to generate and supply a complete list of duplicate ids in the db, but it turns out that there are 1355 of them, so it's not really helpful to list them. I've sent the full list to SML and ECH. However, here are the first hundred:

• mix: affix.xml, m.xml
• kaʔ: affix.xml, k.xml
• kiyˀ: affix.xml, n.xml
• ɬəm: affix.xml, t.xml
• maʔ: affix.xml, m.xml
• nas: affix.xml, k.xml
• sal: affix.xml, n.xml
• t: affix.xml, pron.xml
• taʔ: affix.xml, n.xml
• wap: affix.xml, ww-glot.xml
• xit: affix.xml, x.xml
• cʼalˀ: c-glot.xml, k.xml
• cʼalˀən: c-glot.xml, k.xml
• cʼəl: c-glot.xml, k.xml
• sn̩cʼaʔqatkW: c-glot.xml, rescued.xml
• cʼax: c-glot.xml, k.xml
• cʼaʔx: c-glot.xml, k.xml
• kcʼaʔxmenən: c-glot.xml, k.xml
• cʼalˀ_2: c-glot.xml, n.xml
• kən_nacʼalˀsən: c-glot.xml, n.xml
• skacʼcʼalˀ: c-glot.xml, k.xml
• n̩cʼəlˀcʼalˀsən: c-glot.xml, n.xml
• cʼaɬ: c-glot.xml, k.xml
• nacʼaɬən: c-glot.xml, n.xml
• n̩cʼcʼaɬn̩: c-glot.xml, n.xml
• ncʼcʼaɬənˀtxW: c-glot.xml, rescued.xml
• kcʼaʔɬn̩čut: c-glot.xml, k.xml
• sqʼəlˀnaskint: c-glot.xml, k.xml
• cʼař: c-glot.xml, n.xml
• cʼařt: c-glot.xml, k.xml
• cʼawˀ: c-glot.xml, k.xml
• sascʼawˀoxW_sqəlawʔ: c-glot.xml, q.xml
• snacʼawʔsən: c-glot.xml, n.xml
• nacʼawˀəlqWpm̩: c-glot.xml, n.xml
• nacʼawˀɬcʼaʔ: c-glot.xml, n.xml
• nacʼəwmən: c-glot.xml, n.xml
• neʔcʼawpqən: c-glot.xml, n.xml
• kacʼawˀəloptn̩: c-glot.xml, k.xml
• ʔawˀt: c-glot.xml, glottal.xml
• cʼow: c-glot.xml, n.xml
• cʼək: c-glot.xml, k.xml
• ncʼkʼcʼkʼax̣ən: c-glot.xml, n.xml
• cʼəkW: c-glot.xml, n.xml
• n̩cʼkWopsən: c-glot.xml, rescued.xml
• kcʼkWicʼaʔən: c-glot.xml, k.xml
• cʼəkʼW: c-glot.xml, k.xml
• kčˀəkʼWxən: c-glot.xml, k.xml
• cʼəl_2: c-glot.xml, k.xml
• necʼəlot: c-glot.xml, n.xml
• cʼəlˀ: c-glot.xml, n.xml
• cʼəlˀxW: c-glot.xml, n.xml
• cʼəɬ: c-glot.xml, k.xml
• cʼəɬt: c-glot.xml, k.xml
• n̩cʼaʔɬstonən: c-glot.xml, rescued.xml
• kɬcʼmˀosənc: c-glot.xml, rescued.xml
• kɬcʼəmcʼəmtwaxW: c-glot.xml, rescued.xml
• cʼən: c-glot.xml, n.xml
• nacʼə̣np: c-glot.xml, n.xml
• cʼəpq: c-glot.xml, n.xml
• neʔcʼəpq: c-glot.xml, n.xml
• cʼəpʼqʼ: c-glot.xml, n.xml
• kcʼəpʼqʼən: c-glot.xml, k.xml
• n̩cʼəpʼqʼsalos: c-glot.xml, n.xml
• cʼəqʼ: c-glot.xml, k.xml
• ncʼqʼaɬcʼaʔən: c-glot.xml, rescued.xml
• n̩cʼqʼosən: c-glot.xml, rescued.xml
• cʼəsən: c-glot.xml, k.xml
• katcʼsatkWən: c-glot.xml, k.xml
• cʼəxW: c-glot.xml, k.xml
• cʼəxWən: c-glot.xml, k.xml
• ka·cʼəxW: c-glot.xml, k.xml
• na·cʼəxWoxW: c-glot.xml, n.xml
• kcʼxWanaʔn: c-glot.xml, k.xml
• kcʼxWus: c-glot.xml, k.xml
• kcʼxWosč: c-glot.xml, k.xml
• snecʼxWawˀsoxW: c-glot.xml, rescued.xml
• sn̩cʼəxWoxWwel: c-glot.xml, rescued.xml
• skacʼacʼəxW: c-glot.xml, k.xml
• kɬn̩cʼxWapəntaʔ_t_šawɬkW: c-glot.xml, k.xml
• sxWskcʼx̣Wapl̥aʔəm: c-glot.xml, k.xml
• neʔcʼikos: c-glot.xml, n.xml
• cʼikʼ: c-glot.xml, k.xml
• nacʼepʼsmən: c-glot.xml, n.xml
• nacʼipʼcʼipʼšəm: c-glot.xml, n.xml
• cʼex̣WoxW: c-glot.xml, k.xml
• cʼqʼ: c-glot.xml, n.xml
• kʼɬcʼaqʼWonˀən: c-glot.xml, rescued.xml
• cʼaʔqʼWonˀəm: c-glot.xml, rescued.xml
• n̩cʼoʔqa·pasən: c-glot.xml, rescued.xml
• n̩cʼowˀqulˀoxWən: c-glot.xml, rescued.xml
• nacʼopkW: c-glot.xml, n.xml
• niʔcʼuqʼWuʔšən: c-glot.xml, rescued.xml
• leyən_t_ʔencʼoqʼWmaʔ: c-glot.xml, l.xml
• cʼos: c-glot.xml, k.xml
• scʼosəm: c-glot.xml, k.xml
• kcʼosəmtən: c-glot.xml, k.xml
• cʼxW: c-glot.xml, k.xml
• cəs: c-rtr.xml, k.xml
• ʔa: glottal.xml, particles.xml
• ʔacʼx̣: glottal.xml, n.xml

This XQuery will generate them (although it's off-the-cuff and doubtless much slower than it could be):

declare default element namespace "http://www.tei-c.org/ns/1.0";
declare namespace f="http://exist-db.org/f-functions";
declare namespace util="http://exist-db.org/xquery/util";


declare function f:getIds() as xs:string*
{
let $e := collection('/db/moses')//entry
    for $id in distinct-values($e/@xml:id[string-length(.) gt 0])
	where count($e[@xml:id = $id]) gt 1
	return xs:string($id)
};

declare variable $ids := f:getIds();

for $i in $ids
return concat($i, ': ', util:document-name(collection('/db/moses')//entry[@xml:id=$i][1]), ', ', util:document-name(collection('/db/moses')//entry[@xml:id=$i][2]))

As a partial response to SMK's concerns about the search engine, I've defined the <gloss> tag as "inline" in the context of <entry>, <def>, <sense> and <seg>, in the index configuration file. This means that it does not constitute a word-break for the purpose of indexing, so that if you search for "dropped", you'll now find instances of <gloss>drop</gloss>ped. A natural by-product, though, is that you'll no longer find this item if you search for "drop". That's the trade-off. Without a working English stemming analyzer for eXist (and there doesn't seem to be one at the moment), there's no way to have your cake and eat it, unfortunately.

Separations etc. done, and proof to JT for checking.

Martin and I have been discussing what the search engine looks for. I'm posting our discussion here for reference, because I know I will be confused about it again in the future!

SMK:

The search engine doesn't always find a string within a longer string. For example, some, but not all, instances of "dropped" come up if I search for "drop". I'm not sure why this would be, if it's searching in all text fields. Or does it only find "dropped" if there is a <gloss type="u"> included with "drop"?

MDH:

It searches for words, so if it's looking for "drop", it won't find
"dropped". You don't want to search for "bed" and find "bedazzled".

Currently, the search uses Lucene's StandardAnalyzer to tokenize the
text, which means that it tokenizes on word-boundaries, and does no
stemming.

SMK:

I see. There are a couple of cases in the result set for "drop"
where "drop" is highlighted within "dropped". I'm guessing this is
because it's gloss-tagged? If this is the case, won't we have
effectively stemmed everything on the English side once we have
edited and gloss-tagged the rest of the entries?

MDH:

If you have this:

<gloss>drop</gloss>ped

then it would see the tag as a word-boundary, and find "drop". That's not a bad thing, I suppose.

SMK:

Yeah, so when we have gloss-tagged all the entries, most cases of a given stem, like "drop" will be split off from their affixes. So someone might search for "dropped" and not get any hits - or perhaps only the entries where "dropped" is in a dicteg. And then the user would have to be clever enough to work backwards and realize he should be searching for "drop" too.

As part of the Moses project search engine, I've been looking at the possibility of using a more sophisticated Analyzer for Lucene, to enable stemming in indexing and searching. There is a Lucene analyzer called the SnowballAnalyzer which does stemming for many languages, but unfortunately it can't be used directly in eXist because its constructor requires different parameters than the default constructor for an Analyzer that eXist is expecting. This post suggests a way around this by creating a wrapper class, but this seems a bit complicated, especially in contexts where we expect to rebuild eXist regularly.

However, this post refers to a package that contains stemming-capable analyzers that will work with eXist as it is. However, none of them appears to handle English. So for the moment, we're a bit stuck, but it may be that someone patches eXist so it can use the Snowball analyzer.

Added PB tags to 17 files from 1858, that required CO 6/27 images to do so.

UPDATE: I noticed that I had some pretty big arrays being pumped out, and the global settings I was using were no where near enough. I did some research and discovered that I can set a session variable using PHP. So, I can remove the global setting completely.
In my global config file I do:
mysql_query("SET SESSION group_concat_max_len = 20480");
and I'm sorted out. I can adjust this when necessary and it won't affect anyone else.

I noticed that MySQL views that use group_concat may have truncated output. Specifically, the output is cropped at 341 characters (specifically varchar(341).
It turns out that MySQL has a global variable called group_concat_max_len and it's set to 1024 bytes by default.
A bit of trial and error showed that if I set the max_len to 8192 I get an non-truncated result. Here's the command:
SET GLOBAL group_concat_max_len = 8192

Output now shows as a longtext field.

** However, queries that produce HUGE output (like finding all of the characters associated with the place "Nowhere" still truncates. I'm leaving the GLOBAL at 8KB, though, as the exception is an extreme one and is not actually possible for end-users to request.

EDIT: It looks like the problem might be caused by a failed transform of the conf.xml file near the end of the process. I need to look at that transformation and see why it's failing.

This morning I cleaned out the eXist tree and ran our build script with a fresh checkout. eXist seemed to build successfully, but I had a number of errors which had to be fixed before I could get the build of Cocoon to complete:

• Cocoon was finding a duplicate entry for commons-pool-1.5.4.jar in its jars.xml file. I don't know what is causing this, although the eXist build script now seems to be adding the second copy, so in my exist-jars_supplementary.xsl file, which post-processes the Cocoon jars.xml file, I put in a section to remove the second copy.
• The entry for ws-commons-util-1.0.2.jar was not being added into jars.xml. I think this was due to an error in the <xsl:if> clause that was checking for it in my exist-jars_supplementary.xsl, so I fixed that, and the error went away -- don't know why it worked last year.
• Cocoon complained about the presence of commons-pool-1.5.3.jar alongside 1.5.4. I don't know how that file makes it into the build, since eXist now uses 1.5.4, but I added a line to the script to delete it.
• The build now worked fine, and Cocoon comes up, but there are two problems we haven't seen before: the Cocoon samples page fails, and the eXist client dies with an error. Going to the admin page, I see an error with conf.xml. Replacing the conf file with a known good one does no good, so something is obviously screwy here. I have a help request up on the eXist list -- hopefully someone can tell me more about the error.

Spent most of my day on this, but I'm still not having any success. It appears that some patches were committed for some of the issues our script works around, but either they don't work properly, or there are new issues introduced by later changes. RvdB has been in touch, and is also interested in getting a working build, so I think we'll keep hacking at this.

This is volume 3 of the 1858 material, covering British North America 1858 offices: General and Individuals A-E. There are 1421 new page-images.