FrancoToile

Went through the process (with help from MH) of building a simple index using the Lucene text search engine for the FrancoToile rebuild:

<?xml version="1.0" encoding="UTF-8"?>
<collection xmlns="http://exist-db.org/collection-config/1.0" xmlns:tei="http://www.tei-c.org/ns/1.0">
    <index>
        <lucene>
            <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
            <analyzer id="ws" class="org.apache.lucene.analysis.WhitespaceAnalyzer"/>
            <text qname="tei:body" analyzer="ws">
                <ignore qname="tei:note"/>
            </text>
        </lucene>
    </index>
</collection>

The important notes are:

• You need to declare the namespaces you use in your documents in the <collection> element (e.g. xmlns:tei above)
• Use the full element names with namespaces when adding the indexes (e.g. tei:body)

Used these Exist docs as guides:

• http://demo.exist-db.org/exist/kwic.xml
• http://demo.exist-db.org/exist/lucene.xml

Met with PS and CC on May 11 to discuss the state of FrancoToile. All systems go; no bugs or issues to report. The completion of full English/French translations for all text on the site is ongoing. Other than that, no outstanding issues were noted.

Starting an exploratory rewrite of FrancoToile with a pure eXist 1.5 architecture. Learning eXist along the way. Should be interesting. Not anticipating to be done for quite a while since this is my first-ever eXist site, and my first use of XQuery beyond simple functions in XSLT stylesheets.

Fixed the search functions - both the standalone search page and the 'search inside this lecture' feature - so that annotation text is no longer searched. Also changed the format of the ajax search results so that they match the results of the global search: ordered list with the excerpt of text surrounding the search term.

The structure of the utterances for the FrancoToile transcripts looks something like this:

<u>
    Text of an utterance here...
</u>
<u>
    Some more text here. <ref type="info">Special keyword <note>with an annotation</note></ref> here.
</u>

Each utterance is within a <u> element. Some of the words and phrases in utterances are marked by annotations. These special phrases are inside <ref> elements, and the annotations to go along with them inside <note> elements.

When searching the utterances for a keyword, I needed a way to exclude all text within <note> elements from the search, since they're not part of the actual utterance text. Martin and I spent a long (long!) time coming up with a good solution, but couldn't find anything satisfying. Then, thanks to Stack Overflow, I finally found found something that works:

//textNodeToSearch//text()[not(ancestor::note) and contains(., "searchTerm")]

Phew. This will search your text node (whatever you use for textNodeToSearch) for the search term but exclude all <note> elements from the search.

The complete XQuery used in the FrancoToile search, which orders the results by number of utterances found and also returns those utterances, is:

declare namespace tei = "http://www.tei-c.org/ns/1.0";
declare namespace exist = "http://exist.sourceforge.net/NS/exist";

for $result in collection('francotoile/data')//tei:TEI[tei:text/tei:body//text()[not(ancestor::note) and contains(., "searchTerm")]]

return 
<match> 
    {
    let $articleBody := $result//tei:body
    let $id := $result//tei:TEI/@xml:id
    let $articleTitle := $result//tei:titleStmt/tei:title
    let $timeline := $result//tei:TEI/tei:text/tei:body/tei:timeline
    return
    <info>
        <title>{data($articleTitle)}</title>
        <refid>{data($id)}</refid>
	<count>{text:match-count($result)}</count>
	<timeline>{data($timeline)}</timeline>
    </info>
    }
    <utterances>
        {
        for $utter in $result//tei:u
	    let $start := $result//tei:timeline/tei:when[@xml:id=$utter/@start]/@absolute
	    let $end := $result//tei:timeline/tei:when[@xml:id=$utter/@end]/@absolute
	    where matches($utter, 'searchTerm') 
            return
	    <utterance>
	        <start>{data($start)}</start>
		<end>{data($end)}</end>
		<text>{data($utter)}</text>
            </utterance>
	}
    </utterances>
</match>

Made the following changes today from my to-do list from the last meeting (earlier today, April 07/11):

• Moved the 'dev' site to http://francotoile.uvic.ca
• Made off-site links in transcript annotations open in a new window
• Fixed the "Browse" link from the help page so that it goes to the correct map page
• Removed timestamps from the search results page and replace with a simple ordered list of results (#1, #2, etc.)
• Removed superfluous controls from the Google map
• Added a "No related videos" message when there are no matches found for a video's "related" sidebar

The old version of FrancoToile has been replaced with the new: http://francotoile.uvic.ca

CC, PS and myself met this morning to discuss the state of the website and any changes that need to be done before CC presents the site at CALICO. Our next meeting is scheduled for Monday, May 9th at 10:00 AM. Our task list before that meeting looks like this:

Jamie will:

• Replace the old version of the website with the "dev" site, so that http://francotoile.uvic.ca points to the new version
• Make off-site links in transcript annotations open in a new window
• Investigate and fix "pulsating" utterance hover boxes
• Fix the "Browse" link from the help page so that it goes to the correct map page
• Remove timestamps from the search results page and replace with a simple ordered list of results (#1, #2, etc.)
• Ensure that search queries do not search annotations within transcripts
• Remove superfluous controls from the Google map
• Add a "No related videos" message when there are no matches found for a video's "related" sidebar

Pat will:

• Change the size and location of the UVic and HCMC logos on the Help page
• Make a new 'Reel' favicon
• Add a copyright notice to the site-wide footer

Fixed a longstanding bug with the Google map that was causing Firefox to freeze on page load. It looks like Firefox can't handle this line (which loads the Javascript API file) in the <body>; it needs to be in the <head>:

<script type='text/javascript' src='http://maps.google.com/maps/api/js?sensor=false'></script>

Not sure exactly why Firefox freezes when the script is loaded in the body. Ideas welcome.

Quirky little gotcha when using the mb_* suite of PHP functions, such as mb_substr(), with UTF-8 data. If you don't explicitly set the encoding to UTF-8, then unicode characters will be chopped up incorrectly if handled by a mb_* function which is expecting a two-byte character rather than a three-byte character. Found this out the hard way when constructing the excerpt() function for the Francotoile search results page. So, before using the mb_* functions with your UTF-8 data, remember to set the encoding:

mb_internal_encoding("UTF-8");