FrancoToile

Throughout the day yesterday and this morning I made some significant progress on the development website, completing most of my tasks. All of the changes are documented in the Subversion log, but here's a rundown:

• Completed the Quicktime -> HTML5 video player conversion. While the player itself has been functional for a couple of weeks, the peripheral functionality - bookmarks, searching, etc. - has been added slowly. Bookmarks are now in and working with more or less the same functionality as in the old site, but with less Javascript.
• The behavior of the "Subtitle" tab has also changed to become a little more streamlined. Rather than opening up a separate tab with a "show/hide subtitles" option, the Subtitle tab link itself simply toggles the subtitles.
• The Google map has been moved from a development page to the main map page. The functionality of the map is complete (pending any changes CC wants to make); the map markers just need some styling magic from PS. I also removed the search column from the map as requested by CC.

I'm still working, with substantial help from MH, on improving the XQuery search query. The basic issue is that I'm having trouble formulating a search request that will only search utterances (and not any ref notes within the utterances) AND order by number of utterances found. I'll post the query here once I've found one that works. I may have to resort to ordering the results of the query in post-production.

While writing the function to grab an 'excerpt' of text from an utterance (i.e. a chunk of text surrounding a search term), I discovered that substr_replace() is not multibyte-safe. So, if you use it with a multibyte string you might get strange results.

The excerpt() function, which is otherwise very solid and flexible, is adapted from CakePHP's text helper. I had to replace these two lines:

$excerpt = substr_replace($excerpt, $ending, 0, $phraseLen);

$excerpt = substr_replace($excerpt, $ending, -$phraseLen);

With these multibyte-safe lines:

$excerpt = mb_substr($excerpt, 0, 0) . $ending . mb_substr($excerpt, $phraseLen + 1);

$excerpt = mb_substr($excerpt, 0, -$phraseLen) . $ending . mb_substr($excerpt, $textLen);

So, in a nutshell, here's how to convert your substr_replace() call to a multibyte version:

// Original call
$string = substr_replace($text, $replacement, $start, $length);

// MB-safe call
$string = mb_substr($text, 0, $start) . $replacement . mb_substr($text, $length);

I've migrated the new version of the website into a Subversion repository. Since there are multiple people editing the site, this'll eliminate the risk of overwriting changes, having things fall through the cracks, etc. The videos are not under version control but all other files within /dev are.

The repository is at: /svn/francotoile

So it turns out that the Google Maps API breaks when used on a page served up as XHTML (that is, sent with a content type of application/xhtml+xml). This is because the API itself uses document.write(), which isn't available in XHTML. While there's a workaround which involves using AJAX to load the map (this is a good discussion about it), the work required to change the Google maps PHP class that I'm using (view the class) would be significant. Instead, I found a script to emulate document.write() in XHTML. I'm not sure if this is the 'best' solution, but it's time-effective and doesn't break validation.

Martin, Greg and I spent quite a bit of time getting the video page (player.php) to validate as XHTML5. The validation started out as part of the debugging process while figuring out why the HTML5 video player wasn't displaying in Firefox.

So, first we changed the document to XHTML5:

• Changed the doctype to html
• Changed the content-type header to: application/xhtml+xml;charset=utf-8

But that was causing hitherto unnoticed character encoding problems to throw outright validation errors. The characters came from the XML data and were mostly French accent characters. The XML itself was fine, so I finally boiled it down to an incorrect usage of PHP's XSLTProcessor library by the PHP eXist database library (which we didn't write). Following a comment made on the PHP page for XSLTProcessor::__construct, I changed this:

$xml_result = $xslt->transformToDoc($dom);

To this:

$xml_result = $xslt->transformToXML($dom);

In other words, from a DOMDocument object to plain old XML string. The DOMDocument transformation was resulting in improper character conversion.

Switching to transformToXML worked, EXCEPT that now, self-closing tags defined in the XSLT (e.g. img) stylesheets weren't being closed when transformed into XHTML, even though they were closed in the XSL files (weird). The solution, after some trial and error, was to change this tag in the XSL file:

<xsl:output method="html" omit-xml-declaration="yes" /> 

To:

<xsl:output method="xml" omit-xml-declaration="yes" /> 

So, simply changing the output method from html to xml. And luckily, that did the trick. The page was validating as XHTML5! Yay! Except for one problem...

The OGV videos still weren't playing in Firefox. D'oh. Then something clicked in Greg's mind and he remembered a similar problem on another project that turned out to have something to do with MIME types. Lo and behold, Apache doesn't have an OGV mime type (at least, not in the version running on the Francotoile server). So, I added this to the .htaccess file (along with MP4 types for good measure):

AddType video/ogg .ogv
AddType video/mp4 .mp4
AddType video/x-m4v .m4v

And that was it - video now working in Firefox.

Greg and I met with CC and PS to check in on the progress since our last meeting (two weeks ago) and to discuss some refinements and changes to the website leading up to the April 20th deadline.

A number of tasks emerged from the meeting:

For PS:

• Remove the search sidebar from the map page. The map will simply be a way to 'browse' all the videos; searching will be done on its own page.
• Update the navigation so that the options are: Browse, Search, Help
• Work on validating the site in HTML5
• Minor UI tweaks throughout the site

For CC (and/or assistant):

• Write about us/help text
• Provide translation text throughout site where appropriate
• Decide on various "related videos" categories (nationality, age, gender, subject, etc.)

For Greg:

• Investigate TEI standards for nationality and residence
• Ensure, with CC, that the nationality and residence data follows the TEI standards

For Jamie:

• Investigate (and hopefully implement) Google Maps API for video map points
• Change format of video text search results so that, instead of a timestamp, the search term with ~2 words before and after is displayed
• Ensure the HTML5 video player works in Firefox, IE (8), Safari, and Chrome
• Finish HTML5 version of video captions

No further face-to-face meeting was scheduled, though we'll be in email contact with each other.

Due to certain browsers handling HTML5 differently, we also need MP4 versions of the movies in addition to OGG. So, I used HandBrake (specifically HandBrakeCLI) to batch convert all of the MOV files to MP4. Two commands this time - one to convert and one to rename:

find media -name *.mov | xargs -I file HandBrakeCLI -e x264 -q 20.0 --input file --output file.mp4

find media -name '*.mov.mp4' | xargs rename 's/\.mov.mp4$/\.mp4/'

Be sure to specify "-e x264" when using HandBrakeCLI or else it'll use ffmpeg.

Since we're switching from Quicktime to HTML5 for the video player, we need to have both OGG and MP4 versions of the movie files. To convert the MOV files to OGG I installed ffmpeg2theora and then ran it via the command line to find all of the movie files in the /media directory and convert as found:

find media -name *.mov -exec ffmpeg2theora --videoquality 6 {} \;

Finished the first pass of the dynamic 'related videos' sidebar, which displays a random list videos related to the one the user is currently watching. The code is contained within a findRelatedVideos() function to keep the player.php page decently clean. The function is fully documented and fairly simple to use. Though the function is currently displaying videos based on the <nationality> element within the <person> element, it can switch to any other element value within <person> simply by changing the function arguments.

Here's the function doc verbatim:

 * Retrieves a list of videos related to $id based on the contents of $field. 
 * The field - for example, 'nationality' or 'residence' - should be within the
 * <person> element in the video XML file. The default number of videos returned
 * is 5, but can be configured with the optional $options array. If the number
 * of related videos is greater than the number of videos to be returned, then
 * they're chosen randomly.
 *
 * The videos are transformed into <li> elements in includes/xslt/related.xslt.
 * 
 * Due to the structure of the system, the eXist $db object and the results of 
 * the XML query must also be passed.
 *
 * The function returns an associative array:
 *  - videos: the formatted video <li> elements, transformed via XSLT
 *  - field: the value of $field
 *
 * @param   string  $id         Video ID
 * @param   object  $db         DB object
 * @param   mixed   $xmlResult  Result of query from player.php
 * @param   string  $field      Name of related field within <person>
 * @param   array   $options    Optional settings                       OPTIONAL
 * @return  array   An array with the videos and the value of $field
 * @author  Jamie Nay
 * @date    2011-01-31
 */

Spent a bit of time this morning familiarizing myself with the structure of the web app and the eXist DB. As this is my first eXist project I also spent some time getting to know eXist and figuring out how to connect to the database with the client (thanks to Martin for the help). My first task is to implement a simple "related videos" feature on the sidebar of the player page, which involves querying the eXist database for other videos with the same nationality as the current video. I'll need to confirm with CC at our next meeting (Feb 2) whether the related videos should use 'nationality' or 'residence' for the search.