HCMC Blogs

A fair bit of admin to do with RA and WS timesheets, due to changes in schedule for Christmas, and booking new schedules for next semester.

Submitted the job description for posting using the online service.

URL to log in to the UVic HR's website to manage the job posting for the temporary programmer is
https://uhire.uvic.ca/cgi-bin/WebObjects/uhire.woa/wa/admin

In case that URL fails:
go to http://web.uvic.ca/hr/
click on "Recruitment Tools" in the "Recruitment" section in the list on the left, which takes you to http://web.uvic.ca/hr/employment/recruitmentlinks.html
On that page, I click on the "Administrators online requisition for staff (Management Excluded, PEA, CUPE 951, CUPE 917 and Exempt Support)" link in the "Forms" section, which takes you to the authentication page for the recruitment service.

Stewart has access to that service and has delegated to Martin and Greg, so they should have access too.

On late duty, and entangled in a long hard battle with Cocoon and FOP over fonts and Unicode.

RE and co tried following the procedures outlined in my previous post yesterday, to enable Apache to talk to Tomcat over UTF-8, but the final stage in the Apache config screwed up all our virtual host mappings, so that's no good. I went back and looked at the problem again, and determined that the only issue remaining was the quicksearch feature on the Mariage site, and this was because it used a straightforward form submission. I ended up coding a workaround. What I do is to use a JavaScript function instead of a straight form submission. The JS constructs a URL, which is then encoded with encodeURIComponent (this hexes the UTF-8 octets); then as the search page loads, more JS parses the search string and decodes the results back to UTF-8, which it can then submit to the regular search.

This seems to be working, so we'll stick with it for the moment. If any more problems show up with sites proxied through Apache, we might have to revisit this, but I'm not inclined to take risks with the virtual host sites; they're our most public and important, in some cases.

On to the last category now...

Today I expanded the test site so that it covers tests of Unicode characters and ft:query searches with match-tagging. These work fine. Then I started working on the FOP configuration, to make fonts available to FOP through a portable configuration in the sitemap, following my own instructions here. However, various things have changed. First of all, to generate the font-metrics files, you need slightly different jar file names. Change to the cocoon/build/webapp/WEB-INF/lib directory, and then issue something like this:

java -cp fop.jar:xercesImpl-2.9.1.jar:xml-apis-1.3.04.jar org.apache.fop.fonts.apps.TTFReader /home/mholmes/cocoon_with_exist/testsite/fop-fonts/GentiumPlus-R.ttf /home/mholmes/cocoon_with_exist/testsite/fop-fonts/GentiumPlus-R.ttf.xml

where the first is the path to the TTF, and the second the path to the metric file you're creating. I did this for the two Gentium Plus fonts, and four base DejaVu Sans fonts.

Next, I created this customized fo2pdf serializer definition in the sitemap:

<map:components>
    <map:serializers>
      <map:serializer logger="sitemap.serializer.fo2pdf" mime-type="application/pdf" name="fo2pdf_custom" src="org.apache.cocoon.serialization.FOPSerializer">
        <user-config>cocoon://testsite/fop-config.xml</user-config>
      </map:serializer>
    </map:serializers>

This tells FOP to use a specific config file, which is delivered from a pipeline elsewhere in the sitemap. That pipeline looks like this:

    <map:match pattern="fop-config.xml">
      <map:generate src="fop-fonts/fop-config-src.xml" />
      <map:transform type="saxon" src="fop-config.xsl">
        <map:parameter name="fontPath" value="{realpath:/}testsite/" />
      </map:transform>
      <map:serialize type="xml"/>	
    </map:match>

It gets its source from fop-config-src.xml, which looks like this:

<fop version="1.0">
  <renderers>
    <renderer mime="application/pdf">

  <fonts>
   
    <font metrics-url="fop-fonts/GentiumPlus-I.ttf.xml" 
      kerning="yes" embed-url="fop-fonts/GentiumPlus-I.ttf">
      <font-triplet name="GentiumPlus" style="italic" weight="normal"/>
    </font>
    <font metrics-url="fop-fonts/GentiumPlus-R.ttf.xml" 
      kerning="yes" embed-url="fop-fonts/GentiumPlus-R.ttf">
      <font-triplet name="GentiumPlus" style="normal" weight="normal"/>
    </font>
    
    <font metrics-url="fop-fonts/DejaVuSans.ttf.xml" 
      kerning="yes" embed-url="fop-fonts/DejaVuSans.ttf">
      <font-triplet name="DejaVuSans" style="normal" weight="normal"/>
    </font>
    
    <font metrics-url="fop-fonts/DejaVuSans-Bold.ttf.xml" 
      kerning="yes" embed-url="fop-fonts/DejaVuSans-Bold.ttf">
      <font-triplet name="DejaVuSans" style="normal" weight="bold"/>
    </font>
    
    <font metrics-url="fop-fonts/DejaVuSans-Oblique.ttf.xml" 
      kerning="yes" embed-url="fop-fonts/DejaVuSans-Oblique.ttf">
      <font-triplet name="DejaVuSans" style="italic" weight="normal"/>
    </font>
    
    <font metrics-url="fop-fonts/DejaVuSans-BoldOblique.ttf.xml" 
      kerning="yes" embed-url="fop-fonts/DejaVuSans-BoldOblique.ttf">
      <font-triplet name="DejaVuSans" style="italic" weight="bold"/>
    </font>
    
  </fonts>
      
    </renderer>
  </renderers>
</fop>

That source is transformed into the actual config file using this XSLT:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  exclude-result-prefixes="xs xd"
  version="2.0">
  
  <xsl:param name="fontPath" />
  
  <!-- On Windows, Cocoon's {realpath:/} omits the trailing slash. Since we 
    know where to expect it, we should be able to fix this. -->
  <xsl:variable name="fixedFontPath" select="if (contains($fontPath, '\')) then replace(replace($fontPath, 'testsite', '/testsite'), '/', '\\') else $fontPath" />
  
  <!-- XSLT Template to copy anything, priority="-1" -->
  
  <xsl:template match="@*|node()|text()|comment()|processing-instruction()" priority="-1">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()|text()|comment()|processing-instruction()"/>
    </xsl:copy>
  </xsl:template>
  
  <!-- Massage the path attributes. -->
  <xsl:template match="@metrics-url">
    <xsl:attribute name="metrics-url"><xsl:value-of select="$fixedFontPath"/><xsl:value-of select="."/></xsl:attribute>
  </xsl:template>    
  <xsl:template match="@embed-url">
    <xsl:attribute name="embed-url"><xsl:value-of select="$fixedFontPath"/><xsl:value-of select="."/></xsl:attribute>
  </xsl:template>  
  
</xsl:stylesheet>

Note: the structure of this file has changed considerably for the new FOP 1.0. The reason we use this setup instead of a hard-coded fop-config.xml is that we want our projects to be entirely portable without configuration changes; this system fills in the required hard-coded paths to the fonts directory on the fly, using Cocoon's realpath module, so it effectively makes those paths relative in the source XML file.

This actually works, in that I can use the GentiumPlus font family to render Russian characters (which will not render with the default fonts). One gotcha: I didn't realize how important it is to restart the servlet (Cocoon, or better, Tomcat); just making changes to the sitemap and other files seems to have little effect without this, so perhaps Cocoon reads and caches the config file for FOP on startup (or FOP gets started up when Cocoon is started, and reads it then).

I've also set up my test setup so that:

• The built webapp is symbolic-linked from the Tomcat webapps directory, so I actually don't need to copy it into Tomcat.
• The testsite code is symbolic-linked from inside the built webapp, so I don't need to copy it into the webapp. Obviously I'll have to have the Ant build recreate this link at the end of the build. I haven't done that yet.
• To make the tests work, you still have to manually upload some stuff into the database through the client. This is OK, really, as it's a test of the client, but it would be cool if this could be done somehow from the Ant build as well.

Next steps:

• Create a collection of fonts and font-metrics files that cover all the ranges we care about (we need something for CJK, and perhaps also Aboriginal Sans). We need a licence-free redistributable font array that will cover all our needs. For any given project, it would be trimmed down, of course.
• Enhance the Ant build script as above.
• Run another test build and archive the results.
• Start testing real sites inside the new build.

I've refined the build a bit to add a couple of things we need, and modified some of RVDB's preferences to match our own; there will probably be some more of this work to do. I've also written a tiny test site that we can use to quickly confirm that everything is working. And it is working! We now have what looks like a reliable build script that can be run any time. We should run it regularly (once a month?) and archive the builds, so that we have the potential to roll things back if a future build goes bad. Next steps:

• Archive this build.
• Add steps to the script for copying the build webapp to my local Tomcat, copying the test site into it, and running Tomcat.
• Add tests to the test site for accented characters (display, submission from forms, and submission through GET), to ensure that UTF-8 encoding is working.
• Test the new build with full working sites (Mariage and Moses). The latter is a good test of indexing and Lucene search with Unicode.
• Look at the Analyzer patch, and see if it works.

A further 817 page-images have been added to the manuscript image browser, covering British North America correspondence from 1858 (Individuals, N-Z). Transcriptions are now being linked into these images.

To access taporshare on the MacOS:
command-k to open the dialogue
URI - smb://taporshare.tapor.uvic.ca/<sharename>
Username - uvic\<netlinkId>
If you forget the "uvic\" in the username field the authentication will fail.

Investigation and testing related to the container-encoding setting in the new Cocoon build process led me to discover a bug that's currently affecting sites on Pear's Tomcat-dev when accessed through Apache. Here's an illustration of the problem:

If you go to the Mariage site search page on Pear, accessed on its Tomcat port, and search for "mariée", you'll get correct results. However, if you access the site through Apache and the virtual domain and do the same search, you'll get garbled results.

The problem seems to be this:

We build our recent Cocoon stacks as all-UTF-8, and set up Tomcat as well to use UTF-8, but it appears that the last stage in the process, when Apache talks to Tomcat, is not working in UTF-8. We've done a bit of research, and based on this page:

Two things may need to be changed:

• The AJP connector in Tomcat's conf/server.xml file may need to be tweaked to add a URIEncoding="UTF-8" parameter:

      <!-- Define an AJP 1.3 Connector on port 8009 -->
      <Connector port="8019" protocol="AJP/1.3" redirectPort="8081" />
  

  changed to:

      <!-- Define an AJP 1.3 Connector on port 8009 -->
      <Connector port="8019" protocol="AJP/1.3" redirectPort="8081" URIEncoding="UTF-8" />

• This needs to be added to the Apache configuration:

  JkOptions +ForwardURICompatUnparsed

For the moment, this only applies to Tomcat-dev; Pear's Tomcat-stable is running legacy projects which operate in 8859-1 encoding, and they're working fine.

Wrote to sysadmin to request that they look at this and see if it makes sense.