Maintenance

I've made the following changes to RVDB's Ant script:

• Inside the build directory (where the root build.xml is located), created a fop subdirectory, and inside that, a jars_replaced directory, where I'm going to store Cocoon jars that have been superceded by newer versions from FOP.
• Created a fop-jars.xsl file inside the fop directory, to process and update Cocoon's jars.xml file (modelled on exist-jars.xsl, which does the same thing).
• Created a new target in the Ant file:

  <target description="Replace FOP 0.95 with FOP 1.0" name="upgrade_fop">
        <echo message="-----------------------------------------------"/>
        <echo message="Upgrading FOP from 0.95 to 1.0             "/>
        <echo message="-----------------------------------------------"/>
  
        <echo message="-----------------------------------------------"/>
        <echo message="Patching jars.xml            "/>
        <echo message="-----------------------------------------------"/>
        
        <xslt-saxon in="${cocoon.home}/lib/jars.xml" style="fop/fop-jars.xsl" out="${cocoon.home}/lib/jars.xml.patched">
        </xslt-saxon>
        <move file="${cocoon.home}/lib/jars.xml.patched" tofile="${cocoon.home}/lib/jars.xml"/>
        
        <echo message="-----------------------------------------------"/>
        <echo message="Removing jars to be replaced        "/>
        <echo message="-----------------------------------------------"/>
        
        <move failonerror="false" file="${cocoon.home}/lib/optional/batik-all-1.6.jar" tofile="fop/jars_replaced/optional/batik-all-1.6.jar"/>
        <move failonerror="false" file="${cocoon.home}/lib/optional/fop-0.95.jar" tofile="fop/jars_replaced/optional/fop-0.95.jar"/>
        <move failonerror="false" file="${cocoon.home}/lib/optional/xmlgraphics-commons-1.3.1.jar" tofile="fop/jars_replaced/optional/xmlgraphics-commons-1.3.1.jar"/>
        
        <echo message="-----------------------------------------------"/>
        <echo message="Adding replacement jars and new required jars        "/>
        <echo message="-----------------------------------------------"/>
        
        <copy todir="${cocoon.home}/lib/optional" overwrite="true">     
          <fileset dir="${fop.home}/lib">
            <include name="batik-all-1.7.jar"/>
            <include name="xmlgraphics-commons-1.4.jar"/>
            <include name="serializer-2.7.0.jar"/>
          </fileset>
        </copy>
        
        <copy todir="${cocoon.home}/lib/optional" overwrite="true">     
          <fileset dir="${fop.home}/build">
            <include name="fop.jar"/>
          </fileset>
        </copy>
        
        <copy todir="${cocoon.home}/lib/endorsed" overwrite="true">     
          <fileset dir="${fop.home}/lib">
            <include name="xml-apis-ext-1.3.04.jar"/>
          </fileset>
        </copy>
        
      </target>

• Added that target to the main "prepare" target:

  <target description="Prepare the cocoon and eXist directories"
        name="prepare" depends="update_exist, copy_exist, prepare_cocoon, upgrade_fop, build_cocoon, copy_additional_jars, patch_cocoon">
      </target>
  

• Ran a build, which seemed to work fine, and confirmed that Cocoon and the eXist Webstart client still work. They do.

What I don't know yet is whether anything else works properly. This is what I'll have to do next:

• Check through the sitemap changes made in RVDB's transformations, and alter anything that doesn't match our existing projects. For instance, he changes the XHTML doctype to transitional; that's something we definitely don't want. There may also be changes in our original transformation missing from RVDB's.
• Identify, or create, a simple project that can be used to test all the functionality. The natural candidate is the IALLT Journal project, which is quite small, and uses FOP. However, it's not simple. It might be worth simply creating something straightforward for easier testing of future builds. We want to confirm that XQuery Generators, Saxon XSLT transformations, and FOP transformations all work.
• Test character encoding issues. This is particularly needed because of this comment at the head of patch_web_xml.xsl:

  <!-- change default encodings to UTF-8 -->
  <!-- except for servlet[servlet-name='cocoon']/init-param/param-name[. = 'container-encoding']:
    should stay ISO-8859-1, see 
    	-http://markmail.org/message/nm6bnvqztbee4s5o
    	-http://markmail.org/message/jt256gl3magir6g4
    	-http://wiki.apache.org/cocoon/RequestParameterEncoding#A3._Decoding_incoming_requests:_Servlet_Container
  -->  
  

  We had form submission working fine with both GET and POST, using all-UTF-8 settings; if possible, I'd prefer this, but it may be that something has changed in Cocoon 2.1.12x which screws this up.

I discovered that the problem I was having with RVDB's Ant script, which I've modified, was actually with Saxon 9.3, which for some reason was failing to do transformations properly. I've rolled back to Saxon 9.2, and now the build goes OK and the resulting webapp works fine. Next I need to integrate the FOP instructions into the Ant build.

RVDB sent a solution to my Ant build problem, which basically works -- the eXist build.sh is now found -- so I've integrated that, and I've also found a better solution to using Saxon for XSLT, that doesn't depend on its being on the CLASSPATH (which I want to avoid, because its version numbers are likely to change over time). I found the solution in one of the comments on this page, and it's basically this:

1. Define a macro for Saxon, specifying a way to call it with the parameters we need to use:

     <macrodef name="xslt-saxon">
       <attribute name="in"/>
       <attribute name="out"/>
       <attribute name="style"/>
       <sequential>
         <echo level="info">XSLT Generating @{out}</echo>
         <java classname="net.sf.saxon.Transform"
           classpath="${saxon.home}/saxon9he.jar"
           logError="true"
           output="@{out}"
           fork="true">
           <arg value="@{in}"/>
           <arg value="@{style}"/>
         </java>
       </sequential>
     </macrodef>
   

2. Call the macros when required like this:

   <xslt-saxon in="${cocoon.home}/build/webapp/WEB-INF/cocoon.xconf" 
               style="cocoon/patch_cocoon_xconf.xsl" 
               out="${cocoon.home}/build/webapp/WEB-INF/cocoon.xconf.patched">
   </xslt-saxon>
   

Now I have RVDB's whole build process working OK, but the resulting webapp is broken; XSLT transformations don't seem to work. I suspect this is something to do with the way Saxon is being set up in the sitemap, because I'm using a more recent version of Saxon (HE, 9.3, as opposed to the three-jar version 9). I'll work on this issue tomorrow.

More progress, and some regression:

• I can't get RVDB's Ant script to run; it dies early in the process claiming that it can't find a build.sh file which is obviously there. Rather than struggling with Ant, I've decided first to go back and work from my own script, and integrate RVDB's steps (the ones we want to keep) into that. Later, once I understand it all and it's working, I can think again about rewriting it as an Ant task.
• Both FOP and Cocoon 2.1.12dev are pretty static -- no changes for a long time -- so I'm not going to worry about checking out fresh copies of those as part of the script.
• I have my script working up to the point where the FOP Jars are copied over into the Cocoon lib directory, but I haven't yet implemented the required changes to the Cocoon jars.xml file yet. I'm looking at a range of changes RVDB is making through his exist-gump.xsl and exist-jars.xsl files, to patch gump.xml and jars.xml, and I think I'll have to take those and modify them so that they include the other changes required for FOP, which are documented here.

This is going to be a long slow process, but by the end of it I'll have a clearer idea of the changes that need to be done and the reasons for them. That will help when it comes to integrating the Analyzer patch.

Started my work to try and get a working build system for Cocoon 2.1.12 + FOP + eXist, based on RVDB's Ant script:

• First I had to get my head around Ant, which I haven't really used before. It isn't so complicated, but there's stuff I have to learn.
• I downloaded the FOP 1.0 stable release, and built it (with Ant), which is straightforward.
• I got the latest Saxon, in anticipation of using it for the XSLT in Ant tasks, and incorporating it into the Cocoon build. There's also a saxon9-ant.jar, which can be used for Ant tasks. RVDB's script doesn't use this; he relies on Saxon being on the classpath, and sets up his Ant task some other way, but I think I'll try to follow this explanation (the part on creating a <taskdef> with saxon9-ant.jar).
• I've downloaded a Cocoon 2.1.12 branch nightly, but I should really set this up through SVN so I can check out a new copy whenever we build; we do want to be able to include Cocoon bugfixes as well as those for eXist.
• I'm now working through and modifying RVDB's script, to make sure I understand everything -- how Ant works, as well as what modifications and patches he's doing. Ours will differ slightly. His documentation on how to build in FOP will have to be merged into the build script too, and that has a duplicate item in its task list; I'm not sure whether it's just a dupe or should be something else; I've written to check.

Logging time spent backing up various mySQL dbs (I do this regularly, but don't usually log the time).

As part of the Moses project search engine, I've been looking at the possibility of using a more sophisticated Analyzer for Lucene, to enable stemming in indexing and searching. There is a Lucene analyzer called the SnowballAnalyzer which does stemming for many languages, but unfortunately it can't be used directly in eXist because its constructor requires different parameters than the default constructor for an Analyzer that eXist is expecting. This post suggests a way around this by creating a wrapper class, but this seems a bit complicated, especially in contexts where we expect to rebuild eXist regularly.

However, this post refers to a package that contains stemming-capable analyzers that will work with eXist as it is. However, none of them appears to handle English. So for the moment, we're a bit stuck, but it may be that someone patches eXist so it can use the Snowball analyzer.

EDIT: It looks like the problem might be caused by a failed transform of the conf.xml file near the end of the process. I need to look at that transformation and see why it's failing.

This morning I cleaned out the eXist tree and ran our build script with a fresh checkout. eXist seemed to build successfully, but I had a number of errors which had to be fixed before I could get the build of Cocoon to complete:

• Cocoon was finding a duplicate entry for commons-pool-1.5.4.jar in its jars.xml file. I don't know what is causing this, although the eXist build script now seems to be adding the second copy, so in my exist-jars_supplementary.xsl file, which post-processes the Cocoon jars.xml file, I put in a section to remove the second copy.
• The entry for ws-commons-util-1.0.2.jar was not being added into jars.xml. I think this was due to an error in the <xsl:if> clause that was checking for it in my exist-jars_supplementary.xsl, so I fixed that, and the error went away -- don't know why it worked last year.
• Cocoon complained about the presence of commons-pool-1.5.3.jar alongside 1.5.4. I don't know how that file makes it into the build, since eXist now uses 1.5.4, but I added a line to the script to delete it.
• The build now worked fine, and Cocoon comes up, but there are two problems we haven't seen before: the Cocoon samples page fails, and the eXist client dies with an error. Going to the admin page, I see an error with conf.xml. Replacing the conf file with a known good one does no good, so something is obviously screwy here. I have a help request up on the eXist list -- hopefully someone can tell me more about the error.

Spent most of my day on this, but I'm still not having any success. It appears that some patches were committed for some of the issues our script works around, but either they don't work properly, or there are new issues introduced by later changes. RvdB has been in touch, and is also interested in getting a working build, so I think we'll keep hacking at this.

It's been a year since we built what seems to have been a remarkably stable Cocoon+eXist, and we've been using it for many projects. However, today, working on the Moses project, I came across what looks to me like a bug in the functioning of util:expand(), which is preventing me from making the search engine work exactly as I had hoped. So I've finally bitten the bullet, and I'm trying to use our Cocoon+eXist build script again. These are the hoops I've had to jump through so far:

• In the rebuild of my machine, a directory called build_files had unaccountably disappeared from my build tree. This contained all our key files. I restored it from Rutabaga.
• JAVA_HOME was not set on my machine. I set it to point to the Sun JRE, only to discover that I need the JDK, so I installed that.
• Next I got a Cocoon error due to a duplicate entry in the jars.xml file; I don't know why this error is suddenly appearing, but perhaps a new file got included into eXist which was previously already in Cocoon.
• However, the showstopper is a stack of errors that show up when building eXist itself, during the build of the Lucene extension. I can't see what's causing these, but tomorrow I'm going to do a full new checkout and try it all again.

Frustrating and annoying, but I guess necessary.

I've spent a while over the last few days working on HTML5, with the hope of developing a policy for HCMC with regard to its adoption and use. These are my conclusions so far:

• For a lot of the sites we develop, HTML5 will be a definite advantage, because of a variety of useful new features such as media support, navigation and menu tags, etc.
• HTML5 also enables us to avoid the difficulties associated with XHTML vs. HTML mime types, which have always afflicted us with XHTML 1.1, through the use of a polyglot document. This enables us to generate well-formed and valid XHTML5 code, but serve it with a text/html mime type if necessary. This has several important implications:
  • "Processing Instructions and the XML Declaration are both forbidden in polyglot markup", so we must dispense with the XML declaration and avoid processing instructions.
  • "When polyglot markup uses UTF-8, it must not include a BOM." We will always be using UTF-8, so we must never include BOMs.
  • The charset should probably be declared in the http header if possible (easy with PHP and with Cocoon). We should also include the meta <charset="utf-8"/> tag, although this is only meaningful in the context of HTML, not XML.
  • We should use the minimal doctype declaration: <!DOCTYPE html>.
  • The polyglot markup specification imposes a number of recommendations which we already tend to favour:
    • Elements which CANNOT contain content MUST be self-closing (<br/>).
    • Elements which CAN contain content must NOT be self-closing, even if empty (<p></p>).
    • Only the five core named entities (quote, ampersand, angle brackets, apostrophe) are used; all others must be numeric, and SHOULD be hexadecimal.
    Some of these requirements will be problematic for Cocoon, because its serializers typically self-close all empty tags if not bullied by injected comments. But HTML5 is already problematic, because there is no serializer in Cocoon 2.1 that can handle it. We're working on that now.

For new projects, or projects undergoing a rewrite, which have a short development horizon, this is the recommended approach:

• Create HTML5 polyglot documents, as above. See the linked article for more detail on the rules for namespaces, elements, etc.
• From among the available HTML5 elements and attributes, choose only those which are unchanged from XHTML 1.1, UNLESS there is a need to use a new HTML5 feature. This should allow older browsers to display pages unchanged, but will allow the inclusion of new HTML5 features in future without reworking the whole document. Here are simple lists of elements and attributes.
• Serve as text/html, allowing support for IE7 and 8.

For longer-term projects and those which will benefit significantly from HTML5 features, and for which it can be argued that support for older browsers (IE7 and 8) is not important, these are the recommendations:

• Again, create polyglot documents. I can see no reason why we should not do this. We have all the transformability of XML, should we need it, but we get reliable browser behaviour. If a compelling reason emerges for switching to application/xhtml+xml in the future, it's a simple change which will not require alteration of the actual documents.
• Wherever possible, use elements and attributes which already have broad browser support. There are two reasons for this: first, you can test your code on a wider variety of browsers, and identify any differences in behaviour which might indicate instability in the specification (still a big danger), and second, your collaborators will be able to test features without having to depend on a specific browser. Wikipedia has a good browser support matrix page.
• For video and audio, include both MP4/H.264 and Theora, or MP3 and OGG, versions of files on the site. Firefox and Opera support the open standards but not the closed, and vice versa with other browsers. See my earlier post on this.
• Where unusual character ranges are used on the page, or where a particular font is preferred for its display characteristics, use a WOFF file and block of CSS to make it available to the browser without the user needing to do anything:

  @font-face{
    font-family: "GentiumPlus";
    src: url(GentiumPlus-R.woff);
  }
  
  @font-face{
    font-family: "GentiumPlus";
    font-style: italic;
    src: url(GentiumPlus-I.woff);
  }
  

  Needless to say, make sure you have the right to distribute the font in this way.
• Use CSS3 features, but don't depend on them if they don't yet have broad support. I have not been able to find a well-maintained browser support matrix for CSS3, so trial and error is our best guide at this point, although the Acid3 test can help with this. After running the text, click on the A in the graphic to get detailed results.
• Use SVG with care and testing. It should be most useful for small images which function as part of a scalable GUI; browser support for animation and other features is less extensive, so will need more testing.
• Use MathML only if necessary. IE does not support it natively. In IE9 it has support through a plugin, but is known to be buggy.