Maintenance

Moved the last few machines off the 128 subnet, and did the inevitable tidying up, including troubleshooting chou's inexplicable behaviour (not bringing up apache httpd on reboot - thanks, Michael J).
DNS was a bit sluggish as well, so we didn't really know whether things had turned out for about 2 hours.
All is well now. Chou and rutabaga have returned, and I've removed the node for king-henry completely. If ETCL brings it back online I'll re-instate the node.

Half way through a rebuild of a Windows box to add to the pool.
Cruft cleanup, user removal, updates, AV install etc. has been done.
Physical set up, network adjustments and final tweaks still outstanding.

Ran the process again, and nothing new was checked out from SVN, so it was building exactly the same tree as before (which worked). This version has the broken link. Next, I tried running ./build.sh clean on the eXist tree before running the process again. This one also failed, so build clean doesn't help. Finally, I tried another full delete of the entire eXist folder and ran it all again. That worked. So the only system that actually worked was to delete the eXist tree before starting. That's what I'll do from now on.

This morning I did a checkout (without deleting the existing tree) and a build; the JNLP link failed, and required that the cocoon app directory be inserted into it to get a working link. However, immediately afterwards I deleted the source tree and did a fresh checkout, and it worked correctly. It's clear that the file controller-config.xml is nothing to do with the problem, because that file is identical in the broken and working versions. It seems that deleting the source tree solves the problem. Now we need to see whether a simple build clean will solve it. More experiments continuing.

Did a fresh checkout and build of eXist this morning, after deleting my source tree, and it worked fine: the resulting web app was functional in every detail (so I saved it, just in case). Did another checkout at the end of the day, and the JNLP link on the eXist page didn't work -- but inserting the name of the web application's directory into it gives you a working link. It seems that a change to the source in the repository triggered this problem, and the suspect file is /exist_from_svn/eXist/webapp/WEB-INF/controller-config.xml, which had been changed during the day. So now we have a suspect. I've stashed a copy of it; if and when a checkout DOES work correctly, we can compare the two, and report the exact bug (if they're different in a way that suggests they are the cause).

We've now written XSLT to patch the four config files for Cocoon and exist automatically, and that seems to be working, as well as using patch to patch the build.properties file. The whole automated system is running very well, but the results currently don't have a working JNLP -- don't know why, but perhaps something in the trunk has changed which has broken it. I'll try a fresh build with a completely new checkout tomorrow. This is really tiring, but we'll eventually get there...

I finally figured this out. This is the sequence to make a working Cocoon webapp:

1. Checkout eXist from SVN.
2. Copy our local.build.properties file into the Cocoon tree (this turns off the Lucene and querybean in Cocoon, per instructions from WM.
3. Copy our own build.properties file into the eXist tree (this contains the path to the Cocoon tree on our system).
4. Sign the eXist jars (required for the JNLP client to work):
   ./build.sh -f build/scripts/jarsigner.xml
5. Do a complete build of eXist (required to generate a conf.xml):
   ./build.sh all
6. Build eXist for Cocoon:
   ./build.sh -f build/scripts/cocoon/build.xml
7. Clean the Cocoon build tree using the Cocoon build.sh:
   ./build.sh clean
8. Rebuild Cocoon:
   ./build.sh
9. Copy missing jar files:
   ws-commons-util-1.0.2.jar
sunxacml-1.2.jar
   from EXIST_HOME/lib/core into the generated
   COCOON/build/webapp/WEB-INF/lib
10. Copy some custom jars of our own, xqSearchUtils.jar and TitleSortComparator, along with SaxonHE.jar, into the Cocoon lib directory -- not part of the core process.

This gives us Cocoon 2.1.11 with eXist as a block, with a working admin interface and a working JNLP client. Eventually, I suppose step 9 will be redundant as those jars will be handled along with the others as part of the normal build.

Meanwhile, these steps are also going to be needed to really refine the process:

1. Abstract hard-coded paths from the script.
2. Patch the root sitemap in Cocoon to add the Saxon 9 XSLT 2 generator, and a couple of other generators we like to use.
3. Figure out the implications of the build source and build target settings. In my successful build this morning, I built with 1.5, 1.5, although the deployment environment will be 1.6.

Greg and I have been working on setting up an automated system for building eXist from SVN trunk and then making a Cocoon package. The script I've got is basically working (assuming the SVN checkout is not broken, which it sometimes is), but it does involve some patching of files which we've learned about by trial and error, and the Cocoon webapp that it builds doesn't seem to successfully start eXist up. This is a painful process, but if in the end we manage to make it work, we'll be able to file bug reports and patches that get into the trunk, making it much easier to build a Cocoon block version in future, and we'll also gain enough understanding of the process to debug future problem. If we can easily build a distro from the SVN trunk, we can be regularly testing our projects on future versions and preparing them for migration, which is going to be essential. Meanwhile, the old version stumbles along, requiring two or three restarts every day...