Maintenance

Today I blew up a couple of the apps and had to restart them, through doing this the wrong way. When you have a new XAR to deploy:

1. Use Chrom*, not FF.
2. Connect over the internal URL on :8080.
3. Upload the new package.
4. If it goes wrong and you see an error message, the chances are the db is now set to read-only.
5. If that happens, try shutting down the db from the web interface. If that works, restart it from /etc/init.d/jetty. If it fails, you may need to kill all the relevant processes on Peach before restarting.

With these big XARs, we may need to consider testing an alternative process where we uninstall the old XAR and then put the new one in the autodeploy folder before restarting eXist.

We've hit a problem with the normal situation where we're running eXist listening on a different domain from localhost. To summarize:

If eXist is running on the default localhost:8080, I can use startup.sh and shutdown.sh to start and stop it.

If it's listening instead on another ip name, I can still start it with startup.sh, but I can't stop it with shutdown.sh or with start.jar directly. I get:

ERROR: Failed to create input stream: Unexpected end of file from server
CAUSE: Failed to create input stream: Unexpected end of file from server

Disowning the process in the terminal where it was started makes no difference. Supplying credentials also makes no difference. Waiting to see if anyone on the eXist list has any ideas.

The Mariage XAR is large, and when I try to deploy it to a running eXist from Firefox, it dies after a couple of minutes with an anonymous "error". This must be due to some file size constraint in Firefox; logging it here, along with the workaround (which is to use Chromium instead), and if I have the chance to research the reason I'll post it.

The eXist team tagged 3.1.1, so I've rebuilt our template from that tag, tested it, and pushed it to the existDeployer folder on home1t.

~/.local/share/applications/mimeapps.list

Our MySQL server was down ("too many connections"); spent some time reporting, investigating, and fielding and responding to queries from users. In the end a restart fixed it.

Tested a build of the dev branch with my script and deployment stuff locally; all good, and the bug with the java client is fixed.

Added a new transform of webapp/WEB-INF/web.xml to hide the REST interface from the public. Tested and working.

I've successfully configured my local machine for testing purposes so that eXists can be configured and run alongside each other, available individually on port 8080, and proxied through apache on port 80. The details of the eXist build and configuration are best discovered in my scripts in the repo, but the local configuration details are:

In /etc/hosts:

127.0.2.1	graves2-internal.hcmc.uvic.ca
127.0.3.1	mapoflondon6-internal.hcmc.uvic.ca
127.0.4.1	mariage6-internal.hcmc.uvic.ca

In /etc/apache2/sites-enabled/graves2.conf (one example for graves2, others analogous):

<VirtualHost 127.0.2.1:80>
	ServerAdmin webmaster@localhost
	ServerName graves2-internal.hcmc.uvic.ca
	ServerAlias graves2
	ProxyRequests Off
	<Proxy *>
		Order deny,allow
		Allow from all
	</Proxy>
	ProxyPreserveHost on
	ProxyPass / http://graves2-internal.hcmc.uvic.ca:8080/ nocanon
        ProxyPassReverse / http://graves2-internal.hcmc.uvic.ca:8080
        AllowEncodedSlashes NoDecode
</VirtualHost>

<VirtualHost [YOUR_REAL_IP_ADDRESS]:80>
	ServerAdmin webmaster@localhost
	ServerName graves2-internal.hcmc.uvic.ca
	ServerAlias graves2
	ProxyRequests Off
	<Proxy *>
		Order deny,allow
		Allow from all
	</Proxy>
	ProxyPreserveHost on
	ProxyPass / http://graves2-internal.hcmc.uvic.ca:8080/ nocanon
        ProxyPassReverse / http://graves2-internal.hcmc.uvic.ca:8080
        AllowEncodedSlashes NoDecode
</VirtualHost>

<VirtualHost 127.0.2.1:443>
        ServerAdmin webmaster@localhost
        ServerName graves2-internal.hcmc.uvic.ca
        ServerAlias test
        SSLEngine on
        SSLCertificateFile /etc/ssl/certs/spud.crt
        SSLCertificateKeyFile /etc/ssl/private/spud.key
        ProxyRequests Off
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>
        ProxyPreserveHost on
        ProxyPass / http://graves2-internal.hcmc.uvic.ca:8080/ nocanon
        ProxyPassReverse / http://graves2-internal.hcmc.uvic.ca:8080
        AllowEncodedSlashes NoDecode
</VirtualHost>

The one with an external ip address is optional, but needed if you want external hosts to be able to access the apps (based on their own hosts file being configured to point these domains at your ip). The third stanza allows HTTPS access; in fact Apache provides encryption, using the specified cert, but when it talks to eXist, it does that over an insecure local connection. If you try to have Apache from eXist's 8443 encrypted port for this, you'll get an error because eXist doesn't have a cert set up. You can still use the full :8443 port to access eXist directly over an encrypted connection. In both cases when using https, your browser will complain about a self-signed cert, of course.

With this approach, I've had Mariage, MoEML and Graves all running side-by-side on my machine. I've also finished and commented the script to roll out new eXists on Peach, and I'm waiting for RE to look it over before we actually test it. eXist 3.1 is out today and I built the latest dist from that tag.

This has been a relatively long process to figure out how best to configure a Jetty/eXist instance to run happily alongside others, on a test domain, and how to test that setup. This is what I've done:

• Install apache locally from the repos.
• Install mod_jk from the repos.
• Turn on SSL (sudo a2enmod ssl) and set up a self-signed cert (lots of docs on this available).
• Set up test domains in the local hosts file:

  127.0.0.1	localhost
  127.0.1.1	spud
  127.0.2.1	test-internal.hcmc.uvic.ca
  127.0.3.1	moeml-internal.hcmc.uvic.ca
  

• Set up virtual domains in Apache -- example sites-enabled/test.conf:

  <VirtualHost 127.0.2.1:80>
  	ServerAdmin webmaster@localhost
  	ServerName test-internal.hcmc.uvic.ca
  	ServerAlias test
  	ProxyRequests Off
  	<Proxy *>
  		Order deny,allow
  		Allow from all
  	</Proxy>
  	ProxyPreserveHost on
  	ProxyPass / http://test-internal.hcmc.uvic.ca:8080/ nocanon
          ProxyPassReverse / http://test-internal.hcmc.uvic.ca:8080
          AllowEncodedSlashes NoDecode
  </VirtualHost>
  
  <VirtualHost 127.0.2.1:443>
          ServerAdmin webmaster@localhost
          ServerName test-internal.hcmc.uvic.ca
          ServerAlias test
          SSLEngine on
          SSLCertificateFile /etc/ssl/certs/spud.crt
          SSLCertificateKeyFile /etc/ssl/private/spud.key
          ProxyRequests Off
          <Proxy *>
                  Order deny,allow
                  Allow from all
          </Proxy>
          ProxyPreserveHost on
          ProxyPass / http://test-internal.hcmc.uvic.ca:8080/ nocanon
          ProxyPassReverse / http://test-internal.hcmc.uvic.ca:8080
          AllowEncodedSlashes NoDecode
  </VirtualHost>
  

• In these four files in the Jetty instance:

  tools/jetty/etc/jetty-http.xml
  tools/jetty/etc/jetty-ssl.xml
  tools/jetty/etc/standalone-jetty-http.xml
  tools/jetty/etc/standalone-jetty-ssl.xml
  

  change the <Set name="host"> to <Set name="host">test-internal.hcmc.uvic.ca</Set>. (I think only the first two matter for our purposes, but it does no harm to change the others.)
• Start the Jetty instance, and restart apache. Access the jetty app on test-internal.hcmc.uvic.ca.

I still have to test this with a second Jetty running side-by-side on a different domain; I'll do that tomorrow.