Today we were confronted with a PermGen space error on Pear, which took some sorting out. I'm documenting it here in case it happens again, and we might be able to figure out what cause it.

I was using the eXist admin client to upload four XML files into the Moses project db, and I got errors in the admin client. This resulted in a corrupted database, such that I wasn't even able to delete the problem files or the collection containing them, making the web application useless. Going to the Tomcat manager to restart it, I found Tomcat had become unresponsive, and other projects gradually seemed to die as we tested them. I tried using the tomcat stopdev / tomcat startdev script to get everything going again, but Tomcat never came back up. Two tomcat-dev processes were still running, so it had obviously failed to shut down properly. As user hcmc (under which tomcat runs), we were able to kill -9 those processes, and restart Tomcat, and the other projects came back up, but the Moses db was still borked. Rather than try to clean it up manually, which can take time, I just stopped it in the Tomcat manager, deleted the whole project, and uploaded my local copy, then restarted it from the Tomcat manager. This worked fine, and after that, I was able to successfully upload the four files that had caused the problem in the first place.

Moral of the story: have a good working local copy of your project ready to go. Also, in the case of PermGen space errors, make sure all Tomcat processes have actually stopped before you try restarting Tomcat; if necessary, kill -9 them.