Maintenance

In order to avoid bottlenecks I rebuilt an old machine and set it up as an apt mirror of ubuntu, virtualbox, google chrome and google earth repos. Here's how:
Install apt-mirror 'sudo apt-get install apt-mirror'
Configure apt-mirror at /etc/apt/mirror.list

My mirror.list looks like this:
############# config ##################
#
set base_path /mnt/repo/apt-mirror
#
set mirror_path $base_path/mirror
set skel_path $base_path/skel
set var_path $base_path/var
set cleanscript $var_path/clean.sh
set defaultarch amd64
set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads 20
set _tilde 0
#
############# end config ##############

deb http://mirror.peer1.net/ubuntu precise main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-security main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-updates main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-proposed main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-backports main restricted universe multiverse
# if you want to use the mirror as an install base (using, say, the mini.iso image) you need to mirror the debian-installer directories.
deb http://mirror.peer1.net/ubuntu precise main main/debian-installer restricted restricted/debian-installer universe universe/debian-installer multiverse multiverse/debian-installer
deb http://archive.canonical.com/ubuntu precise partner
deb http://extras.ubuntu.com/ubuntu precise main

deb http://download.virtualbox.org/virtualbox/debian precise contrib

deb http://dl.google.com/linux/earth/deb/ stable main
deb http://dl.google.com/linux/chrome/deb/ stable main

#### Multiarch (oneiric and newer) mirrors need to contain both architectures ####
deb-i386 http://mirror.peer1.net/ubuntu precise main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-security main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-updates main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-proposed main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-backports main restricted universe multiverse

deb-i386 http://archive.canonical.com/ubuntu precise partner
deb-i386 http://extras.ubuntu.com/ubuntu precise main

deb-i386 http://download.virtualbox.org/virtualbox/debian precise contrib

deb-i386 http://dl.google.com/linux/earth/deb/ stable main
deb-i386 http://dl.google.com/linux/chrome/deb/ stable main

clean http://mirror.peer1.net/ubuntu
clean http://archive.canonical.com/ubuntu
clean http://extras.ubuntu.com/ubuntu
clean http://download.virtualbox.org.virtualbox/debian
clean http://dl.google.com/linux/earth/deb/
clean http://dl.google.com/linux/chrome/deb/

Notice that I'm mirroring i386 trees even though I only build 64bit machines - as of oneiric (11.10) Ubuntu has gone to a multiarch system, so this is no longer optional.

Note that in order to configure the mirroring of repos that uses gpg keys (like virtualbox and google), you need to add the key to the apt-mirror machine. Do it like this:
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add - &&
wget -q -O - http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc | sudo apt-key add - &&

I want it to autoupdate every day, so I edited /etc/cron.d/apt-mirror (uncommented the cronjob line in the file and pointed the log where I wanted to store them).

Each time the mirror job runs it tries to run the postmirror script whether you uncomment the path to it in the mirror.list or not. Oddly, the script doesn't ship with the apt-mirror package, so I had to create it myself. I put it where the mirror.list expects it - in this case at /mnt/repo/apt-mirror/var/postmirror.sh) - and put this in the script itself:
/mnt/repo/apt-mirror/var/clean.sh
This ensures that the cleanup script gets run (it doesn't seem to run automatically).

The apt-mirror requires 'mirror', 'var' and 'skel' directories, so if you're setting this up on a separate drive you'll need to add them on the storage drive 'sudo mkdir /mnt/repo/apt-mirror /mnt/repo/apt-mirror/var /mnt/repo/apt-mirror/skel'. Note that the script wants to run as the apt-mirror user, so we need to change the ownership on those directories 'sudo chown -R apt-mirror:apt-mirror /mnt/repo/apt-mirror'.

I need it readily accessible to clients, so I installed apache 'sudo apt-get install apache2' and added symlinks to the appropriate directories e.g. 'ln -s /mnt/storage/mirror/ftp.heanet.ie/pub/ubuntu/ /var/www/ubuntu' - do that for each repo you're mirroring. I made them like this:
sudo ln -s /mnt/repo/apt-mirror/mirror/archive.canonical.com/ubuntu /var/www/canonical
sudo ln -s /mnt/repo/apt-mirror/mirror/extras.ubuntu.com/ubuntu /var/www/extras
sudo ln -s /mnt/repo/apt-mirror/mirror/dl.google.com/linux /var/www/google
sudo ln -s /mnt/repo/apt-mirror/mirror/mirror.peer1.net/ubuntu /var/www/ubuntu
sudo ln -s /mnt/repo/apt-mirror/mirror/download.virtualbox.org/virtualbox/debian /var/www/virtualbox

Now that everything is configured, go ahead and fill up the new repo. Remember to run the job as the apt-mirror user 'sudo su - apt-mirror -c apt-mirror' - my repo takes up ~70GB of disk.

Configure clients to point at the new machine - remember that the http request needs to point to your symlinked directories in /var/www. Here is the sources.list I'm using on our lab machines (note the correlation between symlinks and sources lines):

deb http://your-machine.your-domain.ca/ubuntu precise main restricted universe multiverse
deb http://your-machine.your-domain.ca/ubuntu precise-updates main restricted universe multiverse
deb http://your-machine.your-domain.ca/ubuntu precise-backports main restricted universe multiverse
deb http://your-machine.your-domain.ca/ubuntu precise-security main restricted universe multiverse
deb http://your-machine.your-domain.ca/canonical precise partner
deb http://your-machine.your-domain.ca/extras precise main
deb http://your-machine.your-domain.ca/google/earth/deb stable main
deb http://your-machine.your-domain.ca/google/chrome/deb stable main
deb http://your-machine.your-domain.ca/virtualbox precise contrib

I've also built a few packages of my own and am making them available from a private repo, which I set up using a combination of these instructions:
http://wiki.debian.org/SettingUpSignedAptRepositoryWithReprepro
http://blog.jonliv.es/2011/04/26/creating-your-own-signed-apt-repository-and-debian-packages/
http://ianlawrence.info/random-stuff/setting-up-a-personal-debian-repository

We have more info about build targets that have to be run prior to the main build to get the correct version information into the app, so I've updated and tested my build script for an eXist WAR.

Searching with find
Example: find all jpg images from this location and copy them to another location - and don't change permissions or timestamps.
find . -iname '*.jpg' -exec cp -p {} ~/Desktop/test/ \;
and
find . -iname '*.php' -exec chmod 700 {} \;
which will find all php files from here and change permissions on them to conform to the whole suPHP thing (only user-readable php files etc.)

Another one:
find . -type l -name '*.png' -exec mv {} ~/Desktop/crap/ \;
will find all of the symlink-ed png files and move them somewhere

One that matches all text files on the entire system (that is, searching recursively from /) over 10KB, owned by paul, that are not readable by other users, and then use chmod to enable reading, like this:

find / -name "*.txt" -size +10k -user paul -not -perm +o=r -exec chmod o+r {} \;

Find all recently modified files:
find . -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort

NOTE: When typing -exec parameters, be sure to include a space before \;. Otherwise, you might see an error such as missing argument to ´-exec'.

---

We had a problem on beet where a local user had Gnome Keyring issues when trying to commit changes to svn. It *appears* that if the local user name and the remote user name are the same, but have different passwords, things can get a bit weird. We deleted the local user's keyring, but the problem re-occurred on reboot.
When the user logged in to beet with their netlink id the problem did not occur.

Note to self - do not provision local users that have the same name as the project itself.

Made an attempt at using my maverick script to build the new machine (beet) but there were so many changes (package dependencies, config deprecations etc.) that I gave up and started on an oneiric builder - A LOT HAS CHANGED under the hood!

Spent a few days working on that and deployed beet with the new HCMC oneiric build. It seems OK, but still requires a few tweaks after a person logs in. I have a script on the desktop that will set things up and delete itself when done. One apparent bug that I'm tracking down: one cannot shut the machine down from the login screen. I think it's a lightdm problem at this point.

I'm continuing work on the oneiric build in preparation for pangolin as I intend to upgrade the lab to pangolin once it lands.

Sysadmin notified us that there were ISE logs filling up viola's filesystem and needed to be removed. I had them (all 4.9GB) moved to /home1t/hcmc/iselogs. We need to get the log4j system on the new viola set up to rotate logs effectively (all the while ensuring that urchin is gobbling up the current logs and making stats available to MB).

SR has added something thatrequires curl, so add it to the build script.

I'm finding it useful to attack the TEI Guidelines with XQuery once in a while, to dig out and format stuff for review and discussion. However, you can't just load the P5 source into eXist, because you'll have problems with entities and have to mess with DTD catalogues. Instead, you can just expand the guidelines-en.xml file to pull in all the content referenced through entities, like this:

.../P5/Source$ xmllint --noent guidelines-en.xml --output guidelines-expanded-en.xml

That gives you a 6MB file you can upload into eXist and query easily.

We have many Cocoon + eXist projects currently running on Tomcat 6 on Pear. At some point, they'll migrate to Tomcat 7, so I've been using Tomcat 7 on my desktop to see what problems emerge. Here's one of them:

When I started Tomcat 7 with the Mariage project running in it, the site worked, except for the home page, which showed an error to the effect that "index.html" was not there. The site doesn't use index.html at all; its sitemap matcher for the home/root looks like this:

<map:match pattern="">

and all the site links point to the directory root. I eventually determined that this was caused by a change in the way Tomcat handles default "welcome files". If you look in Tomcat's conf/web.xml, you'll see this:

    <welcome-file-list>
        <welcome-file>index.html</welcome-file>
        <welcome-file>index.htm</welcome-file>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>

Obviously, there's no handler for an empty directory root; Tomcat 7, when presented with a directory root, goes to this list to see what to pass to the webapp. This list is identical in Tomcat 6, so it's clearly a change in the logic of the handler rather than the default configuration.

First I tried adding an empty <welcome-file> element to the top of the list, but that didn't work, so I commented out all three of the <welcome-file> elements, and then normal service was resumed. Incidentally, you have to restart Tomcat between changes to web.xml to cause them to have an effect (according to the web).