Setting up a local apt-mirror (updated for precise)

25/11/11

Permalink 01:20:58 pm, by Greg, 1459 words, 7648 views   English (CA)
Categories: Servers, Activity log, Documentation; Mins. worked: 240

Setting up a local apt-mirror (updated for precise)

In order to avoid bottlenecks I rebuilt an old machine and set it up as an apt mirror of ubuntu, virtualbox, google chrome and google earth repos. Here's how:
Install apt-mirror 'sudo apt-get install apt-mirror'
Configure apt-mirror at /etc/apt/mirror.list

My mirror.list looks like this:
############# config ##################
#
set base_path /mnt/repo/apt-mirror
#
set mirror_path $base_path/mirror
set skel_path $base_path/skel
set var_path $base_path/var
set cleanscript $var_path/clean.sh
set defaultarch amd64
set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads 20
set _tilde 0
#
############# end config ##############

deb http://mirror.peer1.net/ubuntu precise main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-security main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-updates main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-proposed main restricted universe multiverse
deb http://mirror.peer1.net/ubuntu precise-backports main restricted universe multiverse
# if you want to use the mirror as an install base (using, say, the mini.iso image) you need to mirror the debian-installer directories.
deb http://mirror.peer1.net/ubuntu precise main main/debian-installer restricted restricted/debian-installer universe universe/debian-installer multiverse multiverse/debian-installer
deb http://archive.canonical.com/ubuntu precise partner
deb http://extras.ubuntu.com/ubuntu precise main

deb http://download.virtualbox.org/virtualbox/debian precise contrib

deb http://dl.google.com/linux/earth/deb/ stable main
deb http://dl.google.com/linux/chrome/deb/ stable main

#### Multiarch (oneiric and newer) mirrors need to contain both architectures ####
deb-i386 http://mirror.peer1.net/ubuntu precise main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-security main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-updates main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-proposed main restricted universe multiverse
deb-i386 http://mirror.peer1.net/ubuntu precise-backports main restricted universe multiverse

deb-i386 http://archive.canonical.com/ubuntu precise partner
deb-i386 http://extras.ubuntu.com/ubuntu precise main

deb-i386 http://download.virtualbox.org/virtualbox/debian precise contrib

deb-i386 http://dl.google.com/linux/earth/deb/ stable main
deb-i386 http://dl.google.com/linux/chrome/deb/ stable main

clean http://mirror.peer1.net/ubuntu
clean http://archive.canonical.com/ubuntu
clean http://extras.ubuntu.com/ubuntu
clean http://download.virtualbox.org.virtualbox/debian
clean http://dl.google.com/linux/earth/deb/
clean http://dl.google.com/linux/chrome/deb/

Notice that I'm mirroring i386 trees even though I only build 64bit machines - as of oneiric (11.10) Ubuntu has gone to a multiarch system, so this is no longer optional.

Note that in order to configure the mirroring of repos that uses gpg keys (like virtualbox and google), you need to add the key to the apt-mirror machine. Do it like this:
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add - &&
wget -q -O - http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc | sudo apt-key add - &&

I want it to autoupdate every day, so I edited /etc/cron.d/apt-mirror (uncommented the cronjob line in the file and pointed the log where I wanted to store them).

Each time the mirror job runs it tries to run the postmirror script whether you uncomment the path to it in the mirror.list or not. Oddly, the script doesn't ship with the apt-mirror package, so I had to create it myself. I put it where the mirror.list expects it - in this case at /mnt/repo/apt-mirror/var/postmirror.sh) - and put this in the script itself:
/mnt/repo/apt-mirror/var/clean.sh
This ensures that the cleanup script gets run (it doesn't seem to run automatically).

The apt-mirror requires 'mirror', 'var' and 'skel' directories, so if you're setting this up on a separate drive you'll need to add them on the storage drive 'sudo mkdir /mnt/repo/apt-mirror /mnt/repo/apt-mirror/var /mnt/repo/apt-mirror/skel'. Note that the script wants to run as the apt-mirror user, so we need to change the ownership on those directories 'sudo chown -R apt-mirror:apt-mirror /mnt/repo/apt-mirror'.

I need it readily accessible to clients, so I installed apache 'sudo apt-get install apache2' and added symlinks to the appropriate directories e.g. 'ln -s /mnt/storage/mirror/ftp.heanet.ie/pub/ubuntu/ /var/www/ubuntu' - do that for each repo you're mirroring. I made them like this:
sudo ln -s /mnt/repo/apt-mirror/mirror/archive.canonical.com/ubuntu /var/www/canonical
sudo ln -s /mnt/repo/apt-mirror/mirror/extras.ubuntu.com/ubuntu /var/www/extras
sudo ln -s /mnt/repo/apt-mirror/mirror/dl.google.com/linux /var/www/google
sudo ln -s /mnt/repo/apt-mirror/mirror/mirror.peer1.net/ubuntu /var/www/ubuntu
sudo ln -s /mnt/repo/apt-mirror/mirror/download.virtualbox.org/virtualbox/debian /var/www/virtualbox

Now that everything is configured, go ahead and fill up the new repo. Remember to run the job as the apt-mirror user 'sudo su - apt-mirror -c apt-mirror' - my precise repo takes up ~95GB of disk.

Configure clients to point at the new machine - remember that the http request needs to point to your symlinked directories in /var/www. Here is the sources.list I'm using on our lab machines (note the correlation between symlinks and sources lines):

deb http://your-machine.your-domain.ca/ubuntu precise main restricted universe multiverse
deb http://your-machine.your-domain.ca/ubuntu precise-updates main restricted universe multiverse
deb http://your-machine.your-domain.ca/ubuntu precise-backports main restricted universe multiverse
deb http://your-machine.your-domain.ca/ubuntu precise-security main restricted universe multiverse
deb http://your-machine.your-domain.ca/canonical precise partner
deb http://your-machine.your-domain.ca/extras precise main
deb http://your-machine.your-domain.ca/google/earth/deb stable main
deb http://your-machine.your-domain.ca/google/chrome/deb stable main
deb http://your-machine.your-domain.ca/virtualbox precise contrib

I've also built a few packages of my own and am making them available from a private repo, which I set up using a combination of these instructions:
http://wiki.debian.org/SettingUpSignedAptRepositoryWithReprepro
http://blog.jonliv.es/2011/04/26/creating-your-own-signed-apt-repository-and-debian-packages/
http://ianlawrence.info/random-stuff/setting-up-a-personal-debian-repository

UPDATE: I just added raring (Ubuntu 13.04) to the mix by copy/pasting the appropriate bits above - although I did not duplicate repos that are not distro-dependant (e.g. google). I also removed openshot as it appears to be up-to-date in the 13.04 repo.
I did nothing else to add the new mirror set, and the mirror seems to be building right now. So, adding a new set seems to be as simple as that!

UPDATE: I've noticed that several sub-directories in the ubuntu repo do not get mirrored. It turns out that they are treated differently, so the behaviour is expected.
I'm getting around this by rsync-ing the directories in the postmirror script (apt-mirror/var/postmirror.sh). I've added rsync commands for all the directories I'm interested in for precise (12.04), which is actually quite a few, and the command I'm using looks like this:

rsync --recursive --times --links --hard-links --delete --delete-after rsync://archive.ubuntu.com/ubuntu/dists/precise/main/debian-installer /mnt/repo/apt-mirror/mirror/mirror.peer1.net/ubuntu/dists/precise/main/

I'm specifically mirroring the main/debian-installer, main/dist-upgrader-all, main/i18n, main/installer-amd64, main/installer-i386, main/uefi directories (where they exist) in all of the various 'precise' sub-directories (like precise-updates and so forth).

This unfortunately does not mirror udebs (necessary for tasks like netboot installs), so I had to write a script to get them. I've added it to postmirror.sh. The standalone version looks like this:


#!/bin/bash
# modified version of this:
#	https://wiki.debian.org/DebianEdu/HowTo/NetbootInstall/BuildMirror#Fetching_the_udebs

# Mirrors with some serious speed
# Canadian: http://ubuntu-archives.mirror.nexicom.net (20 Gbps)
# German: http://mirror.skylink-datacenter.de/ubuntu (20 Gbps)
# Irish: http://ftp.heanet.ie/pub/ubuntu/ (10 Gbps)

# Mirrors nearby
# UBC: http://mirror.it.ubc.ca/ubuntu/ (1 Gbps)

MIRROR="http://mirror.skylink-datacenter.de/ubuntu"
UDEBMIRRORS="$MIRROR/dists/precise/main
             $MIRROR/dists/precise/restricted
             $MIRROR/dists/precise/universe
             $MIRROR/dists/precise/multiverse             
             $MIRROR/dists/precise-updates/main
             $MIRROR/dists/precise-updates/restricted
             $MIRROR/dists/precise-updates/universe
             $MIRROR/dists/precise-updates/multiverse             
             $MIRROR/dists/precise-security/main
             $MIRROR/dists/precise-security/restricted
             $MIRROR/dists/precise-security/universe
             $MIRROR/dists/precise-security/multiverse             
             $MIRROR/dists/precise-proposed/main
             $MIRROR/dists/precise-proposed/restricted
             $MIRROR/dists/precise-proposed/universe
             $MIRROR/dists/precise-proposed/multiverse             
             $MIRROR/dists/precise-backports/main
             $MIRROR/dists/precise-backports/restricted
             $MIRROR/dists/precise-backports/universe
             $MIRROR/dists/precise-backports/multiverse"
LOCALPOOL="/mnt/bfd/Software/hcmc-utilities/udebs"
#LOCALPOOL="/mnt/repo/apt-mirror/mirror/archive.ubuntu.com/ubuntu"

for UDEBMIRROR in $UDEBMIRRORS ; do
    for UDEB in $(wget --quiet -O - $UDEBMIRROR/debian-installer/binary-amd64/Packages.bz2 | bzip2 -cd | sed -ne 's/^Filename: //p') ; do
        if [ ! -e $UDEB ];then
		curl $MIRROR/"$UDEB" --create-dirs --silent -o $LOCALPOOL/$UDEB
	fi
    done
done

# adjust perms
chown -R apt-mirror:apt-mirror $LOCALPOOL
chmod -R 775 $LOCALPOOL

Pingbacks:

No Pingbacks for this post yet...

Maintenance

This blog is the location for all work involving software and hardware maintenance, updates, installs, etc., both routine and urgent, in the server room, the labs and the R&D rooms.

Reports

Categories

September 2014
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30        

XML Feeds