Upgrading a Debian Lenny dom0 to Squeeze

From Wiki
Jump to: navigation, search

Before upgrading a Xen dom0 to squeeze, you'll find it much more manageable if the domUs are configured to use pygrub.

pygrub? Oh, fuck off!

No, really. If you used xen-tools to set up a Debian domU, Xen will use the kernel kept in /boot on the dom0. This is usually fine because the dom0 and domU will be running the same distro. But, when it becomes time to upgrade from, say, Lenny to Squeeze, the domU will be running a Squeeze userspace and a Lenny kernel. Or, when you upgrade the dom0, the reverse would happen. This is guaranteed to fuck your shit up at some later stage.

You could just keep all the kernels you need in /boot, but this would necessitate manual intervention each time Debian made an update to the kernel. Frankly, this would be a pain in the cock. pygrub provides a way to furtle around in the root partition of your domU, fish out the kernel and use it to boot the domU. This gives a way of managing Debian stable release upgrades without having to do all the machines in one go. By all means, skip ahead if you want, but it'll go wrong.

Get on with it

By default, xen-tools creates device names like:

root        = '/dev/xvda2 ro'
disk        = [

For some reason or other, grub simply can't work with this layout, so we must alter things so that grub is happy. As far as I can tell, the conditions that must be met are:

  • The root device must be a whole block device such as xvda. xvda1/2/3 is no good.
  • The root device must be first in the list.
  • The root file system must be ext3.

Edit the /etc/xen/*.cfg file for the domU in question, so that the disk layout is more like:

root        = '/dev/xvda ro'
disk        = [

Now, through whatever means is easiest, edit the /etc/fstab on the domU to match the new disk layout:

# /etc/fstab: static file system information.
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/xvdb1 none swap sw 0 0
/dev/xvda / ext3 sync,errors=remount-ro 0 1
/dev/xvdb3 /var ext3 nodev,nosuid 0 1
/dev/xvdb4 /usr ext3 nodev 0 1
/dev/xvdb5 /tmp ext3 nodev,nosuid 0 1

Shut down the domU and re-create it on the dom0 to ensure it picks up the changes from the config file:

dom0 # xm shutdown git1.spruce.toastputer.net
dom0 # xm create /etc/xen/git1.spruce.toastputer.net.cfg

Once the domU has booted up again, we re-connect to it and install grub. grub will not create /boot/grub on its own, so we must do it:

domU # mkdir -p /boot/grub

Install a kernel and grub. I assume amd64, use the appropriate kernel for your dom0 architecture:

domU # apt-get install linux-image-2.6-xen-amd64 grub

This ought to run anyway, but for safety:

domU # update-grub

Note: At this point grub threw an error at this point: grub-set-default [OPTION] entry. It is necessary to examine the /boot/grub/device.map, find the boot device (should be first in the list), and then set the default accordingly, such as:

domU # grub-set-default 0

Shut down the domU again.

Now, on the dom0, we can test pygrub:

dom0 #/usr/lib/xen-default/bin/pygrub /dev/xen-vol/git1.spruce.toastputer.net-root


    pyGRUB  version 0.6
 │ Debian GNU/Linux, kernel 2.6.26-2-xen-amd64                            │
 │ Debian GNU/Linux, kernel 2.6.26-2-xen-amd64 (single-user mode)         │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 │                                                                        │
     Use the ^ and v keys to select which entry is highlighted.
     Press enter to boot the selected OS. 'e' to edit the
     commands before booting, 'a' to modify the kernel arguments
     before booting, or 'c' for a command line.

Ok, now edit the domU's xen config, so that it uses pygrub instead of the dom0 kernel:

#  Kernel + memory size
# kernel and ramdisk not specified - handled with pygrub
#kernel      = '/boot/vmlinuz-2.6.26-1-xen-amd64'
#ramdisk     = '/boot/initrd.img-2.6.26-1-xen-amd64'
bootloader  = '/usr/lib/xen-default/bin/pygrub'
memory      = '256'

Create and start the domU using the new config. You will find that it boots using its own kernel:

dom0 # grep 'Launching bootloader' /var/log/xen/xend.log
[2011-02-05 11:07:07 18903] DEBUG (XendBootloader:100) Launching bootloader as ['/usr/lib/xen-default/bin/pygrub', '--args=root=/dev/xvda ro ', '--output=/var/run/xend/boot/xenbl.11623', '-q', '/dev/xen-vol/git1.spruce.toastputer.net-root']

Actually upgrade the domUs

Any domUs that you aren't ready to upgrade can be left running Lenny. Thanks to pygrub, they will still be working when the dom0 us upgraded to Squeeze.

domU # apt-get update && apt-get upgrade
domU # sed -i s/lenny/squeeze/g /etc/apt/sources.list
domU # apt-get update && apt-get dist-upgrade

Irritatingly, grub2 does not work with the Lenny version of Xen. So, we must use the grub-legacy package until it is time to upgrade the dom0.

domU # apt-get install grub-legacy

And finally, reboot the domU.

domU # uname -r

The numbers are bigger. This must be better!

Upgrade the dom0

Hold tight, this involved much more dicking around than I was expecting.


dom0 # apt-get update && apt-get upgrade

to make sure you have the latest Lenny packages. Then alter your sources.list to use the squeeze distribution. Also, add the non-free and contrib repositories - this is necessary because the firmware for some network interface cards has been moved out of the kernel. If you know that you hardware isn't affected by this then by all means leave them out.

dom0 # vi /etc/apt/sources.list

deb http://ftp.uk.debian.org/debian/ squeeze main non-free contrib
deb-src http://ftp.uk.debian.org/debian/ squeeze main non-free contrib

deb http://security.debian.org/ squeeze/updates main non-free contrib
deb-src http://security.debian.org/ squeeze/updates main non-free contrib

Now run:

dom0 # apt-get update && apt-get dist-upgrade

The upgrade will take some time, but you can leave the domUs running while it works through. Don't reboot when the upgrade completes - we've not finished! Now would be a good time to cleanly shut down all your domUs.

I talked about firmware a moment ago; here we can install that. At the same time, I reverted to grub for the time being, since I didn't want to change too much at once:

dom0 # apt-get install firmware-linux grub-legacy

I suspect it would be perfectly OK to go straight to grub2 as well as upgrading the kernel and Xen, but I didn't try it.

For some reason, Xen doesn't get upgraded by apt-get dist-upgrade, so we must intervene:

dom0 # apt-get install xen-hypervisor-4.0-amd64 linux-image-xen-amd64

and if you need hardware virtualisation:

dom0 # apt-get install xen-qemu-dm-4.0

Check the differences between the installed version and the dpkg-old versions of the config files /etc/xen/xend.conf and /etc/xen-tools/xen-tools.conf. Now reboot. When the dom0 enters grub, be careful, Xen 4.0-amd64 / Debian GNU/Linux, kernel 2.6.32-5-xen-amd64 won't necessarily be listed first, however this is the entry you must select.

Hurrah! All being well, you'll now have Xen 4 running on you new Squeeze system. Any VMs still running the Lenny kernel will using their own copy of the old kernel thanks to pygrub. Now we know that everything is working, lets upgrade to grub2 (it'll have to be done sooner or later!):

dom0 # apt-get install grub-pc

This removes grub-legacy and installs grub2. Pay attention when it asks which devices to install to. I have two drives in a raid1, so I want it installed to the boot record of both my drives

┌───────────────┤ Configuring grub-pc ├────────────────┐            
│ GRUB install devices:                                │            
│                                                      │            
│    [*] /dev/sda (160041 MB; GB0160EAPRR)             │            
│    [*] /dev/sdb (160041 MB; WDC_WD1600AAJS-00L7A0)   │            
│    [ ] /dev/md1 (3002 MB; ???)                       │            
│    [ ] /dev/md0 (98 MB; ???)                         │            
│                                                      │            
│                                                      │            
│                        <Ok>                          │            
│                                                      │            

...otherwise it's probably just /dev/sda. If you're really not sure, it's probably a safe strategy to simply install grub2 on all available physical drives. When this package has installed, there's more to do.

grub2's setup is not quite right for a dom0, since it lists the plain Linux kernels first. We want the Xen hypervisor+Linux combination to be the default. Change the priority:

dom0 # cd /etc/grub.d
dom0 # mv 10_linux 50_linux

Optionally, you can disable the OS prober, which will prevent grub2 poking around all your logical volumes and adding boot entries for them:

dom0 # echo "" >> /etc/default/grub
dom0 # echo "# Disable OS prober to prevent virtual machines on logical volumes from appearing in the boot menu." >> /etc/default/grub
dom0 # echo "GRUB_DISABLE_OS_PROBER=true" >> /etc/default/grub

Finally, update grub2, check that it shows the kernel that you intend to boot from or you will be royally boned. Again do not reboot until you are sure that grub2 is configured properly.

dom0 # update-grub2
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64
Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64
Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64
Found linux image: /boot/vmlinuz-2.6.26-1-xen-amd64
Found initrd image: /boot/initrd.img-2.6.26-1-xen-amd64

When satisfied, shutdown the domUs and reboot:

dom0 # xm shutdown --all
dom0 # shutdown -r now

All being well, grub2 will choose the right kernel, this will boot and your dom0 will be running Squeeze.