Upgrading a Debian Lenny dom0 to Squeeze
Before upgrading a Xen dom0 to squeeze, you'll find it much more manageable if the domUs are configured to use pygrub.
pygrub? Oh, fuck off!
No, really. If you used xen-tools to set up a Debian domU, Xen will use the kernel kept in
/boot on the dom0. This is usually fine because the dom0 and domU will be running the same distro. But, when it becomes time to upgrade from, say, Lenny to Squeeze, the domU will be running a Squeeze userspace and a Lenny kernel. Or, when you upgrade the dom0, the reverse would happen. This is guaranteed to fuck your shit up at some later stage.
You could just keep all the kernels you need in
/boot, but this would necessitate manual intervention each time Debian made an update to the kernel. Frankly, this would be a pain in the cock. pygrub provides a way to furtle around in the root partition of your domU, fish out the kernel and use it to boot the domU. This gives a way of managing Debian stable release upgrades without having to do all the machines in one go. By all means, skip ahead if you want, but it'll go wrong.
Get on with it
By default, xen-tools creates device names like:
root = '/dev/xvda2 ro' disk = [ 'phy:/dev/xen-vol/git1.spruce.toastputer.net-swap,xvda1,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-root,xvda2,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-var,xvda3,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-usr,xvda4,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-tmp,xvda5,w', ]
For some reason or other, grub simply can't work with this layout, so we must alter things so that grub is happy. As far as I can tell, the conditions that must be met are:
- The root device must be a whole block device such as
xvda1/2/3is no good.
- The root device must be first in the list.
- The root file system must be ext3.
/etc/xen/*.cfg file for the domU in question, so that the disk layout is more like:
root = '/dev/xvda ro' disk = [ 'phy:/dev/xen-vol/git1.spruce.toastputer.net-root,xvda,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-swap,xvdb1,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-var,xvdb3,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-usr,xvdb4,w', 'phy:/dev/xen-vol/git1.spruce.toastputer.net-tmp,xvdb5,w', ]
Now, through whatever means is easiest, edit the
/etc/fstab on the domU to match the new disk layout:
# /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/xvdb1 none swap sw 0 0 /dev/xvda / ext3 sync,errors=remount-ro 0 1 /dev/xvdb3 /var ext3 nodev,nosuid 0 1 /dev/xvdb4 /usr ext3 nodev 0 1 /dev/xvdb5 /tmp ext3 nodev,nosuid 0 1
Shut down the domU and re-create it on the dom0 to ensure it picks up the changes from the config file:
dom0 # xm shutdown git1.spruce.toastputer.net dom0 # xm create /etc/xen/git1.spruce.toastputer.net.cfg
Once the domU has booted up again, we re-connect to it and install grub. grub will not create
/boot/grub on its own, so we must do it:
domU # mkdir -p /boot/grub
Install a kernel and grub. I assume amd64, use the appropriate kernel for your dom0 architecture:
domU # apt-get install linux-image-2.6-xen-amd64 grub
This ought to run anyway, but for safety:
domU # update-grub
Note: At this point grub threw an error at this point:
grub-set-default [OPTION] entry. It is necessary to examine the
/boot/grub/device.map, find the boot device (should be first in the list), and then set the default accordingly, such as:
domU # grub-set-default 0
Shut down the domU again.
Now, on the dom0, we can test pygrub:
dom0 #/usr/lib/xen-default/bin/pygrub /dev/xen-vol/git1.spruce.toastputer.net-root
pyGRUB version 0.6 ┌────────────────────────────────────────────────────────────────────────┐ │ Debian GNU/Linux, kernel 2.6.26-2-xen-amd64 │ │ Debian GNU/Linux, kernel 2.6.26-2-xen-amd64 (single-user mode) │ │ │ │ │ │ │ │ │ │ │ │ │ └────────────────────────────────────────────────────────────────────────┘ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS. 'e' to edit the commands before booting, 'a' to modify the kernel arguments before booting, or 'c' for a command line.
Ok, now edit the domU's xen config, so that it uses pygrub instead of the dom0 kernel:
# # Kernel + memory size # # kernel and ramdisk not specified - handled with pygrub #kernel = '/boot/vmlinuz-2.6.26-1-xen-amd64' #ramdisk = '/boot/initrd.img-2.6.26-1-xen-amd64' bootloader = '/usr/lib/xen-default/bin/pygrub' memory = '256'
Create and start the domU using the new config. You will find that it boots using its own kernel:
dom0 # grep 'Launching bootloader' /var/log/xen/xend.log [2011-02-05 11:07:07 18903] DEBUG (XendBootloader:100) Launching bootloader as ['/usr/lib/xen-default/bin/pygrub', '--args=root=/dev/xvda ro ', '--output=/var/run/xend/boot/xenbl.11623', '-q', '/dev/xen-vol/git1.spruce.toastputer.net-root']
Actually upgrade the domUs
Any domUs that you aren't ready to upgrade can be left running Lenny. Thanks to pygrub, they will still be working when the dom0 us upgraded to Squeeze.
domU # apt-get update && apt-get upgrade domU # sed -i s/lenny/squeeze/g /etc/apt/sources.list domU # apt-get update && apt-get dist-upgrade
Irritatingly, grub2 does not work with the Lenny version of Xen. So, we must use the
grub-legacy package until it is time to upgrade the dom0.
domU # apt-get install grub-legacy
And finally, reboot the domU.
domU # uname -r 2.6.32-5-xen-amd64
The numbers are bigger. This must be better!
Upgrade the dom0
Hold tight, this involved much more dicking around than I was expecting.
dom0 # apt-get update && apt-get upgrade
to make sure you have the latest Lenny packages. Then alter your
sources.list to use the squeeze distribution. Also, add the non-free and contrib repositories - this is necessary because the firmware for some network interface cards has been moved out of the kernel. If you know that you hardware isn't affected by this then by all means leave them out.
dom0 # vi /etc/apt/sources.list deb http://ftp.uk.debian.org/debian/ squeeze main non-free contrib deb-src http://ftp.uk.debian.org/debian/ squeeze main non-free contrib deb http://security.debian.org/ squeeze/updates main non-free contrib deb-src http://security.debian.org/ squeeze/updates main non-free contrib
dom0 # apt-get update && apt-get dist-upgrade
The upgrade will take some time, but you can leave the domUs running while it works through. Don't reboot when the upgrade completes - we've not finished! Now would be a good time to cleanly shut down all your domUs.
I talked about firmware a moment ago; here we can install that. At the same time, I reverted to grub for the time being, since I didn't want to change too much at once:
dom0 # apt-get install firmware-linux grub-legacy
I suspect it would be perfectly OK to go straight to grub2 as well as upgrading the kernel and Xen, but I didn't try it.
For some reason, Xen doesn't get upgraded by
apt-get dist-upgrade, so we must intervene:
dom0 # apt-get install xen-hypervisor-4.0-amd64 linux-image-xen-amd64
and if you need hardware virtualisation:
dom0 # apt-get install xen-qemu-dm-4.0
Check the differences between the installed version and the dpkg-old versions of the config files
/etc/xen-tools/xen-tools.conf. Now reboot. When the dom0 enters grub, be careful, Xen 4.0-amd64 / Debian GNU/Linux, kernel 2.6.32-5-xen-amd64 won't necessarily be listed first, however this is the entry you must select.
Hurrah! All being well, you'll now have Xen 4 running on you new Squeeze system. Any VMs still running the Lenny kernel will using their own copy of the old kernel thanks to pygrub. Now we know that everything is working, lets upgrade to grub2 (it'll have to be done sooner or later!):
dom0 # apt-get install grub-pc
This removes grub-legacy and installs grub2. Pay attention when it asks which devices to install to. I have two drives in a raid1, so I want it installed to the boot record of both my drives
┌───────────────┤ Configuring grub-pc ├────────────────┐ │ GRUB install devices: │ │ │ │ [*] /dev/sda (160041 MB; GB0160EAPRR) │ │ [*] /dev/sdb (160041 MB; WDC_WD1600AAJS-00L7A0) │ │ [ ] /dev/md1 (3002 MB; ???) │ │ [ ] /dev/md0 (98 MB; ???) │ │ │ │ │ │ <Ok> │ │ │ └──────────────────────────────────────────────────────┘
...otherwise it's probably just /dev/sda. If you're really not sure, it's probably a safe strategy to simply install grub2 on all available physical drives. When this package has installed, there's more to do.
grub2's setup is not quite right for a dom0, since it lists the plain Linux kernels first. We want the Xen hypervisor+Linux combination to be the default. Change the priority:
dom0 # cd /etc/grub.d dom0 # mv 10_linux 50_linux
Optionally, you can disable the OS prober, which will prevent grub2 poking around all your logical volumes and adding boot entries for them:
dom0 # echo "" >> /etc/default/grub dom0 # echo "# Disable OS prober to prevent virtual machines on logical volumes from appearing in the boot menu." >> /etc/default/grub dom0 # echo "GRUB_DISABLE_OS_PROBER=true" >> /etc/default/grub
Finally, update grub2, check that it shows the kernel that you intend to boot from or you will be royally boned. Again do not reboot until you are sure that grub2 is configured properly.
dom0 # update-grub2 Generating grub.cfg ... Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64 Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64 Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64 Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64 Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64 Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64 Found linux image: /boot/vmlinuz-2.6.26-1-xen-amd64 Found initrd image: /boot/initrd.img-2.6.26-1-xen-amd64 done
When satisfied, shutdown the domUs and reboot:
dom0 # xm shutdown --all dom0 # shutdown -r now
All being well, grub2 will choose the right kernel, this will boot and your dom0 will be running Squeeze.