Fun with legacy XEN and LVM

, , ,

At work, we run a lot of automated integration, unit and system tests. This requires virtual machines that can be started up, get deployed with the software we want to test, run tests against it, shut it down and reset to a non-tainted state for the next test. While we’re working on a OpenStack based solution for that kind of internal deployment, i’ve had the joy of working with a, by all means, legacy Xen paravirtual hypervisor setup. It basically runs on a old machine with old debian and old XEN. The XEN guests (aka. domU’s) are stored within LVM on an old storage system.

The requirement in my case is, that all partitions exist as raw LVs without a partition table, so i can continue using existing tools for backup/restore. Sadly virt-install does not allow that (from what i found out at least) so i needed quite a workaround. To start with, i used virt-install to install the domain to a temporary image file.

Hint: virt-install requires a valid locale, just in case there is none set:

1
$ export LC_ALL=C

When installing, i’m not using a separate /boot or SWAP partition, just for the sake of ease. However, the provided steps may be used for more complex partition setups as well. When installing, do NOT create a LVM based storage setup within the VM.

In this case, i’m going to use Debian Wheezy, but when following some tricks, it’s possible to install every Linux Distro using this workflow. See “Other Distros” for more details. Since the old hardware does not support Hardware-assisted virtualization (HVM) either provided by AMD-V or Intel VT-x, i’ve to stick with “paravirt”.

1
$ virt-install --paravirt --name debian7-install --ram 1024 --file /root/debian7-install.img --file-size 5 --nographics --location http://ftp.de.debian.org/debian/dists/wheezy/main/installer-amd64/

Virt-install may print some error output, however in most cases it still succeeds. Just jump into the domU. For SLES, please see the “Other Distros” section.

1
$ xm console debian7-install

Observe the image partition table, sector size and starting block of a partition

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ fdisk -lu debian7-install.img
You must set cylinders
You can do this from the extra functions menu.
Disk debian7-install.img: 0 MB, 0 bytes
181 heads, 40 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b3bee

Device               Boot     Start   End      Blocks    Id    System
debian7-install.img  *        2048    10483711 5240832   83    Linux
Partition 1 has different physical/logical beginnings (non-Linux?):
     phys=(0, 32, 33) logical=(0, 51, 9)
Partition 1 has different physical/logical endings:
     phys=(652, 180, 40) logical=(1448, 55, 40)

In this case, the sector size is 512b and the primary (only) partition starts at block 2048.

Now mount the image file to a loop device, starting at the offset calculated based on sector size and start block.

1
$ losetup /dev/loop0 debian7-install.img -o $((2048 * 512))

Check if the partition has been mounted correctly

1
2
$ file -s /dev/loop0
/dev/loop0: Linux rev 1.0 ext4 filesystem data, UUID=18af6c1e-1c7e-4ccc-9828-a9a46770d0f8 (extents) (large files) (huge files)

As we see, the mounted data is a valid ext4 partition.

Create a new LV based on the size of the partition, calculated by start and end block, multiplied by the sector size.

1
$ lvcreate -L $(((10483711 - 2048) * 512))b --name debian7-install-disk vg

Now use partimage to create an image from the mounted partition.

1
$ partimage -z2 -o -d -M -b save /dev/loop0 /root/backups/debian7-install.new

Compression and use of partimage will save a lot of space since only used blocks are stored within the new image.

1
2
$ ls -lh /root/backups/debian7-install.new.000
-rw------- 1 root root 192M Nov 28 10:38 /root/backups/debian7-install.new.000

Unmount the loop device after getting the partition image

1
$ losetup -d /dev/loop0

You may also move the original image to another location and remove the domain from XEN

1
2
$ mv debian7-install.img debian7-install.img.old
$ xm delete debian7-install

Create a SWAP partition for later usage

1
2
3
4
5
$ lvcreate -L 512M --name debian7-install-swap vg
$ mkswap /dev/vg/debian7-install-swap
mkswap: /dev/vg/debian7-install-swap: warning: don't erase bootbits sectors on whole disk. Use -f to force.
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=46d4b88f-1731-459d-9b98-edf8b2066717

Now restore the image created on the original image and it’s partition to the LV

1
$ partimage restore /dev/vg/debian7-install-disk /root/backups/debian7-install.new.000

Fetch the LVs UUIDs, these are required for the next steps

1
2
3
4
$ blkid
[...]
/dev/mapper/vg-debian7--install--disk: UUID="36aa2e94-9f2f-4ab1-b013-fd2cb960b55a" TYPE="ext4"
/dev/mapper/vg-debian7--install--swap: UUID="46d4b88f-1731-459d-9b98-edf8b2066717" TYPE="swap"

The image has been successfully restored to LVM, so the hard work is done. Next, some modifications within the image need to be performed, to make the system bootable.

For that, just mount the LV to an arbitrary mount point.

1
2
$ mkdir /mnt/debian7-install
$ mount /dev/vg/debian7-install-disk /mnt/debian7-install/

XEN will use pygrub to boot the machine. Pygrub only understands the menu.lst style GRUB configuration files, not the more recent grub.cfg files. Therefor we have to create a new menu.lst file based on the new partition settings. The UUID value for the / disk is used here.

1
2
3
4
5
6
7
8
9
10
11
12
13
$ vim /mnt/debian7-install/boot/grub/menu.lst
default         0
timeout         2

title         Debian GNU/Linux 7
root          (hd0,0)
kernel        /boot/vmlinuz-3.2.0-4-amd64 root=UUID=36aa2e94-9f2f-4ab1-b013-fd2cb960b55a ro
initrd        /boot/initrd.img-3.2.0-4-amd64

title         Debian GNU/Linux 7 (Single-User)
root          (hd0,0)
kernel        /boot/vmlinuz-3.2.0-4-amd64 root=UUID=36aa2e94-9f2f-4ab1-b013-fd2cb960b55a ro single
initrd        /boot/initrd.img-3.2.0-4-amd64

Adjust the kernel path, version and name based on the files located in /boot

Also adjust the fstab file, use the UUID of the / and SWAP disk.

1
2
3
$ vim /mnt/debian7-install/etc/fstab
UUID=36aa2e94-9f2f-4ab1-b013-fd2cb960b55a /     ext4     errors=remount-ro 0    1
UUID=46d4b88f-1731-459d-9b98-edf8b2066717 none  swap     sw                0    0

Unmount the LV after modifying these settings

1
$ umount /mnt/rhel6-install/

Create a new XEN configuration file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ vim /etc/xen/debian7-install.cfg
bootloader = '/usr/lib/xen-default/bin/pygrub'

vcpus   = '1'
memory  = '1024'

root = '/dev/xvda2 rw'
disk = [
         'phy:/dev/vg/debian7-install-disk,xvda2,w',
         'phy:/dev/vg/debian7-install-swap,xvda1,w',
       ]

name = 'debian7-install'
vif  = [ 'bridge=eth0,mac=00:16:36:1d:0f:b6' ]

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

In some cases, the journal got corrupted, for the sake of consistency, run fsck on the newly created disk:

1
$ fsck /dev/vg/debian7-install-disk

Now we’re ready to boot the new XEN domain from LVM:

1
$ xm create -c /etc/xen/debian7-install.cfg

Other Distros

Debian 6 (Squeeze)

For Squeeze, just modify the location URL to point to a different dist.

1
$ virt-install --paravirt --name debian6-install --ram 1024 --file /root/debian6-install.img --file-size 5 --nographics --location http://ftp.de.debian.org/debian/dists/squeeze/main/installer-amd64/

CentOS6

Just call virt-install using a mirror repository:

1
$ virt-install --paravirt --name centos6-install --ram 1024 --file /root/centos6-install.img --file-size 5 --nographics --location http://mirror.netcologne.de/centos/6.4/os/x86_64/

Note that the default text-mode CentOS installer is somewhat limited. It does not allow to modify the partition table, for example not using LVM but just a plain partition. If that’s the case for you, use the VNC (launched from the text-mode installer) or virt-viewer to launch the more sophisticated (GTK’ish) installer of CentOS. This will allow you to work around LVM.

RHEL6

For this, you need the binary DVD images of RHEL6 and make them accessible via HTTP (aka. create your own installation mirror). Therefor, you require a HTTP server and need to mount it to a directory which gets served:

1
$ mount -o loop /root/rhel-server-6.4-x86_64-dvd.iso /var/www/rhel/

Then, use virt-install to start the installation process from this HTTP site:

1
$ virt-install --paravirt --name rhel6-install --ram 1024 --file /root/rhel6-install.img --file-size 5 --nographics --location http://example.com/rhel/

Note that the default text-mode RHEL6 installer is somewhat limited. It does not allow to modify the partition table, for example not using LVM but just a plain partition. If that’s the case for you, use the VNC (launched from the text-mode installer) or virt-viewer to launch the more sophisticated (GTK’ish) installer of RHEL6. This will allow you to work around LVM.

When booting, this errors tend to show up, however they won’t keep the domU from booting

1
2
3
4
PCI: Fatal: No config space access function found
Could not set up I/O space
Could not set up I/O space
Could not set up I/O space

SLES11

For SLES, the process is similar to the one used at RHEL, but enhanced by the pitfall of SLES requiring some kind of “real” graphical interface rather than just a XEN console. In my case i had no X running at the XEN dom0, so i had to improvise quite a bit.

xm console just stopped responding at:

1
2
3
4
5
6
[    9.479067] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    9.479084] EDD information not available.
[    9.646300] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    9.646317] EDD information not available.
[    9.813624] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    9.813641] EDD information not available.

For this, you need the binary DVD images of SLES11 and make them accessible via HTTP (aka. create your own installation mirror). Therefor, you require a HTTP server and need to mount it to a directory which gets served:

1
$ mount -o loop SLES-11-SP3-DVD-x86_64-GM-DVD1.iso /var/www/sles/

Then, use virt-install to start the installation process from this HTTP site:

1
$ virt-install --paravirt --name sles11-install --ram 1024 --file /root/sles11-install.img --file-size 5 --nographics --location http://example.com/sles/ --vnc

Now the fun part starts. Usually there is a VNC console available from virt-install. This did not work for me when using a “real” remote VNC client. However, using “virt-viewer” did the trick. To use any kind of VNC, i’ve added the “vnc” parameter to the virt-install command.

Right after executing virt-install, call virt-viewer to jump into the installation process. In my case, i had to use a client computer running X, install xserver-xorg, xauth at the dom0 to run “virt-viewer”:

1
$ apt-get install xauth xserver-xorg

Check that SSHd on dom0 allows X forwards:

1
2
3
4
5
$ vim /etc/ssh/sshd_config
[...]
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no

Connect to the dom0 machine from a client using X and enable forwarding:

1
2
$ ssh -X root@your.dom0
$ virt-viewer sles11-install

When installing SLES, make sure to install the bootloader to MBR (the default is the “root” partition!) and select the option to install within a paravirtualizied environment. After installation, you’ll also need vnc to boot the machine since SLES uses an awkward “graphical” boot console. You can do that by adding this to your domU config file:

1
vfb         = [ 'type=vnc' ]

Later on, modify the /boot/grub/menu.lst file and set

1
splash=0