<> = Overview = Updating a debian etch (v4.1) system to lenny (v5.0) also involves an upgrade of xen from v3.0 to v3.2. This can create quite some hassle for specific setups. The documentation below describes some of the issues, that we ran into, when we upgraded the https://systemausfall.org server. == The xen network setup == Our xen network setup is not very common. Thus it should be helpful to give you a short introduction. === Overview === Network interfaces: * dom0 has one physical network interface: ''eth0'' * dummy network interfaces (kernel module ''dummy'') are used for most bridges in dom0 * most domU share a common network bridge ''xenbr0'' * dom0's ''eth0'' is bridged to only one ''gateway'' domU as ''xenbr1'' * some domU are connected to the ''gateway'' domU via separate bridges (''xenbr2'' and ''xenbr3'') Advantages of this setup: * the traffic of all domU hosts flows through the ''gateway'' domU: * effective traffic control is possible * global routing or tunneling (vpn?) setups are possible * separate networks for different zones of trust Disadvantages: * complexity :) === Configuration details === Our network configuration script is linked in ''/etc/xen/xend-config.sxp'': {{{ (network-script custom_separate_bridges.sh) }}} The above script is located in ''/etc/xen/scripts/custom_separate_bridges.sh'': {{{ #!/bin/sh dir=$(dirname "$0") # the common network bridge for all sao hosts "$dir/network-bridge" "$@" vifnum=0 netdev=dummy0 bridge=xenbr0 # the bridge connected to the internet (only for the gateway) "$dir/network-bridge" "$@" vifnum=1 netdev=eth0 bridge=xenbr1 # the bridge for the separated sao-network (www-users) "$dir/network-bridge" "$@" vifnum=2 netdev=dummy1 bridge=xenbr2 # the bridge for the network of public IP addresses (mail server ...) "$dir/network-bridge" "$@" vifnum=3 netdev=dummy2 bridge=xenbr3 }}} = Upgrade details = == Disable save/restore for xen == You need to shutdown the domU hosts completely during the upgrade. Thus you may want to disable the following line in ''/etc/default/xendomains'': {{{ #XENDOMAINS_SAVE=/var/lib/xen/save }}} Additionally you may also want to remove any old saved states of xm machines: {{{ rm /var/lib/xen/save/* }}} Now the xen hosts will be shutdown instead of being saved during ''invoke-rc.d xendomains stop''. This is what we want during the upgrade. afterwards you may change the behaviour according to your preference. == Update the kernel == === Update the kernel link for all domU === To ease the maintenance of the domU kernel during upgrades, we use symlinks in ''/boot'': {{{ thorax:~# ls -l /boot lrwxrwxrwx 1 root root 35 2008-12-28 05:20 domU-initrd -> /boot/initrd.img-2.6.18-6-xen-amd64 lrwxrwxrwx 1 root root 32 2008-12-28 05:20 domU-kernel -> /boot/vmlinuz-2.6.18-6-xen-amd64 -- snipped some more details -- }}} Update these links to the new domU kernel: {{{ lrwxrwxrwx 1 root root 35 2008-12-28 05:20 domU-initrd -> /boot/initrd.img-2.6.26-1-xen-amd64 lrwxrwxrwx 1 root root 32 2008-12-28 05:20 domU-kernel -> /boot/vmlinuz-2.6.26-1-xen-amd64 }}} Make sure, that all xen host configuration files (e.g. in ''/etc/xen/'' or ''/etc/xen/vm/'') contain the following lines: {{{ kernel = '/boot/domU-kernel' ramdisk = '/boot/domU-initrd' }}} Update the configuration file of ''xen-tools'' (''/etc/xen-tools/xen-tools.conf'') to use these kernel symlinks by default for new xen hosts: {{{ kernel = /boot/domU-kernel initrd = /boot/domU-initrd #kernel = /boot/vmlinuz-`uname -r` #initrd = /boot/initrd.img-`uname -r` }}} === Update the new module directory in all domU === * shutdown all xen hosts * mount the system partitions of the xen hosts * copy the new modules directory: {{{ cp -a /lib/modules/`uname -r` $MNT_DIR/lib/modules/ }}} == No terminal for ssh login == === Behaviour === Login to domU hosts via ssh fails with the following output: {{{ PTY allocation request failed on channel 0 stdin: is not a tty }}} Even ''xm console $HOST_NAME'' stops before the ''login'' prompt. === Workaround === Debian bug [[http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=502798|#502798]] suggests to add the following lines to the configuration file of each domU: {{{ # necessary for ssh login since lenny # see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=502798 extra = "console=hvc0" }}} Additionally you need to install ''udev'' in the host: ''aptitude install udev''. Maybe you will need to reset ''/etc/udev/rules.d/z25_persistent-net.rules'', if the network interface does not show up after the next reboot. See the next section below for details. This allows to login via ''ssh'', but ''xm console'' still does not work. == Broken network interfaces in dom0 == After the first reboot with the new lenny kernel and xen-utils v3.2, the network interfaces of dom0 were broken. During bootup error messages like the following appeared: {{{ Ignoring unknown interface xenbr0=xenbr0 }}} This was most probably caused by a problem of the ''network-bridge'' script in ''xen-utils-common''. See this [[http://lists.xensource.com/archives/html/xen-users/2008-09/msg00475.html|post in the xen-users mailing list]] for details. Apply the following patch on ''/etc/xen/scripts/network-bridge'' to solve this problem: {{{ --- network-bridge.orig 2008-12-28 06:30:54.000000000 +0100 +++ network-bridge 2008-12-28 06:31:08.000000000 +0100 @@ -96,7 +96,7 @@ } do_ifup() { - if ! ifup $1 ; then + if ! ifconfig $1 up ; then if [ -n "$addr_pfx" ] ; then # use the info from get_ip_info() ip addr flush $1 }}} == Dummy network interfaces stopped working == The way of using the ''dummy'' kernel module changed slightly between 2.6.18 and 2.6.26 (at least in debian). === Previous setup for linux 2.6.26 === For 2.6.18 we used something like the following to setup a dummy network interface in ''/etc/network/interfaces'': {{{ auto dummy0 iface dummy0 inet static pre-up modprobe -o dummy0 dummy post-down modprobe -r -o dummy0 dummy && modprobe -r dummy address 10.0.1.5 netmask 255.255.255.0 }}} Thus we needed to have a separate ''modprobe -o dummyX dummy'' call for every single dummy module network interface. === New setup for 2.6.26 === In 2.6.26 you need to load the module only once. The number of available dummy network interfaces is determined by the module option ''numdummies=X''. If you omit ''numdummies'', then only one dummy network interface is available. Thus you can add the dummy module to the ''/etc/modules'' file: {{{ dummy numdummies=8 }}} This gives you access to the dummy interfaces ''dummy0'' to ''dummy7'' after bootup. You may now safely remove all ''pre-up'' and ''post-down'' hooks for the dummy interfaces in ''/etc/network/interfaces''. === No network interfaces in domU hosts === One of our domU hosts could not access any network interfaces after the upgrade to xen 3.2. The previously used eth0 to eth2 were simply gone. Maybe this was caused by the new dummy network interface module, that was described in the section above. The basic problem was probably a change of the MAC addresses of the shared network devices. This invalided the dynamically generated content of ''/etc/udev/rules.d/z25_persistent-net.rules'', which is updated only once, when a new interface appears. See http://sysadmin.wikia.com/wiki/Network_error_xen for details. Workaround: * empty the udev MAC assignment file: {{{ echo >/etc/udev/rules.d/z25_persistent-net.rules }}} * restart the domU host * check the order of the new interfaces