A while ago I received a new desktop machine (8 cores, 8Gb of memory …) at work. Since for the moment I kinda happy to work on my laptop using an external screen, I decided to put the hw to a good use and to explore a bit more some more exotic (at least for me) xen features.
In particular I spend half a day playing with different xen network settings. The bridge model, that should work out of the shelf, is the easiest one. To setup this up, you basically need to specify a couple of options in the xend-config file and you’re done. This is the “default” network configuration and is should work out of the box in most situations. Using this method, since all VMs’ interfaces are bridged together (surprise !) with the public interface, your network card is left in promiscuous mode (not a big problem if you ask me…). Once your VMs are up, you can then decided to use your default dhcp server, autoconf your VMs with ipv6 only, or do nothing as you please.
An other popular method, albeit a bit more complex, is to setup a natted network using the script network-nat (this one is an evolution of the third method that is ‘network-routed’) . I played with it, but since I wanted to have all my DomU on the same subnet, this setup wasn’t satisfying for me. In particular, by default, ‘network-nat’ assigns a different subnet to each DomU. Using the natted set up you can also configure a local dhcp server to give private IPs to your VMs all done transparently by the xen network scripts. I’ve noticed that there is a bug in the xen script that does not make it very squeeze friendly. Since the default dhcp server in squeeze is isc-dhcp and few configuration files got shuffled in the process (notably /etc/dhcp3.conf is not /etc/dhcp/dhcp.conf) , the script needs a little fix to work properly. I’ll report this bug sometimes soon…
Goggling around I found a different setup that is called brouter, that is a hybrid between a bridge configuration and a routed configuration. This is the original (??) article well hidden in an old suse wiki.
I’ve done few modifications here to add natting. So basically, all virtual interfaces connected each to one DomU are linked together by a bridge (xenbr0). The bridge with address 10.0.0.1 is also the router of the subnet. All DomU are configured to used dhcp that assigns a new ip and specifies the router of the subnet. The dhcp server is configured to answer requests only on the xenb0 interface avoiding problems on the public network…
routing is configured using iptables :
iptables -t nat -A POSTROUTING -o ${netdev} -j MASQUERADE
iptables -A FORWARD -i ${bridge} -j ACCEPT
echo 1 >/proc/sys/net/ipv4/ip_forward
/etc/init.d/isc-dhcp-server restart
Note that since the dhcp server is configured to give addresses only on the virtual network, we need to restart it after creating the bridge interface, otherwise isc-dhcp-server will refuse to run. Mum says that I should configure the bridge in /etc/network/interfaces to make the dhcp server happy at startup, but I felt a bit lazy, so I let to task to xen…
In the next episode, I’ll add ipv6 connectivity to the virtual subnet and then start playing with puppet… ipv6 is almost done, puppet… I started with the doc…
The complete script from the suse wiki and with my modifications is below (only lightly tested):
#!/bin/sh
#============================================================================
# Default Xen network start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates a bridge (default xenbr${vifnum}), gives it an IP address
# and the appropriate route. Then it starts the SuSEfirewall2 which should have
# the bridge device in the zone you want it.
#
# If all goes well, this should ensure that networking stays up.
# However, some configurations are upset by this, especially
# NFS roots. If the bridged setup does not meet your needs,
# configure a different script, for example using routing instead.
#
# Usage:
#
# vnet-brouter (start|stop|status) {VAR=VAL}*
#
# Vars:
#
# bridgeip Holds the ip address the bridge should have in the
# the form ip/mask (10.0.0.1/24).
# brnet Holds the network of the bridge (10.0.0.1/24).
#
# vifnum Virtual device number to use (default 0). Numbers >=8
# require the netback driver to have nloopbacks set to a
# higher value than its default of 8.
# bridge The bridge to use (default xenbr${vifnum}).
#
# start:
# Creates the bridge
# Gives it the IP address and netmask
# Adds the routes to the routing table.
#
# stop:
# Removes all routes from the bridge
# Removes any devices on the bridge from it.
# Deletes bridge
#
# status:
# Print addresses, interfaces, routes
#
#============================================================================
#set -x
dir=$(dirname "$0")
. "$dir/xen-script-common.sh"
. "$dir/xen-network-common.sh"
findCommand "$@"
evalVariables "$@"
vifnum=${vifnum:-0}
bridgeip=${bridgeip:-10.6.7.1/24}
brnet=${brnet:-10.6.7.0/24}
netmask=${netmask:-255.255.255.0}
bridge=${bridge:-xenbr${vifnum}}
##
# link_exists interface
#
# Returns 0 if the interface named exists (whether up or down), 1 otherwise.
#
link_exists()
{
if ip link show "$1" >/dev/null 2>/dev/null
then
return 0
else
return 1
fi
}
# Usage: create_bridge bridge
create_bridge () {
local bridge=$1
# Don't create the bridge if it already exists.
if [ ! -d "/sys/class/net/${bridge}/bridge" ]; then
brctl addbr ${bridge}
brctl stp ${bridge} off
brctl setfd ${bridge} 0
fi
ip link set ${bridge} up
}
# Usage: add_to_bridge bridge dev
add_to_bridge () {
local bridge=$1
local dev=$2
# Don't add $dev to $bridge if it's already on a bridge.
if ! brctl show | grep -wq ${dev} ; then
brctl addif ${bridge} ${dev}
fi
}
# Usage: show_status dev bridge
# Print interface configuration and routes.
show_status () {
local dev=$1
local bridge=$2
echo '============================================================'
ip addr show ${dev}
ip addr show ${bridge}
echo ' '
brctl show ${bridge}
echo ' '
ip route list
echo ' '
route -n
echo '============================================================'
echo ' '
iptables -L
echo ' '
iptables -L -t nat
echo '============================================================'
}
op_start () {
if [ "${bridge}" = "null" ] ; then
return
fi
create_bridge ${bridge}
if link_exists "$bridge"; then
ip address add dev $bridge $bridgeip
ip link set ${bridge} up arp on
ip route add to $brnet dev $bridge
fi
if [ ${antispoof} = 'yes' ] ; then
antispoofing
fi
iptables -t nat -A POSTROUTING -o ${netdev} -j MASQUERADE
iptables -A FORWARD -i ${bridge} -j ACCEPT
echo 1 >/proc/sys/net/ipv4/ip_forward
/etc/init.d/isc-dhcp-server restart
}
op_stop () {
if [ "${bridge}" = "null" ]; then
return
fi
if ! link_exists "$bridge"; then
return
fi
ip route del to $brnet dev $bridge
ip link set ${bridge} down arp off
ip address del dev $bridge $bridgeip
##FIXME: disconnect the interfaces from the bridge 1st
brctl delbr ${bridge}
/etc/init.d/isc-dhcp-server restart
}
case "$command" in
start)
op_start
;;
stop)
op_stop
;;
status)
show_status ${netdev} ${bridge}
;;
*)
echo "Unknown command: $command" >&2
echo 'Valid commands are: start, stop, status' >&2
exit 1
esac