how to create VMs with ganeti / xen and dnsmasq
I’ll start here a small series of posts about ganeti, xen and puppet. For my work I run few servers sitting on xen and it has always been a bit of a pain to create a new instance and keep it up to date. Up to now I’ve used the excellent xen-create-image tool to create my VMs, but I wanted to try something new and more sexy… Last week I finally found some time to learn (and a spare box to run my experiments) how to use ganeti. Ganeti is the only tool I tried out, but it seems to fit the bill for my use and it seems polished and mature project to me… Moreover I’ve seen a presentation about it in every FLOSS conference I’ve attended in the last few years and I thought it was time to give it a try.
Installing and configuring ganeti is fairly easy, there is a lot of documentation available and this post is not going to be about installing it, but rather how to create a new bare instance with ganeti-deboostrap-instance. There is also a way to create a new instance from an image, but I didn’t go that way yet.
This first post is about the first problem I’ve encountered, that is, how to automatically assign a network address and a name to each new instance created by gnt-instance add. Since all my instances should be able to communicate together on the same subnet, I’ve decided to configure xen to create a NATted private network and add each new instance to this network.
The first step is to create an interface in /etc/network/interfaces .
auto xen-br0
iface xen-br0 inet static
address 10.0.0.1
netmask 255.255.255.0
bridge_stp off
bridge_fd 0
bridge_ports none
This is the standard debian way but since xen uses a different naming convention (here I’m using ganeti naming convention xenbr0 vs xen-br0), I need to convince tell xen what I intend to do by adding these lines in /etc/xen/xend.config :
(network-script 'network-virtual bridgeip="10.0.0.1/24" brnet="10.0.0.0/24" bridge="xen-br0"')
(vif-script vif-bridge)
Next I have to connect my real network interface to the private network using few iptables rules in /etc/rc.local (probably there is a better place to do this…):
echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o xen-br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i xen-br0 -o eth0 -j ACCEPT
The xen setup is complete and every new image should have a vif connected to the subbet 10.0.0.0. The xen setup corresponds to the physical wiring of the network. The next step is to configure each instance so to allow them to communicate on this subnet. Since I build my VMs using ganeti-debootstrap-instance, and by default debootstrap does not configure the network, we need to add a new hook in the directory /etc/ganeti/instance-debootstrap/hooks.
#!/bin/bash
if [ -z "$TARGET" -o ! -d "$TARGET" ]; then
echo "Missing target directory"
exit 1
fi
if [ ! -d "$TARGET/etc/network" ]; then
echo "Missing target network directory"
exit 1
fi
if [ -z "$NIC_COUNT" ]; then
echo "Missing NIC COUNT"
exit 1
fi
if [ "$NIC_COUNT" -gt 0 ]; then
cat > $TARGET/etc/network/interfaces <<EOF
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp
EOF
fi
DAEMON_PID_FILES="/var/run/dnsmasq.pid /var/run/dnsmasq/dnsmasq.pid"
instance=$INSTANCE_NAME
[ -n "$instance" ] || exit 1
nic_count=$((NIC_COUNT - 1))
mac_var="NIC_${nic_count}_MAC"
echo $mac_var
echo $nic_count
mac=${!mac_var}
echo $mac
echo "dhcp-host=$mac,$instance" > /etc/dnsmasq.d/$instance.conf
This hook will do two things. First it will configure the interfaces of the new instance to get configured using dhcp, second, it will add an entry to the dnsmasq configuration file to make this instance known to the world. This basically boils down to add a file in /etc/dnsmasq.d/ with the mac address of the new instance and its designated name. Dnsmasq will then provide an ip address for this instance and add it to the dns.
dhcp-host=aa:00:00:24:6c:8a,node1
Configuring dnsmasq is pretty easy as well. First I want it to answer dhcp queries only on the internal network, second I want to configure my clients passing 10.0.0.1 as nameserver and gataway. You can just add the following lines in /etc/dnsmasq.d/general to get it going.
interface=xen-br0
interface=lo
dhcp-range=10.0.0.128,10.0.0.250
domain=localnet.org,10.0.0.128,10.0.0.250
dhcp-option=3,10.0.0.1
bogus-priv
#expand-hosts
local=/localnet.org/
To create your new instance you can just run the following command :
gnt-instance add -t plain -s 5g -B memory=1024 -o debootstrap+unstable --no-ip-check --no-name-check node1
If you are running your dom0 on debian squeeze before running this command you should configure ganeti to pass the right xen parameters to the newly created instance :
gnt-cluster modify --hypervisor-parameter xen-pvm:root_path='/dev/xvda1'
gnt-cluster modify --hypervisor-parameter xen-pvm:initrd_path='/boot/initrd-2.6-xenU'
I use —no-ip-check and —no-name-check to skip ip and dns check performed by ganeti and to avoid a sort of chicken-egg problem, where the name and address of this new instance is yet unknown to dnsmasq and that node1 is the name that will be used by the hook to add an entry in the dnsmasq configuration. debootstrap+unstable is a variant of the default configuration and you need to add it to the list of variants used by ganeti-deboostrap-instance.
That should be it. The new instance should come up with a dynamically assigned ip address, able to talk to the outside world and automatically known by all the other machine on the subnet via dns.
Next post will be about how to add a swap hook for ganete-debootstrap-istance.