I’ll start here a small series of posts about ganeti, xen and puppet.
For my work I run few servers sitting on xen and it has always been a
bit of a pain to create a new instance and keep it up to date. Up to
now I’ve used the excellent xen-create-image tool to create my VMs,
but I wanted to try something new and more sexy… Last week I finally
found some time to learn (and a spare box to run my experiments) how
to use ganeti. Ganeti is the only
tool I tried out, but it seems to fit the bill for my use and it seems
polished and mature project to me… Moreover I’ve seen a presentation
about it in every FLOSS conference I’ve attended in the last few years
and I thought it was time to give it a try.
Installing and configuring ganeti is fairly easy, there is a lot of
documentation available and this post is not going to be about
installing it, but rather how to create a new bare instance with
ganeti-deboostrap-instance. There is also a way to create a new
instance from an image, but I didn’t go that way yet.
This first post is about the first problem I’ve encountered, that is,
how to automatically assign a network address and a name to each new
instance created by gnt-instance add. Since all my instances should
be able to communicate together on the same subnet, I’ve decided to
configure xen to create a NATted private network and add each new
instance to this network.
The first step is to create an interface in /etc/network/interfaces .
auto xen-br0
iface xen-br0 inet static
address 10.0.0.1
netmask 255.255.255.0
bridge_stp off
bridge_fd 0
bridge_ports none
This is the standard debian way but since xen uses a different naming
convention (here I’m using ganeti naming convention xenbr0 vs
xen-br0), I need to convince tell xen what I intend to do by adding
these lines in /etc/xen/xend.config :
(network-script 'network-virtual bridgeip="10.0.0.1/24" brnet="10.0.0.0/24" bridge="xen-br0"')
(vif-script vif-bridge)
Next I have to connect my real network interface to the private
network using few iptables rules in /etc/rc.local (probably there is
a better place to do this…):
echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o xen-br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i xen-br0 -o eth0 -j ACCEPT
The xen setup is complete and every new image should have a vif
connected to the subbet 10.0.0.0. The xen setup corresponds to the
physical wiring of the network. The next step is to configure each
instance so to allow them to communicate on this subnet. Since I build
my VMs using ganeti-debootstrap-instance, and by default
debootstrap does not configure the network, we need to add a new hook
in the directory /etc/ganeti/instance-debootstrap/hooks.
#!/bin/bash
if [ -z "$TARGET" -o ! -d "$TARGET" ]; then
echo "Missing target directory"
exit 1
fi
if [ ! -d "$TARGET/etc/network" ]; then
echo "Missing target network directory"
exit 1
fi
if [ -z "$NIC_COUNT" ]; then
echo "Missing NIC COUNT"
exit 1
fi
if [ "$NIC_COUNT" -gt 0 ]; then
cat > $TARGET/etc/network/interfaces <<EOF
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp
EOF
fi
DAEMON_PID_FILES="/var/run/dnsmasq.pid /var/run/dnsmasq/dnsmasq.pid"
instance=$INSTANCE_NAME
[ -n "$instance" ] || exit 1
nic_count=$((NIC_COUNT - 1))
mac_var="NIC_${nic_count}_MAC"
echo $mac_var
echo $nic_count
mac=${!mac_var}
echo $mac
echo "dhcp-host=$mac,$instance" > /etc/dnsmasq.d/$instance.conf
This hook will do two things. First it will configure the interfaces
of the new instance to get configured using dhcp, second, it will add
an entry to the dnsmasq configuration file to make this instance known
to the world. This basically boils down to add a file in
/etc/dnsmasq.d/ with the mac address of the new instance and its
designated name. Dnsmasq will then provide an ip address for this
instance and add it to the dns.
dhcp-host=aa:00:00:24:6c:8a,node1
Configuring dnsmasq is pretty easy as well. First I want it to answer
dhcp queries only on the internal network, second I want to configure
my clients passing 10.0.0.1 as nameserver and gataway. You can just
add the following lines in /etc/dnsmasq.d/general to get it going.
interface=xen-br0
interface=lo
dhcp-range=10.0.0.128,10.0.0.250
domain=localnet.org,10.0.0.128,10.0.0.250
dhcp-option=3,10.0.0.1
bogus-priv
#expand-hosts
local=/localnet.org/
To create your new instance you can just run the following command :
gnt-instance add -t plain -s 5g -B memory=1024 -o debootstrap+unstable --no-ip-check --no-name-check node1
If you are running your dom0 on debian squeeze before running this
command you should configure ganeti to pass the right xen parameters
to the newly created instance :
gnt-cluster modify --hypervisor-parameter xen-pvm:root_path='/dev/xvda1'
gnt-cluster modify --hypervisor-parameter xen-pvm:initrd_path='/boot/initrd-2.6-xenU'
I use —no-ip-check and —no-name-check to skip ip and dns
check performed by ganeti and to avoid a sort of chicken-egg problem,
where the name and address of this new instance is yet unknown to
dnsmasq and that node1 is the name that will be used by the hook to
add an entry in the dnsmasq configuration. debootstrap+unstable is a
variant of the default configuration and you need to add it to the
list of variants used by ganeti-deboostrap-instance.
That should be it. The new instance should come up with a dynamically
assigned ip address, able to talk to the outside world and
automatically known by all the other machine on the subnet via dns.
Next post will be about how to add a swap hook for ganete-debootstrap-istance.