Managing the puppet manifest using a vcs is a best practice and there is a lot of material on the web. The easier way to do it, is to use git directly in the directory /etc/puppet and use a simple synchronization strategy with an external repo, either to publish your work, or simply to keep a backup somewhere.

Things are a bit more complicated when you would like to co-administer the machine with multiple people. Setting up user accounts, permission and everything can be a pain in the neck. Moreover working from your desktop is always more comfortable then logging in as root on a remote system and make changes there …

The solution I’ve chosen to make my life a bit easier is to use gitolite, that is a simple git gateway that works using ssh public keys for authentication and does not require the creating of local users on the server machine. Gitolite is available in debian and installing it is super easy : apt-get install gitolite .

If you use puppet already you might be tempted to use puppet to manage your gitolite installation. This is all good, but I don’t advice you to use modules like this one http://forge.puppetlabs.com/gwmngilfen/gitolite/1.0.0 as it’s going to install gitolite from source and on debian it’s not necessary … For my purposes, I didn’t find necessary to manage gitolite with puppet as all the default config options where good enough for me.

Once your debian package is installed, in order to initialize your repo, you just need to pass to gitolite the admin public key, that is your .ssh/id_rsa.pub key and then run this command:

sudo -H -u gitolite gl-setup /tmp/youruser.pub

This will create the admin and testing repo in /var/lib/gitolite/repositories and setup few other things. At this point you are ready to test you gitolite installation by cloning the admin repo :

git clone gitolite@example.org:gitolite-admin.git

Gitolite is engineered to use only the gitolite use to manage all your repositories. To add more repositories and users you should have a look at the documentation and then editing the file conf/gitolite.conf to add your new puppet repository.

At this point, you can go two ways. If you use git to manage your puppet directory, you can just make a copy of it somewhere and then add gitolite as a remote

git remote add origin gitolite@example.org:puppet.git

If you didn’t use git before, you can just copy the manifest in your new git repository, make a first commit and push it on the server.

git push origin master

Every authorized users can now use git to clone your puppet repository, hack, commit, push …

git clone gitolite@example.org:puppet

One last step is to add a small post-receive hook on the server to synchronize your gitolite repository with the puppet directory in /etc : This will sync your main puppet directory and trigger changes on the nodes for the next puppetd run. First I created a small shell script in /usr/local/bin/puppet-post-receive-hook.sh :

#!/bin/bash
umask 0022
cd /etc/puppet
git pull -q origin master

This script presuppose that your git repo in /etc/puppet has the gitolite repo as origin … Then I added a simple hook in the gitolite git repo that calls this script using sudo :

sudo /usr/local/bin/puppet-post-receive-hook.sh

And since you are at it you should also add a pre-commit hook to check the manifest syntax. This will save you a lot of useless commits.

If you use a more complicated puppet setup using environments (I’m not there yet, and I don’t think my setup will evolve in that direction in the near future), you can use puppet-sync that seems a neat script to do the job.

For the moment this setup works pretty well. I’m tempted to explore mcollective to trigger puppet runs on my nodes, but I’m there yet…

Puppet has a built-in functionality to serve small files to its clients. However, for my internal use I sometimes find easier to create a custom debian package to install a specific component then to write a puppet recipe and to copy files around.

To create a local debian repository I use the package reprepro. This is a simple tool that creates and manages apt repository, it is easy to configure and for the moment it lived fully to my expectations.

First of all you need to create a configuration file where you describe your distribution. In this case I choose /var/www/debian/conf/distributions and add the following content :

Origin: PCPool
Label: PCPool
Suite: stable
Codename: pcpool
Version: 3.0
Architectures: i386 amd64
Components: contrib
Description: puppet support package repository
SignWith: D3CF695E

Notice that since reprepro wants to sign your repository, you need to provide a gpg keyid for it.

To add a package to the repository it is straightforward :

reprepro -Vb /var/www/debian/ includedeb pcpool /tmp/msm_1-2_all.deb

As I said, since the repository is signed, we need to make have a way to add the keyid to the known keys of the target machine. In order to achieve this, we add the following puppet recipe :

class apt {
    #local repo sign key
    $keyid = "D3CF695E"

    exec { "apt-update":
        command => "/usr/bin/apt-get update",
        refreshonly => true;
    }

    file { "/etc/apt/trusted.gpg.d/pcpool.gpg":
        source => "puppet://$server/etc/apt/trusted.gpg.d/pcpool.gpg"
    }

#    file { "/root/pcpool.key":
#       source => "puppet://$server/files/root/pcpool.key"
#    }

#    exec { "apt-key":
#        path        => '/bin:/usr/bin',
#        environment => 'HOME=/root',
#        command     => "apt-key add /root/pcpool.key",
#        unless      => "apt-key list | grep $keyid",
#        subscribe   => File["/root/pcpool.key"]
#    }

    file { "/etc/apt/sources.list.d/puppet.list":
        content => "deb http://puppet/debian/ pcpool contrib\n",
        owner   => root,
        group   => root,
        mode    => 0644,
        notify  => Exec["apt-update"]
    }
}

class msm {
    package { "msm": ensure => installed }
}

First we copy the keyid that we have stored in the puppet file bucket in the root directory of the client, then we exec the apt-key command. Note that since puppet executes each action in parallel, we must specify an execution order using the attributes subscribe and notify. Similarly as soon as the file /etc/apt/sources.list.d/puppet.list is added to the machine, we run apt-get update to refresh the cache of apt.

The last stanza simply installs the package that we added to the local repository.

Update

There is a better way to add a gpg key, that is to put it in the /etc/apt/trusted.gpg.d directory. Thanks for the suggestion !

Third post about ganeti.

Ganet-debootstrap-instance contains a nice set of scripts to create a debian (or derivatives) image using debootstrap. Images can be configured and customized by writing simple hooks script to modify various aspects of the default installation. However writing these script is not really fun and pushing it too far can lead to long messy scripts, loosing the overall benefit of automatic configuration.

Puppet is my configuration management tool of choice, but installing puppet on a new machine requires few magic incantations that the user should perform manually, or in a semi automatic mode (autosign=true) to make it work. My goal is to install puppet automatically on the newly created instance so it will run and configure the new instance at the first boot. From that moment on I’ll forget about ganeti and configure all remaining services of my new VM using puppet.

In order to do so, we need to install puppet (and apt-get update/upgrade…), create the ssl certificates for the client and enabling the puppet daemon on the client. We add another hook in /etc/ganeti/instance-debootstrap/hooks/ :

if [ -z "$TARGET" -o ! -d "$TARGET" ]; then
  echo "Missing target directory"
  exit 1
fi

LANG=C
chroot "$TARGET" apt-get -y --force-yes update
chroot "$TARGET" apt-get -y --force-yes upgrade

# install puppet on the client
chroot "$TARGET" apt-get -y --force-yes install puppet

DOMAIN=localnet.org
instance=$INSTANCE_NAME.$DOMAIN

echo "Installing puppet certificates for $instance"
puppetca clean $instance
puppetca -g $instance

mkdir -p $TARGET/etc/puppet
mkdir -p $TARGET/var/lib/puppet/ssl/private_keys/
mkdir -p $TARGET/var/lib/puppet/ssl/certs/

cp /var/lib/puppet/ssl/private_keys/$instance.pem $TARGET/var/lib/puppet/ssl/private_keys/
rm -f $TARGET/var/lib/puppet/ssl/public_keys/$instance.pem

cp /var/lib/puppet/ssl/certs/$instance.pem $TARGET/var/lib/puppet/ssl/certs/
cp /var/lib/puppet/ssl/certs/ca.pem $TARGET/var/lib/puppet/ssl/certs/

chown root. $TARGET/var/lib/puppet/ssl/private_keys/$instance.pem
chmod 0400 $TARGET/var/lib/puppet/ssl/private_keys/$instance.pem

chown root. $TARGET/var/lib/puppet/ssl/certs/$instance.pem
chmod 0640 $TARGET/var/lib/puppet/ssl/certs/$instance.pem

chown root. $TARGET/var/lib/puppet/ssl/certs/ca.pem
chmod 0641 $TARGET/var/lib/puppet/ssl/certs/ca.pem

#echo "server=puppet" >> /etc/puppet/puppet.conf

echo "START=yes" > $TARGET/etc/default/puppet
echo "DAEMON_OPTS=\"\"" >> $TARGET/etc/default/puppet

This script uses puppetca to create on the puppet (and ganeti) server the client key, sign it, and then copy it to the target machine. Notice that we create the certificate for a fqnd name $INSTANCE_NAME.$DOMAIN or otherwise puppet will complain loudly. This is not strictly needed, but if you want to do otherwise, you’ll need to fiddle with the puppet configuration a bit more. The procedure to create a puppet certificate server-side is well documented on the puppet website, so if you are curious about the details duck-duck-it .