One important aspect of the mancoosi project is to build a model of the installation process in order to simulate packages upgrades before committing the changes on the machine. This work described here and here (and in upcoming publications) relays on the availability installation plan of the meta-installer (the exact order in which packages will be installed, configured and removed ) to run a simulation of maintainers scripts and check for potential problems accordingly to the model.

Thanks to the help of Julian Andres Klode upstream and maintainer of the python-apt bindings it is now possible to extract the installation plan from apt-get without resorting to debug printout or other hacks.

A forthcoming integration with mpm (Mancoosi-Package-Manager, our testbed meta-installer) will give us the possibility to hook the model checker to the meta-installer and to run it before every upgrade.

The two important changes in the python-apt bindings that will make this possible are the new bindings to the class OrderList and the sub-classing of the class package manager. PackageManager .

Regarding the latter, this is a small example (thanks again Julien !) that you can use to get the installation plan as computed by apt-get. In theory, this will also allow to experiment with different back-ends (like rpm - blah ! :) ) easily without the need to hack the c++ code.

import apt_pkg, sys
import apt

class PkgManager(apt_pkg.PackageManager):

        parent = apt_pkg.PackageManager
        depcache = apt_pkg.DepCache(apt_pkg.Cache())
        installionplan = []

        def install(self, pkg, file):
                #print "Installing", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Inst"))
                return True

        def configure(self, pkg):
                #print "Configuring", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Conf"))
                return True

        def remove(self, pkg, purge):
                #print "Removing", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Rem"))
                return True

        def go(self, fd):
              for (p,a) in self.installionplan :
                  if a == "Inst" or a == "Conf" :
                      ver = self.depcache.get_candidate_ver(p)
                  else :
                      ver = p.current_ver
                  print a, p.name, ver.ver_str, ver.arch

              return True

apt_pkg.PackageManager = PkgManager


cache = apt.Cache()
pkg = cache["python"]
if pkg.is_installed:
        pkg.mark_delete()
else:
        pkg.mark_install()

apt_pkg.config.set("APT::Get::Simulate","true")
apt_pkg.config.set("dir::cache","/tmp")

print "COMMIT"  
cache.commit(install_progress=apt.progress.base.InstallProgress())

And this is the result on my machine :

abate@zed.fr:~$python pkgmanager-test.py 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
COMMIT
Rem alacarte 0.13.2-1 all
Rem apt-listchanges 2.85.7 all
Rem apt-xapian-index 0.41 all
Rem gnome-accessibility 1:2.30+7 all
Rem gok 2.30.0-1 amd64
Rem dasher 4.11-1 amd64
Rem gnome-orca 2.30.2-2 all
Rem python-pyatspi 1.30.1-3 all
Rem at-spi 1.30.1-3 amd64
Rem bluetooth 4.91-1 all
Rem gnome-bluetooth 2.30.0-2 amd64
Rem bluez 4.91-1 amd64
Rem brasero 2.30.3-2 amd64
Rem gthumb 3:2.13.1-1 amd64
Rem libbrasero-media0 2.30.3-2 amd64
Rem brasero-common 2.30.3-2 all
Rem bzr 2.3.1-2 all
Rem gnome-core 1:2.30+7 amd64
Rem gnome-control-center 1:2.30.1-3 amd64
Rem capplets-data 1:2.30.1-3 all
Rem cheese 2.30.1-2 amd64
Rem cheese-common 2.30.1-2 all
Rem cython 0.14.1-5 amd64
Rem dasher-data 4.11-1 all
Rem deluge 1.3.1-1 all
Rem deluge-gtk 1.3.1-1 all
Rem deluge-common 1.3.1-1 all
Rem deskbar-applet 2.32.0-1+b1 amd64
Rem dia 0.97.1-8 amd64
Rem dia-common 0.97.1-8 all
Rem doxygen 1.7.4-1 amd64
Rem doxygen-latex 1.7.4-1 all
Rem dput 0.9.6.2 all
Rem duply 1.5.4.2-1 all
Rem duplicity 0.6.13-1 amd64
Rem edos-debcheck 1.0-9 all
Rem edos-distcheck 1.4.2-12 amd64
Rem eog 2.30.2-1 amd64
Inst libxp6 1:1.0.1-1 amd64
Inst lesstif2 1:0.95.2-1 amd64
Inst xpdf 3.02-12 amd64
Conf libxp6 1:1.0.1-1 amd64
Conf lesstif2 1:0.95.2-1 amd64
Conf xpdf 3.02-12 amd64
[ ... ]

The next 16 of april in hannover at the hotswup workshop we’ll present a joint work with Roberto Di Cosmo prepared in the context of the mancoosi project. Since we used debian for our experiments, we’re also very much interested in the feedback from the community regarding our method. However keep in mind that this is still work in progress and to be considered as research more then a proposal for a concrete application. This is part of our on-going effort of exploring different areas with the goal of providing tools and ideas to enhance the quality of FOSS distributions.

Abstract

Upgrades in component based systems can disrupt other components. Being able to predict the possible consequence of an upgrade just by analyzing inter-component dependencies can avoid errors and downtime. In this paper we precisely identify in a repository the components p whose upgrades force a large set of others components to be upgraded. We are also able to discriminate whether all the future versions of p have the same impact, or whether there are different classes of future versions that have different impacts. We perform our analysis on Debian, one of the largest FOSS distributions.

you can find more info about this paper here

• Attachment 0

A while ago I received a new desktop machine (8 cores, 8Gb of memory …) at work. Since for the moment I kinda happy to work on my laptop using an external screen, I decided to put the hw to a good use and to explore a bit more some more exotic (at least for me) xen features.

In particular I spend half a day playing with different xen network settings. The bridge model, that should work out of the shelf, is the easiest one. To setup this up, you basically need to specify a couple of options in the xend-config file and you’re done. This is the “default” network configuration and is should work out of the box in most situations. Using this method, since all VMs’ interfaces are bridged together (surprise !) with the public interface, your network card is left in promiscuous mode (not a big problem if you ask me…). Once your VMs are up, you can then decided to use your default dhcp server, autoconf your VMs with ipv6 only, or do nothing as you please.

An other popular method, albeit a bit more complex, is to setup a natted network using the script network-nat (this one is an evolution of the third method that is ‘network-routed’) . I played with it, but since I wanted to have all my DomU on the same subnet, this setup wasn’t satisfying for me. In particular, by default, ‘network-nat’ assigns a different subnet to each DomU. Using the natted set up you can also configure a local dhcp server to give private IPs to your VMs all done transparently by the xen network scripts. I’ve noticed that there is a bug in the xen script that does not make it very squeeze friendly. Since the default dhcp server in squeeze is isc-dhcp and few configuration files got shuffled in the process (notably /etc/dhcp3.conf is not /etc/dhcp/dhcp.conf) , the script needs a little fix to work properly. I’ll report this bug sometimes soon…

Goggling around I found a different setup that is called brouter, that is a hybrid between a bridge configuration and a routed configuration. This is the original (??) article well hidden in an old suse wiki.

I’ve done few modifications here to add natting. So basically, all virtual interfaces connected each to one DomU are linked together by a bridge (xenbr0). The bridge with address 10.0.0.1 is also the router of the subnet. All DomU are configured to used dhcp that assigns a new ip and specifies the router of the subnet. The dhcp server is configured to answer requests only on the xenb0 interface avoiding problems on the public network…

routing is configured using iptables :

    iptables -t nat -A POSTROUTING -o ${netdev} -j MASQUERADE
    iptables -A FORWARD -i ${bridge} -j ACCEPT
    echo 1 >/proc/sys/net/ipv4/ip_forward
    /etc/init.d/isc-dhcp-server restart

Note that since the dhcp server is configured to give addresses only on the virtual network, we need to restart it after creating the bridge interface, otherwise isc-dhcp-server will refuse to run. Mum says that I should configure the bridge in /etc/network/interfaces to make the dhcp server happy at startup, but I felt a bit lazy, so I let to task to xen…

In the next episode, I’ll add ipv6 connectivity to the virtual subnet and then start playing with puppet… ipv6 is almost done, puppet… I started with the doc…

The complete script from the suse wiki and with my modifications is below (only lightly tested):

#!/bin/sh
#============================================================================
# Default Xen network start/stop script.
# Xend calls a network script when it starts.
# The script name to use is defined in /etc/xen/xend-config.sxp
# in the network-script field.
#
# This script creates a bridge (default xenbr${vifnum}), gives it an IP address
# and the appropriate route. Then it starts the SuSEfirewall2 which should have
# the bridge device in the zone you want it.
#
# If all goes well, this should ensure that networking stays up.
# However, some configurations are upset by this, especially
# NFS roots. If the bridged setup does not meet your needs,
# configure a different script, for example using routing instead.
#
# Usage:
#
# vnet-brouter (start|stop|status) {VAR=VAL}*
#
# Vars:
#
# bridgeip   Holds the ip address the bridge should have in the
#            the form ip/mask (10.0.0.1/24).
# brnet      Holds the network of the bridge (10.0.0.1/24).
# 
# vifnum     Virtual device number to use (default 0). Numbers >=8
#            require the netback driver to have nloopbacks set to a
#            higher value than its default of 8.
# bridge     The bridge to use (default xenbr${vifnum}).
#
# start:
# Creates the bridge
# Gives it the IP address and netmask
# Adds the routes to the routing table.
#
# stop:
# Removes all routes from the bridge
# Removes any devices on the bridge from it.
# Deletes bridge
#
# status:
# Print addresses, interfaces, routes
#
#============================================================================

#set -x

dir=$(dirname "$0")
. "$dir/xen-script-common.sh"
. "$dir/xen-network-common.sh"

findCommand "$@"
evalVariables "$@"

vifnum=${vifnum:-0}
bridgeip=${bridgeip:-10.6.7.1/24}
brnet=${brnet:-10.6.7.0/24}
netmask=${netmask:-255.255.255.0}
bridge=${bridge:-xenbr${vifnum}}

##
# link_exists interface
#
# Returns 0 if the interface named exists (whether up or down), 1 otherwise.
#
link_exists()
{
    if ip link show "$1" >/dev/null 2>/dev/null
    then
        return 0
    else
        return 1
    fi
}


# Usage: create_bridge bridge
create_bridge () {
    local bridge=$1

    # Don't create the bridge if it already exists.
    if [ ! -d "/sys/class/net/${bridge}/bridge" ]; then
        brctl addbr ${bridge}
        brctl stp ${bridge} off
        brctl setfd ${bridge} 0
    fi
    ip link set ${bridge} up
}

# Usage: add_to_bridge bridge dev
add_to_bridge () {
    local bridge=$1
    local dev=$2
    # Don't add $dev to $bridge if it's already on a bridge.
    if ! brctl show | grep -wq ${dev} ; then
        brctl addif ${bridge} ${dev}
    fi
}

# Usage: show_status dev bridge
# Print interface configuration and routes.
show_status () {
    local dev=$1
    local bridge=$2

    echo '============================================================'
    ip addr show ${dev}
    ip addr show ${bridge}
    echo ' '
    brctl show ${bridge}
    echo ' '
    ip route list
    echo ' '
    route -n
    echo '============================================================'
    echo ' '
    iptables -L
    echo ' '
    iptables -L -t nat
    echo '============================================================'

}

op_start () {
    if [ "${bridge}" = "null" ] ; then
        return
    fi

    create_bridge ${bridge}

    if link_exists "$bridge"; then
        ip address add dev $bridge $bridgeip
        ip link set ${bridge} up arp on
        ip route add to $brnet dev $bridge
    fi

    if [ ${antispoof} = 'yes' ] ; then
        antispoofing
    fi
    iptables -t nat -A POSTROUTING -o ${netdev} -j MASQUERADE
    iptables -A FORWARD -i ${bridge} -j ACCEPT
    echo 1 >/proc/sys/net/ipv4/ip_forward
    /etc/init.d/isc-dhcp-server restart
}

op_stop () {
    if [ "${bridge}" = "null" ]; then
        return
    fi
    if ! link_exists "$bridge"; then
        return
    fi

    ip route del to $brnet dev $bridge
    ip link set ${bridge} down arp off
    ip address del dev $bridge $bridgeip
    ##FIXME: disconnect the interfaces from the bridge 1st
    brctl delbr ${bridge}
    /etc/init.d/isc-dhcp-server restart
}

case "$command" in
    start)
        op_start
        ;;

    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
        ;;

    *)
        echo "Unknown command: $command" >&2
        echo 'Valid commands are: start, stop, status' >&2
        exit 1
esac

For one of our experiments, we ended up analyzing all versions mentioned in a Debian Packages file if mentioned in a constraint or in a version field of a package. It seems that there are a lot of DDs that like to use strange ‘formatting’ when writing down a versioned constraint. This is might be not a choice done directly by the debian maintainer, but just a consequence of a particular version schema from upstream.

The debian policy is rigorous with relation to the algorithm to compare version .

The strings are compared from left to right.

First the initial part of each string consisting entirely of non-digit characters is determined. These two parts (one of which may be empty) are compared lexically. If a difference is found it is returned. The lexical comparison is a comparison of ASCII values modified so that all the letters sort earlier than all the non-letters and so that a tilde sorts before anything, even the end of a part. For example, the following parts are in sorted order from earliest to latest: ~~, ~~a, ~, the empty part, a.[34]

Then the initial part of the remainder of each string which consists entirely of digit characters is determined. The numerical values of these two parts are compared, and any difference found is returned as the result of the comparison. For these purposes an empty string (which can only occur at the end of one or both version strings being compared) counts as zero.

These two steps (comparing and removing initial non-digit strings and initial digit strings) are repeated until a difference is found or both strings are exhausted.

I think the important part is about the numerical comparison The numerical values of these two parts are compared, and any difference found is returned as the result of the comparison .

Our tool finds a number of examples of equivalent versions : version 0.1-1 is equivalent to (0.00001-1,0.001-1,0.01-1,0.1-1) version 2.1.0-1 is equivalent to (2.01.00-1,2.1.00-1,2.1.0-1) * … etc . I’ve one page full of equivalent versions.

For example version ‘0.00001-1’ is from package libbenchmark-progressbar-perl. I don’t know why so many equivalent way of writing a version string are used. Probably they appear in completely unrelated packages I’m pretty sure no harm is intended :) However, to avoid confusion, it might be a good idea to settle on a non-normative canonical representation of versions. Like no ‘0’ in front of dots and hyphen… Maybe we could add a warning in lintian ?

If somebody is interested I can generate a full report that associates each version to a package name.

How to create a debian installer for a usb pen drive ? There are many way.s from a simple sudo zcat boot.img.gz > /dev/sdb to unetbootin.

Last night I discovered a very nice tool from ubuntu (usb-creator) that apparently is still not in the debian archives. There is a bit of doco here if you don’t know it already.

Installing it on debian is a breeze (I used the debs from lucid) and it works pretty well. The reason I’ve tried this one is that unetbootin sometimes failed for me producing an un-usable usb key. The old school method of course works, but sometimes, late at night, I feel the need of a friendly gtk interface to help my sleep.

Debs : usb-creator-gtk usb-creator-common