On the apt-get installation plan

One important aspect of the mancoosi project is to build a model of the installation process in order to simulate packages upgrades before committing the changes on the machine. This work described here and here (and in upcoming publications) relays on the availability installation plan of the meta-installer (the exact order in which packages will be installed, configured and removed ) to run a simulation of maintainers scripts and check for potential problems accordingly to the model.

Thanks to the help of Julian Andres Klode upstream and maintainer of the python-apt bindings it is now possible to extract the installation plan from apt-get without resorting to debug printout or other hacks.

A forthcoming integration with mpm (Mancoosi-Package-Manager, our testbed meta-installer) will give us the possibility to hook the model checker to the meta-installer and to run it before every upgrade.

The two important changes in the python-apt bindings that will make this possible are the new bindings to the class OrderList and the sub-classing of the class package manager. PackageManager .

Regarding the latter, this is a small example (thanks again Julien !) that you can use to get the installation plan as computed by apt-get. In theory, this will also allow to experiment with different back-ends (like rpm - blah ! :) ) easily without the need to hack the c++ code.

import apt_pkg, sys
import apt

class PkgManager(apt_pkg.PackageManager):

        parent = apt_pkg.PackageManager
        depcache = apt_pkg.DepCache(apt_pkg.Cache())
        installionplan = []

        def install(self, pkg, file):
                #print "Installing", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Inst"))
                return True

        def configure(self, pkg):
                #print "Configuring", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Conf"))
                return True

        def remove(self, pkg, purge):
                #print "Removing", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Rem"))
                return True

        def go(self, fd):
              for (p,a) in self.installionplan :
                  if a == "Inst" or a == "Conf" :
                      ver = self.depcache.get_candidate_ver(p)
                  else :
                      ver = p.current_ver
                  print a, p.name, ver.ver_str, ver.arch

              return True

apt_pkg.PackageManager = PkgManager


cache = apt.Cache()
pkg = cache["python"]
if pkg.is_installed:
        pkg.mark_delete()
else:
        pkg.mark_install()

apt_pkg.config.set("APT::Get::Simulate","true")
apt_pkg.config.set("dir::cache","/tmp")

print "COMMIT"  
cache.commit(install_progress=apt.progress.base.InstallProgress())

And this is the result on my machine :

abate@zed.fr:~$python pkgmanager-test.py 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
COMMIT
Rem alacarte 0.13.2-1 all
Rem apt-listchanges 2.85.7 all
Rem apt-xapian-index 0.41 all
Rem gnome-accessibility 1:2.30+7 all
Rem gok 2.30.0-1 amd64
Rem dasher 4.11-1 amd64
Rem gnome-orca 2.30.2-2 all
Rem python-pyatspi 1.30.1-3 all
Rem at-spi 1.30.1-3 amd64
Rem bluetooth 4.91-1 all
Rem gnome-bluetooth 2.30.0-2 amd64
Rem bluez 4.91-1 amd64
Rem brasero 2.30.3-2 amd64
Rem gthumb 3:2.13.1-1 amd64
Rem libbrasero-media0 2.30.3-2 amd64
Rem brasero-common 2.30.3-2 all
Rem bzr 2.3.1-2 all
Rem gnome-core 1:2.30+7 amd64
Rem gnome-control-center 1:2.30.1-3 amd64
Rem capplets-data 1:2.30.1-3 all
Rem cheese 2.30.1-2 amd64
Rem cheese-common 2.30.1-2 all
Rem cython 0.14.1-5 amd64
Rem dasher-data 4.11-1 all
Rem deluge 1.3.1-1 all
Rem deluge-gtk 1.3.1-1 all
Rem deluge-common 1.3.1-1 all
Rem deskbar-applet 2.32.0-1+b1 amd64
Rem dia 0.97.1-8 amd64
Rem dia-common 0.97.1-8 all
Rem doxygen 1.7.4-1 amd64
Rem doxygen-latex 1.7.4-1 all
Rem dput 0.9.6.2 all
Rem duply 1.5.4.2-1 all
Rem duplicity 0.6.13-1 amd64
Rem edos-debcheck 1.0-9 all
Rem edos-distcheck 1.4.2-12 amd64
Rem eog 2.30.2-1 amd64
Inst libxp6 1:1.0.1-1 amd64
Inst lesstif2 1:0.95.2-1 amd64
Inst xpdf 3.02-12 amd64
Conf libxp6 1:1.0.1-1 amd64
Conf lesstif2 1:0.95.2-1 amd64
Conf xpdf 3.02-12 amd64
[ ... ]

Predicting Upgrade Failures Using Dependency Analysis

The next 16 of april in hannover at the hotswup workshop we’ll present a joint work with Roberto Di Cosmo prepared in the context of the mancoosi project. Since we used debian for our experiments, we’re also very much interested in the feedback from the community regarding our method. However keep in mind that this is still work in progress and to be considered as research more then a proposal for a concrete application. This is part of our on-going effort of exploring different areas with the goal of providing tools and ideas to enhance the quality of FOSS distributions.

Abstract

Upgrades in component based systems can disrupt other components. Being able to predict the possible consequence of an upgrade just by analyzing inter-component dependencies can avoid errors and downtime. In this paper we precisely identify in a repository the components p whose upgrades force a large set of others components to be upgraded. We are also able to discriminate whether all the future versions of p have the same impact, or whether there are different classes of future versions that have different impacts. We perform our analysis on Debian, one of the largest FOSS distributions.

you can find more info about this paper here

Misc live v. 4

The results of the 4th run of the misc competition are [http://mancoosi.org/misc-live/20110225/results/ online !] .

The annoncement sent on the mailing list:

This time, it was quite close, and execution times had to be taken into account several times to break ties. However, looking at the different total times for each solver it looks closer than it actually was. The reason for this is the way the “total success time” is defined (see http://www.mancoosi.org/misc-live/20110225/rules/, section “Breaking Ties”). Since we had many cases without a solution, our definition resulted in the same large constant added to the time of each participant. We’ll have to think about changing this for the next time.

We still have to decide if we’re going to have another edition of misc live before the official annual competition (TBA).

enjoy !

easy cudf parsing in python

With the forth run of Misc live, you might wonder how to you can quickly write a parser for a cudf document. If you are writing your solver in C / C++ , I advice to either grab the legacy ocaml parser and use the C bindings or reuse a parser written by other competitors (all frontends have a FOSS-compatible licence).

If you want to write a dirty and quick frontend in python, maybe the following 10 lines of python might help you:

from itertools import groupby

cnf_fields = ['conflict','depends','provides','recommends']

def cnf(k,s) :
    if k in cnf_fields :
        l = s.split(',')
        ll = map(lambda s : s.split('|'), l)
        return ll
    else :
        return s

records = []
for empty, record in groupby(open("universe.cudf"), key=str.isspace):
  if not empty:
    l = map(lambda s : s.split(': '), record)
    # we ignore the preamble here ...
    if 'preamble' not in l[0] :
        pairs = ([k, cnf(k,v.strip())] for k,v in l)
        records.append(dict(pairs))

for i in records :
    print i

we use the function groupby from itertools to create a list of stanzas and then we just trasfrom each of them in a dictionary that should be pretty easy to manipulate. We ignore the preamble, but adding support for it should be straigthforward… I got the idea from this forum post.

the result :

#python cudf.py
{'recommends': [['perl-modules '], [' libio-socket-inet6-perl']], 'package': '2ping', 'replaces': '', 'number': '1.0-1', 'sourceversion': '1.0-1', 'source': '2ping', 'depends': [['perl']], 'version': '4806', 'architecture': 'all', 'conflicts': '2ping'}0.5-3', 'source': '2vcard', 'version': '1523', 'architecture': 'all', 'conflicts': '2vcard', 'recommends': [['true!']]}'package': '3270-common', 'number': '3.3.10ga4-2', 'sourceversion': '3.3.10ga4-2', 'source': 'ibm-3270', 'depends': [['libc6 >= 9784 '], [' libssl0.9.8 >= 2840']], 'version': '11009', 'architecture': 'amd64', 'conflicts': '3270-common', 'recommends': [['true!']]}chess', 'depends': [['libc6 >= 9578 '], [' libx11-6 '], [' libxext6 '], [' libxmu6 '], [' libxpm4 '], [' libxt6 '], [' xaw3dg >= 6582']], 'version': '2409', 'architecture': 'amd64', 'conflicts': '3dchess', 'recommends': [['true!']]} [' libxpm4 '], [' libxt6 '], [' xaw3dg >= 6582']], 'version': '2410', 'architecture': 'amd64', 'conflicts': '3dchess', 'recommends': [['true!']]}6 >= 8923 '], [' libfreetype6 >= 8856 '], [' libftgl2 >= 8661 '], [' libgcc1 >= 14906 '], [' libgl1-mesa-glx ', ' libgl1--virtual ', ' libgl1 '], [' libglu1-mesa ', ' libglu1--virtual ', ' libglu1 '], [' libgomp1 >= 11829 '], [' libmgl5 '], [' libpng12-0 >= 5996 '], [' libstdc++6 >= 11843 '], [' libwxbase2.8-0 >= 9714 '], [' libwxgtk2.8-0 >= 9714 '], [' libxml2 >= 9624 '], [' zlib1g >= 14223']], 'version': '116', 'architecture': 'amd64', 'conflicts': '3depict', 'recommends': [['true!']]} '], [' libstdc++6 >= 11664 '], [' libwxbase2.8-0 >= 9714 '], [' libwxgtk2.8-0 >= 9714 '], [' libxml2 >= 9624 '], [' zlib1g >= 14223']], 'version': '138', 'architecture': 'amd64', 'conflicts': '3depict', 'recommends': [['true!']]}': '14987', 'architecture': 'amd64', 'conflicts': '9base', 'recommends': [['true!']]}.8-5', 'sourceversion': '1.8-5', 'source': '9menu', 'depends': [['libc6 >= 8923 '], [' libx11-6']], 'version': '7010', 'architecture': 'amd64', 'conflicts': '9menu', 'recommends': [['true!']]}sion': '1.2-9', 'source': '9wm', 'depends': [['libc6 >= 9578 '], [' libx11-6 '], [' libxext6']], 'version': '5712', 'architecture': 'amd64', 'provides': [['x-window-manager--virtual']], 'conflicts': '9wm', 'recommends': [['true!']]}
{'replaces': '', 'package': 'abook', 'number': '0.5.6-7+b1', 'sourceversion': '0.5.6-7', 'source': 'abook', 'depends': [['libc6 >= 9022 '], [' libncursesw5 >= 12348 '], [' libreadline5 >= 12239 '], [' debconf >= 1510 ', ' debconf-2.0--virtual ', ' debconf-2.0']], 'version': '1712', 'architecture': 'amd64', 'conflicts': 'abook', 'recommends': [['true!']]}
...

update

Maybe a small example of the input file would help :)

package: m4
version: 3
depends: libc6 >= 8

package: openssl
version: 11
depends: libc6 >= 18, libssl0.9.8 >= 8, zlib1g >= 1
conflicts: ssleay < 1