MPM: A Modular Package Manager

One of our paper has been accepted to The 14th International ACM SIGSOFT Symposium on Component Based Software Engineering (CBSE-2011) happening in June in Boulder, Colorado, USA. This is a joint work with Roberto Di Cosmo, Ralf Treinen and Stefano Zacchiroli.

Abstract

Software distributions in the FOSS world rely on so-called package managers for the installation and removal of packages on target machines. State-of-the-art package managers are monolithic in architecture, and each of them is hard-wired to an ad-hoc dependency solver implementing a customized heuristics. In this paper we propose a modular architecture allowing for pluggable dependency solvers and backends. We argue that this is the path that leads to the next generation of package managers that will deliver better results, accept more expressive input languages, and can be easily adaptable to new platforms. We present a working prototype—-called MPM—-which has been implemented following the design advocated in this paper.

More details about the paper are available here

UPDATE

Ralf (that went all the way to the US to present the paper) has just informed us that the paper won the ‘ACM Sigsoft Distinguished Paper Award’ !!!!!!!!

Dose3 in debian experimental !

Thanks to Ralf’s work, dose3 has been just accepted in debian experimental !!! yuppiiiii \o/

Dose3 is a framework made of several OCaml libraries for managing distribution packages and their dependencies.

Though not tied to any particular distribution, dose3 constitutes a pool of libraries which enable analyzing packages coming from various distributions.

Besides basic functionality for querying and setting package properties, dose3 also implements algorithms for solving more complex problems (monitoring package evolutions, correct and complete dependency resolution, repository-wide uninstallability checks).

For the ocaml affectionados the API of the library is available here (there is still a bit of work to do here…) . The source code shipped in each release does not include a vast range of applications sitting in the experimental directory. You can have a look at these in our svn.

if you want to get a copy of the development / bleeding edge / unstable version from svn, you can check it out from the svn

svn co https://gforge.info.ucl.ac.be/svn/mancoosi/trunk/dose3

user/pass : mancoosi/mancoosi

Eating my own dog food - mpm

After a bit of work, today I decided to start using mpm, the mancoosi package manager, to upgrade my laptop. My first use of it on a production system - until now I run all my experiments in throw-away virtual machines - and it works !

Not rocket science here. During the last month David Kalnischkies (of APT fame) visited our offices in Paris and together with zack worked out a communication protocol between apt-get and the mancoosi cudf solvers (EDSP). I guess somebody is going to announce all details about this endeavor soon. This cooperation enabled us to advance in the integration of apt-get and the mancoosi technology.

Reusing the same protocol, and backend I developed to translate the apt problem to cudf (and to call a suitable solver), I’ve re-wrote large part of mpm and added the possibility to generate the installation plan before calling dpkg and really installing the selected packages.

Below notice the intermediate calls as Inject Model , Simulate, Compare Models . These are at the moment stubs that are going to call the simulation framework developed at mancoosi.

The food :

abate@zed.fr:~/Projects/git-svn-repos/mpm$sudo ./mpm.py -c mpm.conf update
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Inject Model ...


abate@zed.fr:~/Projects/git-svn-repos/mpm$sudo ./mpm.py -c mpm.conf upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following new packages will be installed:
libtracker-client-0.8-0 libnet-ip-perl libio-socket-ssl-perl libasyncns0 libapr1 
5 to install
Simulate
Proceed ? yes/[no] 
yes
Reading package fields... Done
Reading package status... Done
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
apt-listchanges: Mailing root: apt-listchanges: changelogs for zed
Reading package fields... Done
Reading package status... Done
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
apt-listchanges: Mailing root: apt-listchanges: changelogs for zed
(Reading database ... 179534 files and directories currently installed.)
Preparing to replace libapr1 1.4.2-8 (using .../libapr1_1.4.4-1_amd64.deb) ...
Unpacking replacement libapr1 ...
Preparing to replace libasyncns0 0.8-1 (using .../libasyncns0_0.8-2_amd64.deb) ...
Unpacking replacement libasyncns0 ...
Preparing to replace libio-socket-ssl-perl 1.40-1 (using .../libio-socket-ssl-perl_1.43-1_all.deb) ...
Unpacking replacement libio-socket-ssl-perl ...
Preparing to replace libnet-ip-perl 1.25-2 (using .../libnet-ip-perl_1.25-3_all.deb) ...
Unpacking replacement libnet-ip-perl ...
Preparing to replace libtracker-client-0.8-0 0.8.17-2 (using .../libtracker-client-0.8-0_0.8.18-1_amd64.deb) ...
Unpacking replacement libtracker-client-0.8-0 ...
Processing triggers for man-db ...
Setting up libapr1 (1.4.4-1) ...
Setting up libasyncns0 (0.8-2) ...
Setting up libio-socket-ssl-perl (1.43-1) ...
Setting up libnet-ip-perl (1.25-3) ...
Setting up libtracker-client-0.8-0 (0.8.18-1) ...
localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB
localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB

Total disk space freed by localepurge: 0 KiB

localepurge: Disk space freed in /usr/share/locale: 0 KiB
localepurge: Disk space freed in /usr/share/man: 0 KiB
localepurge: Disk space freed in /usr/share/gnome/help: 0 KiB
localepurge: Disk space freed in /usr/share/omf: 0 KiB

Total disk space freed by localepurge: 0 KiB

Inject Model ...
Compare Models ...
abate@zed.fr:~/Projects/git-svn-repos/mpm$

Reg the installation plan, this is an xml file that will be passed to a simulator developed by the university of L’Aquila to make sure that the installation script (well, a model of them), will not cause any problem during the installation.

The format is a very simple xml file as follows :

<selectionStates>
 <selectionState type="Install"> 
  <param name="package" value="libapr1" />
  <param name="version" value="1.4.4-1" /> 
  <param name="architecture" value="amd64" />
 </selectionState>
 <selectionState type="Install">
  <param name="package" value="libasyncns0" />
  <param name="version" value="0.8-2" />
  <param name="architecture" value="amd64" />
 </selectionState>
 <selectionState type="Install">
  <param name="package" value="libio-socket-ssl-perl" />
  <param name="version" value="1.43-1" />
  <param name="architecture" value="all" />
 </selectionState>
[...]

Soon APT will ship a patch to use the very same infrastructure of mpm (yeii !!!). This will on one hand make mpm useless as package manager on its own. It is a simple hack in python and I never tough to compete with its big brothers. On the other hand I think it will stand as a nice workbench to experiment with new ideas, to prototype new features and to make it easier for poeple that are not c++ experts to play with the APT library (thanks to python-apt) in a semi structured environment. The code is in the mancoosi svn if you want to have a look.

On the apt-get installation plan

One important aspect of the mancoosi project is to build a model of the installation process in order to simulate packages upgrades before committing the changes on the machine. This work described here and here (and in upcoming publications) relays on the availability installation plan of the meta-installer (the exact order in which packages will be installed, configured and removed ) to run a simulation of maintainers scripts and check for potential problems accordingly to the model.

Thanks to the help of Julian Andres Klode upstream and maintainer of the python-apt bindings it is now possible to extract the installation plan from apt-get without resorting to debug printout or other hacks.

A forthcoming integration with mpm (Mancoosi-Package-Manager, our testbed meta-installer) will give us the possibility to hook the model checker to the meta-installer and to run it before every upgrade.

The two important changes in the python-apt bindings that will make this possible are the new bindings to the class OrderList and the sub-classing of the class package manager. PackageManager .

Regarding the latter, this is a small example (thanks again Julien !) that you can use to get the installation plan as computed by apt-get. In theory, this will also allow to experiment with different back-ends (like rpm - blah ! :) ) easily without the need to hack the c++ code.

import apt_pkg, sys
import apt

class PkgManager(apt_pkg.PackageManager):

        parent = apt_pkg.PackageManager
        depcache = apt_pkg.DepCache(apt_pkg.Cache())
        installionplan = []

        def install(self, pkg, file):
                #print "Installing", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Inst"))
                return True

        def configure(self, pkg):
                #print "Configuring", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Conf"))
                return True

        def remove(self, pkg, purge):
                #print "Removing", pkg.get_fullname(True)
                self.installionplan.append((pkg,"Rem"))
                return True

        def go(self, fd):
              for (p,a) in self.installionplan :
                  if a == "Inst" or a == "Conf" :
                      ver = self.depcache.get_candidate_ver(p)
                  else :
                      ver = p.current_ver
                  print a, p.name, ver.ver_str, ver.arch

              return True

apt_pkg.PackageManager = PkgManager


cache = apt.Cache()
pkg = cache["python"]
if pkg.is_installed:
        pkg.mark_delete()
else:
        pkg.mark_install()

apt_pkg.config.set("APT::Get::Simulate","true")
apt_pkg.config.set("dir::cache","/tmp")

print "COMMIT"  
cache.commit(install_progress=apt.progress.base.InstallProgress())

And this is the result on my machine :

abate@zed.fr:~$python pkgmanager-test.py 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
COMMIT
Rem alacarte 0.13.2-1 all
Rem apt-listchanges 2.85.7 all
Rem apt-xapian-index 0.41 all
Rem gnome-accessibility 1:2.30+7 all
Rem gok 2.30.0-1 amd64
Rem dasher 4.11-1 amd64
Rem gnome-orca 2.30.2-2 all
Rem python-pyatspi 1.30.1-3 all
Rem at-spi 1.30.1-3 amd64
Rem bluetooth 4.91-1 all
Rem gnome-bluetooth 2.30.0-2 amd64
Rem bluez 4.91-1 amd64
Rem brasero 2.30.3-2 amd64
Rem gthumb 3:2.13.1-1 amd64
Rem libbrasero-media0 2.30.3-2 amd64
Rem brasero-common 2.30.3-2 all
Rem bzr 2.3.1-2 all
Rem gnome-core 1:2.30+7 amd64
Rem gnome-control-center 1:2.30.1-3 amd64
Rem capplets-data 1:2.30.1-3 all
Rem cheese 2.30.1-2 amd64
Rem cheese-common 2.30.1-2 all
Rem cython 0.14.1-5 amd64
Rem dasher-data 4.11-1 all
Rem deluge 1.3.1-1 all
Rem deluge-gtk 1.3.1-1 all
Rem deluge-common 1.3.1-1 all
Rem deskbar-applet 2.32.0-1+b1 amd64
Rem dia 0.97.1-8 amd64
Rem dia-common 0.97.1-8 all
Rem doxygen 1.7.4-1 amd64
Rem doxygen-latex 1.7.4-1 all
Rem dput 0.9.6.2 all
Rem duply 1.5.4.2-1 all
Rem duplicity 0.6.13-1 amd64
Rem edos-debcheck 1.0-9 all
Rem edos-distcheck 1.4.2-12 amd64
Rem eog 2.30.2-1 amd64
Inst libxp6 1:1.0.1-1 amd64
Inst lesstif2 1:0.95.2-1 amd64
Inst xpdf 3.02-12 amd64
Conf libxp6 1:1.0.1-1 amd64
Conf lesstif2 1:0.95.2-1 amd64
Conf xpdf 3.02-12 amd64
[ ... ]

Simple Pcre based url parsing

Date Tags ocaml

A simple module to parse a uri (well, a small subset of the RFC 3986). The Ocamlnet library contains a better implementation, but it also comes with a lot of dependencies…

Grab it, use it, change it, give it away but remember me :) …

(* ref : http://labs.apache.org/webarch/uri/rfc/rfc3986.html#collected-abnf *)
(* ref : http://stackoverflow.com/questions/161738/what-is-the-best-regular-expression-to-check-if-a-string-is-a-valid-url *)

type authority = {
  userinfo : string option;
  host : string;
  port : int option
}

type url = {
  url : string;
  scheme : string;
  authority : authority option;
  path   : string; 
  query  : (string * string) list; 
};;

let try_with f =
  try Some(Lazy.force f) with Not_found -> None
;;

let parse_authority s =
  let re_s = 
    "(?:(?P<userinfo>(?:[A-Za-z0-9\\-._~!$&\\'()*+,;=:]|%[0-9A-Fa-f]{2})*)@)?" ^
    "(?P<host>(?:[A-Za-z0-9\\-._~!$&\\'()*+,;=]|%[0-9A-Fa-f]{2})+)" ^
    "(?::(?P<port>[0-9]*))?"
  in
  let rex = Pcre.regexp re_s in
  let res = Pcre.exec ~rex s in
  {
    userinfo = try_with (lazy(Pcre.get_substring res 1));
    host = Pcre.get_substring res 2;
    port = try_with (lazy(int_of_string (Pcre.get_substring res 3)))
  }
;;

let parse_query s = 
  let re_s = 
    "(?:\\?(?P<query>(?:[A-Za-z0-9\\-._~!$&\\'()*+,;=:@\\\\/?]|%[0-9A-Fa-f]{2})*))?"
  in
  let rex = Pcre.regexp re_s in
  let res = Pcre.exec ~rex s in
  List.map (fun s ->
    let a = Pcre.asplit ~rex:(Pcre.regexp "=") s in
    (a.(0),a.(1))
  )
  (Pcre.split ~rex:(Pcre.regexp ";") (Pcre.get_substring res 1))
;;

(* we parse uri of type schema://user:pass@machine:port/path/to/file?a=1;b=2 *)
let parse_remote_uri s =
  let re_s = "^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?" in
  let rex = Pcre.regexp re_s in
  let res = Pcre.exec ~rex s in
  try
    {
      url = Pcre.get_substring res 1;
      scheme = Pcre.get_substring res 2;
      authority = try_with (lazy(parse_authority (Pcre.get_substring res 4)));
      path = Pcre.get_substring res 5;
      query = try parse_query (Pcre.get_substring res 6) with Not_found -> []
    }
  with Not_found -> failwith (Printf.sprintf "Malformed Url : %s" s)
;;

(* we parse uri of the type schema:/path/to/file *)
let parse_local_uri s =
  let re_s = "^(([^:/?#]+):)?(([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?" in
  let rex = Pcre.regexp re_s in
  let res = Pcre.exec ~rex s in
  try
    {
      url = Pcre.get_substring res 1;
      scheme = Pcre.get_substring res 2;
      authority = None;
      path = (Pcre.get_substring res 4)^(Pcre.get_substring res 5);
      query = []
    }
  with Not_found -> failwith (Printf.sprintf "Malformed Url : %s" s)
;;

let parse_uri ?(local=true) s =
  if local then parse_local_uri s
  else parse_remote_uri s
;;

# parse_url "sqlite://user:secret@machine:42/a:c/b/c?a=1;b=2";;
- : url =
{url = "sqlite:"; scheme = "sqlite";
 authority =
  Some {userinfo = Some "user:secret"; host = "machine"; port = Some 42};
 path = "/a:c/b/c"; query = [("a", "1"); ("b", "2")]}

# parse_url "file:/aa/vv/bb";;
- : url =
{url = "file:"; scheme = "file"; authority = None; path = "/aa/vv/bb"; query = []}