Learning from the Future of Component Repositories - CBSE 2012

Learning from the Future of Component Repositories ( Pietro Abate, Roberto Di Cosmo, Ralf Treinen and Stefano Zacchiroli ) has been accepted to be presented at CBSE 2012 (26-28 June, Bertinoro, Italy)

Abstract

  An important aspect of the quality assurance of large component repositories
  is the logical coherence of component metadata. We argue that it is possible
  to identify certain classes of such problems by checking relevant properties
  of the possible future repositories into which the current
  repository may evolve. In order to make a complete analysis of all possible
  futures effective however, one needs a way to construct a finite set of
  representatives of this infinite set of potential futures. We define a class
  of properties for which this can be done.

  We illustrate the practical usefulness of the approach with two quality
  assurance applications: (i) establishing the amount of ``forced upgrades''
  induced by introducing new versions of existing components in a repository,
  and (ii) identifying outdated components that need to be upgraded in order to
  ever be installable in the future. For both applications we provide
  experience reports obtained on the Debian distribution.
The tools presented in this paper (outdated and challenges) are already in Debian as part of the 'dose-extra' package.

Update

For the second year in a raw our paper won the '''Best Paper Award''' for the CBSE 2012 conference !!!

Update 2

I presented this paper at cbse2012. The slides of my presentations are attached.
AttachmentSize
main.pdf375.94 KB
Average: 5 (1 vote)

terminator terminal

Despite the fact I can't stop talking about xfce4, what I'm have been missing lately is the functionality of gnome-terminal... I know I could install gnome terminal and use it, but this will imply a lot of unwanted dependencies and I prefer not go that way. Chatting on #xfce in IRC it was suggested to give a try to terminator https://launchpad.net/terminator/. I've used it for a couple of days now and I feel very comfortable with it. It has a lot of configuration options, the possibility to have multiple profiles and to group windows together, it has tabs and all bells and whistles you can ask.

The only thing I didn't like from the default settings is the red title bar on top... to remove it you need to edit the config file in ~/.config/terminator/config and add show_titlebar = False . This is my conf file :

[global_config]
  geometry_hinting = False
  dbus = True
  focus = mouse
  borderless = True
[keybindings]
[profiles]
  [[default]]
    scrollbar_position = hidden
    show_titlebar = False
  [[presentation]]
    use_system_font = False
    font = Monospace 13
[layouts]
  [[default]]
    [[[child1]]]
      profile = default
      type = Terminal
      parent = window0
    [[[window0]]]
      type = Window
      parent = ""
[plugins]

Something else I'd like to do is to reduce the size and font of the tab on top of the terminal. Since this is a gtk applications, I should hack my gtk theme in order to style the tab, but I don't know how to do it only for one application. If somebody has an answer I'm all ears.

Average: 2.7 (7 votes)

mancoosi tools in debian testing

Finally the dose3 libraries and tools landed in testing this weekend. We solved a couple of bugs already and it seems nobody complained too loudly. If you used the edos tools in the past you might be interested to check out our new tools in the package dose-extra.

Actually @mancoosi we will be delighted to ear about you experience with our tools and how to make them better and more useful. Please drop me a line !

The next major release of dose will be multi-arch aware and provide performance improvements and other minor features.

If you missed it, and you are now curious, I delivered a talk at fosdem regarding our tools:

  • QA tools for FOSS distributions (FOSDEM 2012) - Pietro Abate (video)

A big thanks to ralf of course for packaging everything !

No votes yet

Xfce terminal and middle mouse button.

just a quicky. If you use the xfce terminal and you like to have more then one tab-terminal open, you might have niticed the the middle button closes it, without confirmation... I just discovered that this annoying feature can be turned off.

Following the freedesktop spec, you can just edit the file .config/Terminal/terminalrc and set the option MiscTabCloseMiddleClick to False.

There are many other settings . The full list is here : http://docs.xfce.org/apps/terminal/advanced

Average: 5 (1 vote)

Mancoosi fosdem talks

A small (and probably not complete) list of videos associated to the mancoosi project @ FOSDEM.

  • QA tools for FOSS distributions (FOSDEM 2012) - Pietro Abate (video)
  • Mancoosi tools for the analysis and quality assurance of FOSS (FOSDEM 2011) - Ralf Treinen (video)
  • Improving Rollback in Linux via DSL approach & distributing (FOSDEM 2011) - John Thomson (video)
  • Cross distro dependency resolution reusing solvers among distros (FOSDEM 2010) - Stefano Zacchiroli (video)
  • The MANCOOSI project (FOSDEM 2008) - Ralf Treinen (video)
No votes yet

Finding all the elementary circuits of a directed graph

Below you can find an implementation in ocaml of the algorithm described in this paper by D. B. Johnson

  Finding all the elementary circuits of a directed graph.
  D. B. Johnson, SIAM Journal on Computing 4, no. 1, 77-84, 1975.
  http://dx.doi.org/10.1137/0204007
the idea of this algorithm is to enumerate all cycles in a strongly connected component. Next step will be to implement the "feedback arc set" of this connected component so to find the optimal way to break these loops in a way that the connected component can be partitioned in smaller DAGs.

This is only the main function, you can find everything else in the git repo given below. If I've time I'll also give a shot to implement tarjan algorithm and to compare the two... Johnson is supposed to be better complexity wise, but I'm curious to check the difference on my domain specific input.

let find_all_cycles_johnson g =
  if not G.is_directed then
    assert false;

  (*  stack of nodes in current path *)
  let path = Stack.create () in

  (* main function. return (true, acc) if in the last iteration we find a new cycle *)
  let rec circuit t result thisnode startnode component =

   (* add the node the candate cycles and block it *)
    Stack.push thisnode path;
    block t thisnode;

    let (closed,result) =
      G.fold_succ (fun nextnode (c,r) ->
        if G.V.equal nextnode startnode then
         (* this is a cycle ! *)
          (true,(Stack.copy path)::r)
        else begin
          if not(is_bloked t nextnode) then
            circuit t r nextnode startnode component
          else
            (c,r)
        end
      ) component thisnode (false,result)
    in
    if closed then
      (* this node is part of a cycles, but we don't want to exclude it from subsequent
          iterations. Longer cycles can contain smaller cycles *)

      unblock t thisnode
    else
      (* since it is not part of a cycles, then it must be part of a portion of the graph that
          does not contain any elementary circuits *)

      G.iter_succ (fun nextnode ->
        let l = get_notelem t nextnode in
        if List.mem thisnode l then
          Hashtbl.replace t.notelem nextnode (thisnode::l)
      ) component thisnode;

    ignore(Stack.pop path);
    (closed,result)
  in

  (* Johnson's algorithm requires some ordering of the nodes.
      We use the order induced by the compare function associated to the
      vertex type. See ocamlgraph Pack.Diagraph  *)

  let vertex_set = G.fold_vertex SV.add g SV.empty in
 
  SV.fold (fun s result ->
    (* Build the subgraph induced by s and following nodes in the ordering *)
    let subset = SV.add s (partition vertex_set s) in
    let subgraph = extract_subgraph g subset in

    if G.nb_edges subgraph > 0 then begin
      (* Find the strongly connected component in the subgraph
       * that contains the least node according to the ordering *)

      let scc = G.Components.scc_list subgraph in
      let minnode = SV.min_elt subset in
      let mincomp = List.find (fun l -> List.mem minnode l) scc in
     
      (* smallest node in the component according to the ordering *)
      let startnode = minnode in
      let component = extract_subgraph subgraph (to_set mincomp) in

      (* init the block table for this component *)
      let t = init_block component in

      snd(circuit t result startnode startnode component);
    end else
      result
  ) vertex_set []
;;

You can get the code here : git clone http://mancoosi.org/~abate/repos/cycles.git . There are two versions of the algorithm. One is a translation in ocaml of the imperative algorithm, the other, presented above, adds a bit of functional flavor ...

Update 1

the code repository now contains also two implementations of approximate algorithms the feedback arc set problem

Update 2

Since the number of cycles in a graph can be very large, I've added a lazy version of the algorithm using ExtLib enums (but can be easily done without). It would be cool to avoid using a global data structure to keep track of which nodes I've already visited like the persistent array of filliatr ... the code could be further optimized of course as many operations on lists and set can be avoided with a smarter DS to build the connected component of the subgraph at each iteration ... Complete code in the git repo above.

let find_all_cycles_enum g =
  let rec circuit path t thisnode startnode component =                                        
     let rec aux = function                                                                    
       |[] -> Enum.empty ()                                                                    
       |nextnode :: rest ->
         if G.V.equal nextnode startnode then begin                                            
           let e = aux rest in                                                                
           unblock t thisnode;
           Enum.push e (List.rev (thisnode::path));                                            
           e
         end else
           if not(is_bloked t nextnode) then begin
             let e = circuit (thisnode::path) t nextnode startnode component in                
             Enum.append e (aux rest)                                                          
           end else begin                                                                      
             aux rest                                                                          
           end                                                                                
     in
     block t thisnode;
     let e = aux (G.succ component thisnode) in                                                
     G.iter_succ (fun nextnode ->
       let l = get_notelem t nextnode in                                                      
       if List.mem thisnode l then
         Hashtbl.replace t.notelem nextnode (thisnode::l)                                      
     ) component thisnode;                                                                    
     e                                                                                        
  in                                                                                          
 
  let vertex_set = G.fold_vertex SV.add g SV.empty in                                          
  let rec aux = function                                                                      
    |[] -> Enum.empty ()                                                                      
    |s :: rest ->  
      let subset = SV.add s (partition vertex_set s) in                                        
      let subgraph = extract_subgraph g subset in                                              
      (* I need only one... not all scc *)
      (* actually I need only the scc that contains the min *)                                
      let scc = G.Components.scc_list subgraph in                                              
      let minnode = SV.min_elt subset in
      let mincomp = List.find (fun l -> List.mem minnode l) scc in                            
      let startnode = minnode in
      let component = extract_subgraph subgraph (to_set mincomp) in                            
      let t = init_block component in
      let partial = circuit [] t startnode startnode component in                              
      Enum.append partial (aux rest)                                                          
  in
  aux (SV.elements vertex_set)                                                                
;;

Average: 2.3 (4 votes)

Nice menus in drupal 7 ?

I wanted to add a nice dropdown menu using drupal 7 and browsing the net I realized that there are so many drupal consultants making soooo many (fantastic) videos that it's kind of difficult to understand which version of drupal they are using and if things changed in the meanwhile.

An I know this blog post is going to be obsolete in a short while...

Anyway, if you are using drupal 7.12 and nice menus 7.x-2.0 creating a dropdown menu does not require any templateing foo (except if you want to change the default nice menu appearance). Just create a menu (I used the main menu) with a hierarchical structure. Then in the them setting (I'm using bartik) disable the main and secondary menus. At this point you need to go in the block configuration panel and add in the "features" region the nice menu block. save, rinse and it's done !

It's actually very easy, but the doco and videos around gave me a different impression. I think mostly because drupal evolves so fast and developers listen a lot to users that simple this are actually simple (and hard things are possible !)

My 002 drupal cents for the day.

Average: 2.3 (10 votes)

Dependency order matters !

Recently I've discovered a subtle consequence of how the order in which dependencies are specified in debian actually matters. While re-factoring the code of dose3, I changed the order in which dependencies are considered by our sat solver (of edos-fame) . I witnessed a twofold performance loss just by randomizing how variables were presented to our sat solver. This highlights, on one hand, how our solver is strongly dependent on the structure of the problem and, on the other hand the standard practice of debian maintainers to assign an implicit priority in the disjunctive dependencies where the first is the most preferred packages (and maybe the most tested, at least dependency-wise).

The basic idea of distcheck is to encode the dependencies information contained in a Packages file in CNF format and then to feed them to a sat solver to find out if a package has broken dependencies or if its dependencies are such that no matter what, it would be impossible to install this package on a user machine.

Conflicts are encoded as binary clauses. So if package A conflicts with package B, I add a constraint they says "not (A and B)" , that is A and B cannot be considered together. The dependencies encoding associates to each disjunction of the depends field a clause that says "A implies B". If a package foo depends on A,B | C,D , I'll add "foo implies A and B" & "foo implies C and D" . This encoding is pretty standard and it is easy to understand.

The problem is how the sat solver will search for a solution to the problem "Is is possible to install package foo in an empty repository". The solver we use is very efficient and can easily deal with 100K packages or more. But in general is not very good at dealing with random CNF instances. The reason because edos-debcheck is so efficient lies in the way it exploits the structure of the sat problems.

The goal of a sat solver is to find a model (that is a variable assignment list) that is compatible with the given set of constraints. So if my encoding of the debian repository is a set of constraints R, the installability problem boils down to add an additional constraint to R imposing that the variable associated to the package foo must be true, and then ask the solver to find a model to make this possible. This installation, in sat terms, would be just an array of variables that must be true in order to satisfy the given set of constraints.

If you look at the logic problem as a truth table, the idea is to find a row in this table. This is the solution of your problem. Brute force of course is not an option and modern sat solvers use a number of strategies and heuristic to guide the search in the most intelligent way possible. Some of them try to learn from previous attempts, some of them, when they are lost try to pick a random variable to proceed.

If we consider problems that have a lot of structure, award winning sat solver do not back track very much. By exploiting the structure of the problem, their algorithm allows them to considerably narrow down the search only to those variables that are really important to find a solution.

All this long introduction was to talk about the solver that is currently used in edos-debcheck and distcheck (that is a rewrite of the edos-debcheck).

So why dependency order matters ? If we consider any package, even if the policy does not specify any order in the dependencies, it's common practice to write disjunctive dependencies specifying the most probable and tested alternative first and all other, more esoteric choices later. Moreover real packages are considered *before* virtual packages. Since every developer seems be doing the same, some kind of structure might be hidden in the order in which dependencies are specified.

Part of the efficiency of the the solver used in our tools is actually due to the fact that its search strategy is strongly dependent on the order in which literal are specified in each clause. Saying the package foo depends on A and B is "different" then saying it depends on B and A, even if semantically equivalent.

In my tests, I found about a twofold performance loss if the order of literals is either randomized or inverted. This is clearly a specific problem related to our solvers, while other solvers might not be susceptible to such small structural changes. Sat competitions often employs some form of obfuscation strategies of well known problems with well known structures in order to make useless to encode a search strategy that exploits the specific structure of a problem.

Since here we're not trying to win a sat competition, but to provide tool to solve a specific problem, we are of course very happy to exploit this structure.

Average: 1 (4 votes)

QA tools for FOSS distributions

I'm going to deliver this talk at fosdem 2012, room H.1301 (CrossDistribution Devroom) at 16:30 on Sat. If you are interested, please come by. In particular I'd like to talk with all the developers out there that are using our work (of edos fame) and to discuss with them future plans to migrate their programs to the new generation of mancoosi - powered QA tools. Scroll down to get the slides .

fosdem link : http://fosdem.org/2012/schedule/event/distro_qa_tools

Abstract

FOSS distributions are increasingly over pressure to deliver stable releases including the most up to date upstream software. Time-based release strategies has exacerbated this problem putting even more pressure on QA teams. The recently concluded Mancoosi project has developed a number of tools to automatically analyse large packages collections for a range of problems, from installability checks to speculative analysis of software repositories. In this talk I'll present four command line tools to identify and correct potential problems as soon as possible during the release cycle.

In particular :

  • Debcheck: This tools helps to identify all broken packages within a repository and provides a detailed explanation of the problem. This can be used to prevent shipping releases that contain packages that cannot be installed because of missing or malformed dependencies.
  • Buildcheck: Given a Sources file and a set of binary repositories, this tool identifies those source packages that cannot be compiled because their build dependencies cannot be satisfied.
  • Outdated: This tool identifies those broken packages that need special attention because of outdated meta-data.
  • Challenged: This tool performs a speculative analysis of the repository to identify those packages that, if upgraded to a specific version, would break a large number of other packages in the repository. This tool would be particularly useful during the upgrade of a specific component to evaluate its impact on the software archive.

Most of our tools support both rpm (version 4 and 5) and deb based distributions.

The mancoosi team.

AttachmentSize
main.pdf504.46 KB
No votes yet

xfce4 and awsome

If you are tracking unstable, and you were using gnome2, then it's futile to resist and not to move to gnome3. A lot has been written about gnome3, some poeple love it, others hate it. Others put their head under the sand by using the gnome3 fall-back mode. I'm on the "hate" category. I've used the fall-back mode for a while, then switched to the full blown gnome-shell and I've tried to use it for one month. I have to admit that is nice looking, intuitive and accessible. However it lacks so many things (gnome shell plug-ins are nice, but we still have to wait quite a while to have back all the fantastic gnome2 plug-ins) that I had in gnome2 that the new shiny look is not enough to keep me using it.

Moreover, apart for the very subjective reasons I gave above (I'm sure then other had different experiences and have a different pain threshold then I do), what I missed the most is the integration with awesome. I started using a tiling window manager last year and my productivity sky rocketed. Going back to manual window placement, overlapping, hiding, and this desperate continuous use of the "expose" functionality of gnome-shell was driving me mad.

So, since I started fresh with the new laptop, I looked around for alternatives. Going KDE is not an option. Many people say it's nice and it works very well, but it's not my cup of tea. Going back to gnome 2 was not really an option either. What I knew is that I wanted a desktop environment that is compatible with the freedesktop standards, modular and that would allow me to use my WM of choice.

The almost natural solution was to try xfce4. It seems to me a very nice desktop environment, light weight, extensible and with all the goodies I was looking for. The feeling is very much of gnome2. All components can work independently and it works very well with awesome.

Since i wanted a minimal subset of components I started by installing the xfce4 panel and awsome. This worked ok, but there were a lot of functionality missing, like plug-ins, notifications, automunting, integrations with consolekit, etc...

So after fighting a while, I've installed the full xcfe4 stack. Running awesome instead of the standard wm is just a matter of creating a custom session in the user preferences. On the awesome side, you need to disable the awesome panel and the awesome menu. This is all pretty easy and it was pretty much the same conf I used to have with gnome2.

I also tried to use slim as display manager. I've to say it works well, but fails to integrate with xfce4 and consolekit leaving me without the correct permissions. Looking for a replacement, I've tried ligthdm. This one more used then slim and intergrates perfectly with consolekit solving all my problems.

On very nice application that comes with xfce4 is thunar, their file manager and it integration with Ristretto, the image viewer. It always stuck me how eog and nautilus work badly together...

And since I was at it, I also dumped rhythmbox for listen and f-spot for shotwell . I like these two applications. They do their job well, they are stable (so far) and have all the functionalities I need. Bonus I finally go rid of mono !

Average: 1 (1 vote)
Syndicate content