edos-distcheck - new YAML output format
During the last two days I spent some time to implement part of the [http://wiki.debian.org/EDOS/ProposalDose3 proposed features for distcheck/ edos-distcheck]. Since everybody is at debconf and talk is silver, but code is gold, I hope that a real implementation can get the ball rolling and get us closer to a stable release of the next generation of edos/mancoosi tools.
In particular this post is about the new YAML output format for distcheck. The rational to use YAML is to have a data structure that is at the same time human and machine friendly. There are a lot of scripts in debian that rely on distcheck and we want to provide a grep friendly output that sat the same time doesn’t hurt your eyes. The other proposed solution was to use json, but was ditched in favor of YAML. We also removed the xml output.
In order to provide a machine readable output and to minimize parsing mistakes, I used the schema language proposed here. This is the resulting data structure definition :
type: seq
sequence:
- type: map
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"status": { type: str, enum: [ broken, ok ], required: true }
"installationset":
type: seq
sequence:
- type: map
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"reasons":
type: seq
sequence:
- type: map
mapping:
"conflict":
type: map
mapping:
"pkg1":
type: map
required: true
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"pkg2":
type: map
required: true
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"paths":
type: seq
sequence:
- type: map
mapping:
"path":
type: seq
sequence:
- type: map
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"vpkg": { type: str, required: false}
"missing":
type: map
mapping:
"pkg":
type: map
required: true
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"vpkg": { type: str, required: false}
"paths":
type: seq
sequence:
- type: map
mapping:
"path":
type: seq
sequence:
- type: map
mapping:
"package": { type: str, required: true }
"version": { type: text, required: true }
"vpkg": { type: str, required: false}
There are few improvements on the old output from edos-distcheck. We are going to discuss these with a real example. The following tow snippets are from the output of distcheck on sid/amd64 (04/08/2010).
Distcheck now outputs a list of broken or installable packages depending on the given options (—failoures , —success, —explain and combinations of thereof ) . Two quick examples :
-
package: python-gi-dbg
version: 0.6.0-1
status: broken
reasons:
-
conflict:
pkg1:
package: python-gobject
version: 2.21.4-1
pkg2:
package: python-gi
version: 0.6.0-1
paths:
-
path:
-
package: python-gi-dbg
version: 0.6.0-1
vpkg: python-gi (= 0.6.0-1)
-
package: python-gi
version: 0.6.0-1
-
path:
-
package: python-gi-dbg
version: 0.6.0-1
vpkg: python-gi (= 0.6.0-1)
-
package: python-gi
version: 0.6.0-1
vpkg: python-gobject (>= 2.20)
-
package: python-gobject
version: 2.21.4-1
In the example above, the package python-gi-dbg is broken
because there is a conflict between the packages python-gobject
and python-gi. The reason why python-gi-dbg is affected by
this conflict is explained by following the dependency chain from
python-gi-dbg to the two offending packages. Note that for each
package element of each path we specify the vpkg, that is the
dependency (as it is written in the control file) that lead to the
conflict. Since a dependency can be a virtual package or a package
with a version constraint, it can be expanded to a disjunction of
packages (think a dependency on mta-agent can be expanded as
postfix. exim or sendmail…). All possible paths to an offending
package are reported.
Likewise if a package is broken because there is an unfulfilled
dependency, distcheck will show the path leading to the problem . In
the following example we show that the package gnash-tools is
broken because there are two dependency that depend on the missing
package libboost-date-time1.40.0 (>= 1.40.0-1).
-
package: gnash-tools
version: 0.8.7-2+b1
status: broken
reasons:
-
missing:
pkg:
package: gnash-common
version: 0.8.7-2+b1
vpkg: libboost-date-time1.40.0 (>= 1.40.0-1)
paths:
-
path:
-
package: gnash-tools
version: 0.8.7-2+b1
vpkg: gnash-common-opengl (= 0.8.7-2+b1) | gnash-common (= 0.8.7-2+b1)
-
package: gnash-common
version: 0.8.7-2+b1
vpkg: libboost-date-time1.40.0 (>= 1.40.0-1)
-
missing:
pkg:
package: gnash-common-opengl
version: 0.8.7-2+b1
vpkg: libboost-date-time1.40.0 (>= 1.40.0-1)
paths:
-
path:
-
package: gnash-tools
version: 0.8.7-2+b1
vpkg: gnash-common-opengl (= 0.8.7-2+b1) | gnash-common (= 0.8.7-2+b1)
-
package: gnash-common-opengl
version: 0.8.7-2+b1
vpkg: libboost-date-time1.40.0 (>= 1.40.0-1)
The code is still in a flux and it is not ready for production yet (everything is in the mancoosi svn). I hope this is a good step in the right direction. Comments on the debian wiki are welcome.
if we compare the output of distcheck with the old edos-debcheck we get the following:
$cat /var/lib/apt/lists/ftp.debian.org_debian_dists_unstable_main_binary-amd64_Packages | edos-debcheck -failures -explain
[...]
holotz-castle-milanb (= 0.0.20050210-1): FAILED
holotz-castle-milanb (= 0.0.20050210-1) depends on one of:
- holotz-castle (= 1.3.14-2)
holotz-castle-data (= 1.3.14-2) and holotz-castle-milanb (= 0.0.20050210-1) conflict
holotz-castle (= 1.3.14-2) depends on one of:
- holotz-castle-data (= 1.3.14-2)
$./debcheck --explain --failures /var/lib/apt/lists/ftp.debian.org_debian_dists_unstable_main_binary-amd64_Packages
[...]
-
package: holotz-castle-milanb
version: 0.0.20050210-1
status: broken
reasons:
-
conflict:
pkg1:
package: holotz-castle-milanb
version: 0.0.20050210-1
pkg2:
package: holotz-castle-data
version: 1.3.14-2
paths:
-
path:
-
package: holotz-castle-milanb
version: 0.0.20050210-1
vpkg: holotz-castle
-
package: holotz-castle
version: 1.3.14-2
vpkg: holotz-castle-data (= 1.3.14-2)
-
package: holotz-castle-data
version: 1.3.14-2
