puppet in a DMZ (firewall piecing with autossh)

6 March 2013 - 11:59am — abate

Recently I had to deploy a couple of new virtual servers and since we are in the 21st century, I decided to configure puppet from the beginning. For the first two machines, on the internal network, the job is easy. Puppet is well packaged in debian and the default configuration was like a charm.

The problem (and solution) I'm describing here is about tunneling puppet thought ssh jump from the DMZ to the internal network and to allow the client in the DMZ to access the puppet master in the internal network. I'm aware that this solution is hackish , but since sometimes is easier not asking (and piercing a firewall) then to open a ticket and go thought the notorious French bureaucracy. Puppet works with ssl certificates, all traffic is encrypted, so in my opinion is much better to add an exception to the firewall rules that to tunnel it. But anyway, here we go.

After duck-ducking a bit, I found on the web a nice suggestion to use autossh to establish a tunnel (master <-> client) and to use it to access my puppet master.

So on the pupper master I simply run :

/usr/bin/autossh -M 20000 -f -R localhost:8140:localhost:8140 -N -n -x client

This will make sure to open a ssh connection and monitor it. Autossh is a very nice tool that I didn't know.

Once this is done, the client can connect to the puppet master via localhost:8140. The next step on the book is to sign his key and allow him to connect and retrieve its catalog. In other to do this, I need to specify the new server name in the puppet.conf file. Simply specifying localhost will immediately give you trouble ...

Once the connection is established you can proceed with the authentication :

client side :

 puppet agent --debug --no-daemonize --verbose  --waitforcert 5

server side :

puppet cert sign client.internal.org

ref : http://serverfault.com/questions/47079/puppet-security-and-network-topologies

Average: 1.4 (20 votes)

ocaml hash function and complex data structures

6 March 2013 - 11:57am — abate

While writing the graphml printer for ocamlgraph (that btw has just been committed in the ocamlgraph svn and it going to be part of the next release of ocamlgraph...), I stumbled in the following pitfall (this is really ocaml for dummies :D). Imagine you want to associate a unique id to each data type. But you also want that equal values have equal ids. The first (wrong !) idea that might come to mind is to use the Hashtbl.hash function compute this integer uid. Even if this approach might work for simple data structure, it is doomed to fail for more complex data structure whose depth is greater then 10. The natural explanation for this lies in the documentation for the function Hashtbl.hash .

For example consider a simple list. If the list has less then 10 elements, then the hash function will return a unique id and equal lists will be associated to the same id. But if the list contains more then 10 elements, you have the following behavior :

# let l2 = [1;2;3;4;5;6;7;8;9;10;11;12;13] ;;
val l2 : int list = [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13]
# let l1 = [1;2;3;4;5;6;7;8;9;10;11;12] ;;
val l1 : int list = [1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12]
# Hashtbl.hash l1;;
- : int = 0
# Hashtbl.hash l2;;
- : int = 0

This is fully in line with the Hashtbl.hash function specification. Using the function Hashtbl.hash_param we can easily solve see that the problem here is related to the number of elements that are used to generate the hash code. If we consider more meaningful values, we get different results ...

# Hashtbl.hash_param 12 100 l1;;
- : int = 0
# Hashtbl.hash_param 13 100 l1;;
- : int = 12
# Hashtbl.hash_param 14 100 l1;;
- : int = 787199

...

# Hashtbl.hash_param 12 100 l2;;
- : int = 0
# Hashtbl.hash_param 13 100 l2;;
- : int = 0
# Hashtbl.hash_param 14 100 l2;;
- : int = 13
# Hashtbl.hash_param 15 100 l2;;
- : int = 852799
# Hashtbl.hash_param 16 100 l2;;
- : int = 108186764
# Hashtbl.hash_param 17 100 l2;;
- : int = 583816830

Now this is of course an example and using hash_param is not an option. If we really want to hash a list we need something more reliable then to guess a value for hash_param.

What we can do is to encapsulate our values in a new data type "Unique" that will make sure that each value is associated to a really unique id. The practice is easy :

module type UniqueType = sig
  type t
  type v
  val create : v -> t
  val value : t -> v
  val uid : t -> int
end
 
module type OrderedType = sig type t val compare : t -> t -> int end
 
module UniqueMake ( S : OrderedType ) : UniqueType with type v = S.t = struct
 
  module VMap = Map.Make(S)
 
  type v = S.t
  type t = Hash of (v * int)
 
  let gentag =
    let r = ref 0 in
    fun () -> incr r; !r
 
  let cons = ref VMap.empty
 
  let create v =
    try Hash(v,VMap.find v !cons) with
    Not_found -> begin
      let i = gentag () in
      cons := VMap.add v i !cons;
      Hash (v,i)
    end
 
  let value = function Hash (v,_) -> v
  let uid = function Hash (_,i) -> i
 
end

In this module I use a variant data type and a Map to keep track the uids. And there is really nothing deep about it. Another idea would be to use a phantom type instead, in this version I've to query the map each time I want to associate an uid with a value.

module type UniqueType = sig
  type unique
  type v
  type 'a t
  val create : v -> unique t
  val value : unique t -> v
  val uid : unique t -> int
end
 
module type OrderedType = sig type t val compare : t -> t -> int end
 
module Unique ( S : OrderedType ) : UniqueType with type v = S.t = struct
 
  module VMap = Map.Make(S)
  type unique
 
  type v = S.t
  type 'unique t = S.t
 
  let gentag =
    let r = ref 0 in
    fun () -> incr r; !r
 
  let cons = ref VMap.empty
 
  let create v =
    if not(VMap.mem v !cons) then
      cons := VMap.add v (gentag ()) !cons;
    v
 
  let value v = v
  let uid v = VMap.find v !cons
 
end

This is a very simple way of getting a unique ID for any custom data structure. Another way would be wrap my data structures in an object and then use Oo.id . I haven't tried this approach yet.

Average: 4.3 (3 votes)

latexdiff + git = rsc-latexdiff

25 February 2013 - 11:16am — abate

The problem of using git to write latex documents is that there is not built-in way to clearly see the differences from two different commits (SO discussion here). Git was originally written for code and the standard diff utils used are line oriented. Using diff --color-words can marginally obviate to this problem, but the solution is still unsatisfactory to me.

One excellent project is latexdiff . La­texd­iff is a Perl script for vi­sual mark up and re­vi­sion of sig­nif­i­cant dif­fer­ences be­tween two la­tex files. The problem is that, if you split your latex documents in more then one file, you will struggle a bit to use latexdiff as it accepts only one latex file as input.

Another project that I discovered recently is rcs-latexdiff. Rcs-latexdiff is a simple tool to generate a diff of a LaTeX file contained in a Revision Control System like git or svn. The other feature of rcs-latexdiff is the ability to concatenate latex documents split in multiple files into one and pass the result to latexdiff.

The "integration" with git is also straightforward. You just need to add this alias in your .gitconfig file :

[alias]
  ldiff = !rcs-latexdiff

Then using calling git as :

git ldiff main.tex HEAD~1 HEAD

will generate a file diff.tex that you can compile and display as usual. You can easily add a micro functionality to rcs-latexdiff to compile and display the latex document for me. This can either be done using a wrapper script on top of rcs-latexdiff or hacking the script itself.

Average: 2.7 (7 votes)

matplotlib and multiple y-axis scales

25 November 2012 - 4:21pm — abate

This week I had to create a plot using two different scales in the same graph to show the evolution of two related, but not directly comparable, variables. This operation is described in this FAQ on the matplot lib website. Nonetheless I'd like to give a small step by step example...

Consider my input data of the form date release total broken outdated .

20110110T034549Z unstable 29989 133 3
20110210T034103Z wheezy 28900 8 0
20110210T034103Z unstable 30125 209 11
20110310T060132Z wheezy 29179 8 0
20110310T060132Z unstable 30230 945 28
20110410T040442Z wheezy 29487 8 0
20110410T040442Z unstable 31142 991 12
20110510T034745Z wheezy 30247 8 0
20110510T034745Z unstable 31867 610 31
20110610T041209Z wheezy 30328 9 0
20110610T041209Z unstable 32395 328 15
20110710T030855Z wheezy 31403 9 0

I want to create one graph containing three sub graphs, each one containing data for unstable and wheezy. For the sub graph plotting the total number of packages, since the data is kinda uniform, the plot is pretty and self explanatory. The problem arise if we compare the non installable packages in unstable and wheezy, since the data from unstable will squash the plot for wheezy, making it useless.

Below I've added the commented python code and the resulting graph. You can get the full source of this example here.

# plot two distribution with different scales
def plotmultiscale(dists,dist1,dist2,output) :

    fig = plt.figure()
# add the main title for the figure
    fig.suptitle("Evalution during wheezy release cycle")
# set date formatting. This is important to have dates pretty printed
    fig.autofmt_xdate()

# we create the first sub graph, plot the two data sets and set the legend
    ax1 = fig.add_subplot(311,title='Total Packages vs Time')
    ax1.plot(dists[dist1]['date'],dists[dist1]['total'],'o-',label=dist1.capitalize())
    ax1.plot(dists[dist2]['date'],dists[dist2]['total'],'s-',label=dist2.capitalize())
    ax1.legend(loc='upper left')

# we need explicitly to remove the labels for the x axis  
    ax1.xaxis.set_visible(False)

# we add the second sub graph and plot the first data set
    ax2 = fig.add_subplot(312,title='Non-Installable Packages vs Time')
    ax2.plot(dists[dist1]['date'],dists[dist1]['broken'],'o-',label=dist1.capitalize())
    ax2.xaxis.set_visible(False)

# now the fun part. The function twinx() give us access to a second plot that
# overlays the graph ax2 and shares the same X axis, but not the Y axis
    ax22 = ax2.twinx()
# we plot the second data set
    ax22.plot(dists[dist2]['date'],dists[dist2]['broken'],'gs-',label=dist2.capitalize())
# and we set a nice limit for our data to make it prettier
    ax22.set_ylim(0, 20)

# we do the same for the third sub graph
    ax3 = fig.add_subplot(313,title='Outdated Packages vs Time')
    ax3.plot(dists[dist1]['date'],dists[dist1]['outdated'],'o-',label=dist1.capitalize())

    ax33 = ax3.twinx()
    ax33.plot(dists[dist2]['date'],dists[dist2]['outdated'],'gs-',label=dist2.capitalize())
    ax33.set_ylim(0, 10)

# this last function is necessary to reset the date formatting with 30 deg rotation
# that somehow we lost while using twinx() ...
    plt.setp(ax3.xaxis.get_majorticklabels(), rotation=30)

# And we save the result
    plt.savefig(output)

aggregate-results.png

AttachmentSize
aggregate-results.png100.04 KB
Average: 1.5 (15 votes)

Mini Debian Conf 2012 in Paris : Bootstrapping Debian for a new architecture

25 November 2012 - 1:00pm — abate

I just finished to address the awesome debian crowd at the Mini Deb conf in paris. My presentation was about a few challenges we have ahead to bootstrap debian on a new architecture. Johannes Schauer and Wookey did a lot of work in the last few months particularly focusing on Linaro/Ubuntu. After Wheezy I think it is important to catch up with their work and integrate it in debian.

The two main take away messages from my presentation :

  • Add Multi Arch annotations to your packages. This is essential to cross compile packages automatically. We are still not able to cross compile a minimal debian system in debian because we still miss many multi-arch annotations. Experiments show that with these annotations, these packages will cross compile just fine. A lot of work has been done in this direction by Wookey.
  • Debian should consider adding build profiles to build dependencies. Build Profiles are global build dependencies filters to create packages with a different set of functionalities. Build profiles are of the form  Build-Depends: foo (>=0.1) [amd64] <!stage1 bootstrap> | bar.

This week we just reached an important milestone toward a fully automatic bootstrap procedure. Hopefully we are going to tell you more about this work during fosdem 2013

My slides are attached.

AttachmentSize
debminiconf2012.pdf271.56 KB
Average: 4.5 (2 votes)

Generic Graphml Printer for OcamlGraph

31 October 2012 - 11:12pm — abate

Graphml is a nice and widely used graph description format. This is a micro module to print OcamlGraph - graphs in this format.

The signature is minimal. Since in GraphMl all attributes are typed, we only need two functions to describe the name, type a default value for the attributes of each vertex and edge, and two functions to map the value of each vertex and edge to a key / value list.

module type GraphmlSig =
  sig
    include Graph.Sig.G
    (** the format is (key, type of the key, default value *)
    val default_vertex_properties : (string * string * string option) list
    val default_edge_properties : (string * string * string option) list

    (** the format is (key, value *)
    val data_map_vertex : vertex -> (string * string) list
    val data_map_edge : edge -> (string * string) list
  end
;;

module type GraphmlPrinterSig =
  sig
    type t
    val pp_graph : Format.formatter -> t -> unit
    val to_file : t -> string -> unit
  end

To give a small example, we build a simple graph with three vertex and two edges . In this case we only print the id of the node.

open Graphml

module V = struct
  type t = int
  let compare = compare
  let hash i = i
  let equal = (=)
end

module G = Graph.Imperative.Digraph.ConcreteBidirectional(V)

module Gr = struct
  include G
  let default_vertex_properties = ["id","string",None]
  let default_edge_properties = []
  let data_map_edge e = []
  let data_map_vertex v = ["id",string_of_int v]
end

module GraphPrinter = GraphmlPrinter (Gr) ;;

let print g = GraphPrinter.pp_graph Format.std_formatter g ;;

let g = G.create () in
G.add_vertex g 1 ;
G.add_vertex g 2 ;
G.add_vertex g 3 ;
G.add_edge g 1 2 ;
G.add_edge g 1 3 ;
print g;;

Use use ocamlbuild to compile the lot.

$ocamlbuild -use-ocamlfind -package ocamlgraph test.native

The result looks like this. I agree the formatting is not perfect ...
<?xml version="1.0" encoding="UTF-8"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
    http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
<key id="id" for="node" attr.name="id" attr.type="string">
 
<graph id="G" edgedefault="directed">
<node id="n1">
 <data key="id">1</data> </node>
 <node id="n2">
 <data key="id">2</data>
 </node>
 <node id="n3">
 <data key="id">3</data>
 </node>
 
<edge id="e131199" source="n1" target="n2">
 </edge>
 <edge id="e196798" source="n1" target="n3">
 </edge>
 
</graph></graphml>

Using GraphTool, you can easily access a zillion more algorithms from the boost graph library. To be sincere, GraphTool accepts also graphs in dot and gml format. Graphml is the default format for GraphTool as it contains precise type information.

#! /usr/bin/env python
from graph_tool.all import *

g = load_graph("ex.xml")
graph_draw(g, pos=None, output_size=(420, 420), output="ex.pdf")

Update

A refined version of the module Graphml is going to be included in the next release of ocamlgraph ! The tgz is attached to this message.

ex1.png

AttachmentSize
ex1.png23.99 KB
graphml.tgz2.05 KB
Average: 4.5 (2 votes)

lesson learned making a cargo kilt

25 October 2012 - 12:31pm — abate

This entry is not really about computers, technology, or other work-related topics, but more about a hard-hack that I wanted to try for a while. How to make a kilt !!!

After a bit of duck-ducking, I decided to follow this excellent tutorial. At first sight the entire process seems a bit long, but you will realize that after the first read, that everything boils down to 3 steps: measure, fold and pin, sew.

For the measure part, I've the impression that the formula that is given in the instructable (waist/3*8+1) is a bit short for my comfort and taste. This is the size for the internal apron, the folded part that goes all around your left hip, back, and right hip, and the front apron. My suggestion would be to make the inner apron a bit longer then the front apron. This way the kilt will feel in my opinion more comfortable and it will envelop you body completely.

For the fold and pin part, you just need a bit of patience and a ruler. Put the pins parallel to the folding as in the instructable and not perpendicular. This will help you later when sewing everything.

The sewing ... If you know how to use a sewing machine, this is going to be a piece of cake. Otherwise, well, I spent more time troubleshooting the machine then sewing the kilt. I broke a few needles in the process and learned how to thread the machine with my eyes blindfolded. Not to mention that you have to learn how to disassemble this machines in a thousand parts to understand how the thread got stuck. It was fun. A lesson that I've learned is that a sewing machine works much better in the morning than late at night when you are tired and sleepy. Really !

Other then that, it was a fun experience. Maybe I'll make another one to commit this skill to mind. Maybe I'll run a kilt making workshop at the next debconf :)

PA250002.JPG PA250003.JPG

AttachmentSize
PA250002.JPG134.19 KB
PA250003.JPG131.11 KB
Average: 5 (1 vote)

mountoverflowtmp

24 October 2012 - 11:05am — abate

So, what happens if your root partition is full and you reboot your machine ? If it is really full, and in particular there is no space to write anywhere, you might be stuck with a no space on device.

To avoid this problem, there exists a script /etc/init.d/mountoverflowtmp that runs a check to see if there is a minimum acceptable space on /tmp and if there is not, it mounts it overflow. It also checks for unneeded overflow tmpfs for /tmp and removes them if that is appropriate.(src).

But if you do not reboot again, you might get stuck with a mini /tmp directory of 1Mb . This is a neat trick, but you need to know about it to avoid unpleasant and unexpected surprises.

Average: 2.3 (3 votes)

bti with git and identi.ca

19 October 2012 - 1:18pm — abate

I've recently added a hook in my git repository to send all my commits to identi.ca. There are a plethora of methods to do so, and I've chosen bti, that is a small program, available in debian with oauth support.

Following the instructions I found here :

$ bti --config ~/.bti_oauth
Please open the following link in your browser, and allow 'bti' to access your account. Then paste back the provided PIN in here.
http://identi.ca/api/oauth/authorize?oauth_token=$token
PIN: $pin_copied_from_above_URL
Please put these two lines in your bti configuration file (~/.bti):
access_token_key=$key
access_token_secret=$secret
  • add access_token_key and access_token_secret to ~/.bti_oauth
  • try to post something:

$ bti --config ~/.bti_oauth
tweet: testing bti with oauth on identi.ca ...

Once you have bti working, you can add the following script in your post-receive script in the directory hooks.

#!/bin/bash

read oldrev newrev refname
short_refname=${refname##refs/heads/}

/usr/bin/git rev-list $oldrev...$newrev --no-merges --pretty=format:"\!dose ($short_refname): %s (%an) https://gforge.inria.fr/plugins/scmgit/cgi-bin/gitweb.cgi?p=dose/dose.git;a=commit;h=%h" --abbrev-commit | grep -v ^commit | while read a; do echo "$a" | cut -f-140 | bti --shrink-urls --logfile bti.log --config~/.bti_identica ; sleep 1; done

exit 0

your next commit should appear on identi.ca . The format string :

"\!dose ($short_refname): %s (%an) https://gforge.inria.fr/plugins/scmgit/cgi-bin/gitweb.cgi?p=dose/dose.git;a=commit;h=%h"

should print appear like this on identi.ca. Have a look at the man page of git rev-list if you want to add other info. Notice that the url gets shorten automatically and we use the !bang syntax to post in the group dose.

!dose (master): More work on the website (fix css issue) (Pietro Abate) http://identi.ca/url/73941892

For twitter is basically the same thing.

Average: 5 (1 vote)

Improved dose-builddebcheck

15 October 2012 - 7:26pm — abate

The new release of dose, apart from a few bug fixes, ships a new and improved version of dose-builddebcheck (man page). All the improvements done to dose-builddebcheck are from a set of patches submitted by Johannes Schauer in the context of the Bootstrap GSoC. We are still actively working on this. I invite you to read josh's recent blog post on the topic. For more background and discussion, you can have a look at the archives of this mailing list.

Dose-builddebcheck (or buildcheck for short) is similar in intent to dose-debcheck but for source packages. Buildcheck allows you to check, just by looking at the Source and Packages files, if all the dependencies can be satisfied and installed, all this in a matter of seconds (less then 30 secs to test all the source packages in sid on my laptop).

A simple example :

deb-buildcheck.native \
  --deb-native-arch=amd64 \
  tests/DebianPackages/Sid-amd64-Packages-050812.bz2
  tests/DebianPackages/Sid-Sources-050812.bz2

native-architecture: amd64
background-packages: 55945
foreground-packages: 18175
broken-packages: 64

real    0m22.223s
user    0m21.937s
sys    0m0.204s

This will check for all source packages in the Source file if their build dependencies can satisfied on amd64 given the Packages binary file. This is nice but nothing new. A script based on the old edos tools has been around for quite a long time. We want more !

The new exciting feature brought by Johannes is the capability of checking source packages for cross-compilation :

dose-builddebcheck -f -e -s --checkonly picolisp \
  --deb-native-arch=amd64 \
  --deb-foreign-archs=armel \
  --deb-host-arch=armel \
  tests/DebianPackages/Sid-amd64-Packages-050812.bz2 \
  tests/DebianPackages/Sid-armel-Packages-050812.bz2 \
  tests/DebianPackages/Sid-Sources-050812.bz2

(I)Sources: Parsing Sources file tests/DebianPackages/Sid-Sources-single-version-050812.bz2...
(I)Format822: total packages 17723
(I)Format822: Merging repositories
(I)Packages: Parsing Packages file tests/DebianPackages/Sid-armel-Packages-050812.bz2...
(I)Packages: Parsing Packages file tests/DebianPackages/Sid-amd64-Packages-050812.bz2...
(I)Format822: total packages 58095

native-architecture: amd64
foreign-architecture: armel
host-architecture: armel
report:
 -
  package: src:apicolisp
  version: 3.1.0.7-1
  architecture: kopensolaris-amd64,solaris-amd64,amd64,any-i386,any-armel,any-armeb,any-arm,any-avr32,any-hppa,any-m32r,any-m68k,any-mips,any-mipsel,any-powerpc,any-s390,any-sh3,any-sh3eb,any-sh4,any-sh4eb,any-sparc,any-armhf
  source: picolisp (= 3.1.0.7-1)
  status: ok
 
background-packages: 75818
foreground-packages: 17723
broken-packages: 0

This will check, if the source package 'picolisp' in the Sources file can be cross compiled for 'armel' on the native architecture 'amd64' given the list of binary packages in the Packages file. The generated report is, as for dose-debcheck, encoded in yaml and it can be simply parsed using an off-the-shelf library.

Apt is the canonical tool that can be used to check if a package can be cross compiled. Josh found a few discrepancies between dose and apt results. This was a very good test for both tools: Bug #683786 is a very interesting read...

Dose 3.1.1 (the latest release) should land soon in experimental. Otherwise you can get it from our (new) project homepage.

Average: 5 (1 vote)
Syndicate content