two simple tips to speed up ocaml compilation

Date Tags ocaml

The dose3 code base is getting larger and larger and waiting 40 secs each time I want to recompile the library is not acceptable anymore. Recently I tried to understand why the ocaml compiler was so slow .

To compile dose, I use a combination of ocamlbuild, ocamlfind, camlp4 and Makefile to drive everything.

The first important problem was related to my naive use of camlp4. I extensively use the Camlp4MacroParser module for the conditional compilation of certain features and actually I use camlp4 to process all my source code. The bottom line her is that using the byte code version of camlp4o is definitely a bad idea.

To use the native binary you need to change the pre-processing directive in the file _tags :

<*/*.ml{i,}>: pp(camlp4o.opt -I_build Camlp4MacroParser.cmxs)

This will use the opt version of camlp4 and the cmxs instead of the .cma. The next problem is that debian, for some reason (I guess lack of time, more then anything else), does not ship the cmxs libraries of the Camlp4MacroParser. In other to make this available, I added a simple compilation rule in my make file to create the file Camlp4MacroParser.cmxs in the _build directory.

 ocamlopt -shared $(shell ocamlc -where)/camlp4/Camlp4Parsers/Camlp4MacroParser.cmx -o _build/Camlp4MacroParser.cmxs

The second BIG problem I solved yesterday was related to the tag traverse. Since I use git as VCS, every invocation to ocamlbuild was triggering a recursive traverse of all the .git directory with a considerable waste of time.

To solve this problem, I’ve reverted the default ocamlbuild and explicitly add only those directory that I’m really interested to traverse.

true: -traverse
<{common,algo,doseparse,deb,rpm,applications,experimental}/**>: traverse

This is actually needed not to compile the code itself, but to generate the documentation with ocamldoc.


Package Managers Comparison - take 2

On year ago, we (the mancoosi team) published a comparison study regarding the state of the art of dependency solving in debian. As few noticed, the data presented had few glitches that I promised to fix. So we’ve repeated our tests using exactly the same data we used one year ago, but now using the latest available versions of all package managers as available in debian unstable.

During the last year, three out of four solver that we evaluated release a major upgrade so I expected many improvements in performances and accuracy.

  • apt-get 0.8.10 -> 0.9.7
  • aptitude 0.6.3 -> 0.6.7
  • smart 1.3-1 -> 1.4
  • cupt 1.5.14.1 -> 2.5.6

Mpm, our test-bench for new technologies, changed quite a bit under the wood as a consequence of the evolution of apt-cudf and the recent work done in apt-get to integrate external cudf dependency solvers.

Overall the results of our study are not changed. All solvers but mpm, that is based on aspcud, are not scalable as the number of packages (and alternatives) grows. It seems that Smart is the solver that does not give up, incurring in a timeout (fixed at 60 seconds) most of the time. Aptitude is the solver that tried to give you a solution, doesn’t matter what and as result providing solutions that do not satisfy the user request for one reason or the other. Apt-get does surprisingly well, but it gives up pretty often showing the incomplete nature of it’s internal solver. Cupt sometimes timeout, sometimes gives up, but when it is able to provide an answer it is usually optimal and it is very fast … Mpm consistently finds an optimal solution, but sometimes it takes really a long time to do it. Since mpm is written in python and not optimized for speed this is not a big problem for us. The technology used by mpm is now integrated in apt-get and I hope this will alleviate this problem.

All the details of our study can be found one the Mancoosi Website as usual with a lot of details. For example here you can find the results when mixing four major releases : sarge-etch-lenny-squeeze.

Comments are more then welcome.


fix low volume with .asoundrc

For quite a while I had this problem with low volume while listening music or watching a video. The default mixer controls of alsa only give you master and PCM, but even setting everything to 100% , the volume is still annoyingly low. For quite a while I suspected this was a software problem. For example VLC is able to boost the volume level at 140% …

A while ago I found an interesting post (that I can’t find anymore) on this subject. The author suggested to add this snippet to your ~/.asoundrc .

pcm.dmixout {
  type plug
  slave {
     pcm "dmix"
  }
}

pcm.!default {
      type plug
      slave.pcm "softvol"
  }

  pcm.softvol {
      type softvol
      slave {
          pcm "dmix"
      }
      control {
          name "Pre-Amp"
          card 0
      }
      min_dB -5.0
      max_dB 20.0
      resolution 6
  }

This will add a new control setting in your mixer controls, named “Pre-Amp” that will allow you to boost considerable the sound in same situations. I’ve noticed that with this new control, the sound can get a bit distorted. Anyhow, better then not hearing anything.


tweaking my awesome rc.lua config file

It’s true, sometimes the title of my post try to be more as web search engine-friendly as possible…

Yesterday I was very unhappy of the new behavior of awesome. All of the sudden windows were not receiving focus automatically on screen change, forcing me to either use the mouse of to do some other keyboard action. so for the first time I decided to have a look at the awesome config file. At file I’ve to admit it’s a bit intimidating. I don’t know a word of lua and it’s syntax is not really intuitive.

First a word about my setup. I use xfce4 + awesome. I don’t want thunar of xfcedestop running and I don’t want the wibox bar hiding somewhere…

I’ll show highlight few chunks of the file. If interested, the whole file is attached to this post. I wrote only a small function, everything else are bit and pieces that I found on the net.

A bit debugging code that is actually very handy when trying to understand what is going on :

if awesome.startup_errors then
    naughty.notify({ preset = naughty.config.presets.critical,
                     title = "Oops, there were errors during startup!",
                     text = awesome.startup_errors })
end

do
    local in_error = false
    awesome.add_signal("debug::error", function (err)
        -- Make sure we don't go into an endless error loop
        if in_error then return end
        in_error = true

        naughty.notify({ preset = naughty.config.presets.critical,
                         title = "Oops, an error happened!",
                         text = err })
        in_error = false
    end)
end

A small function to raise one window in each screen when I move left or right. This is actually not what I want, but it is better then nothing. What I’d really like is to raise the window under the mouse pointer. However the function awful.mouse.client_under_pointer () does not seem working as expected… so this is something I want to fix sometimes in the future.

function myautofucus() 
   awful.client.focus.byidx(1)
   if client.focus then client.focus:raise() end
   local c = awful.mouse.client_under_pointer ()
   if c then client.focus = c; c:raise() end
end

-- {{{ Key bindings
globalkeys = awful.util.table.join(
    awful.key({ modkey,           }, "Left",   function () awful.tag.viewprev() ; myautofucus() end ),
    awful.key({ modkey,           }, "Right",  function () awful.tag.viewnext() ; myautofucus() end ),

And then the rule I use to adjust different type of program I currently use. To get the class of a window you can use xprop .

-- {{{ Rules
awful.rules.rules = {
    -- All clients will match this rule.
    { rule = { },
      properties = { border_width = beautiful.border_width,
                     border_color = beautiful.border_normal,
                     focus = true,
                     keys = clientkeys,
                     buttons = clientbuttons,
                     size_hints_honor = false
                 } },
    { rule = { class = "MPlayer" },
      properties = { floating = true } },
    { rule = { class = "pinentry" },
      properties = { floating = true } },
    { rule = { class = "gimp" },
      properties = { floating = true } },
    { rule = { class = "Chromium" },
      properties = { tag = tags[1][8],
                     floating = true } },
    { rule = { class = "Iceweasel" },
      properties = { tag = tags[1][2] } },
    { rule = { class = "Pidgin" },
      properties = { tag = tags[1][7],
                     floating = true } },
    { rule = { class = "Skype" },
      properties = { tag = tags[1][9],
                     floating = true } },
    { rule = { class = "Orage" },
      properties = { floating = true } },
    { rule = { class = "Wicd-client.py" },
      properties = { floating = true } },
    { rule = { instance = "plugin-container" },
        properties = { floating = true } },

}

And this small bit is to remove the wibox :

a = awful.wibox()
a = nil

All in all lua is a nice language and awesome is a really flexible WM. Having the possibility to script its configuration file gives you unlimited space to experiment and automate boring and repetitive actions. Very nice :)


encrypted unattended backups with anacron and duply

If you are using a laptop as your main computation device having reliable and uptodate backup should be a strong priority. A laptop can be stolen, can fall or as any other computer die in a cloud of smoke (like my last laptop did $^%# ).

For quite sometimes I’ve used duply that is a nice an intuitive frontend to duplicity. Duply allows you to have backups that are : encrypted incremental * offsite

This is basically everything that I want from a backup solution. A nice article about duply here. Since I don’t want to remember to run duply every now and then, I want a daemon to think about it at my place. If you are using a laptop, cron is not your friend as you laptop might be sleeping or simply not connected to the network.

A solution that I knew about for years is anacron. Today I set it up following these instructions. I’ve created 4 different profiles to backup different part of my home directory : home, pictures, media and projects and instructed anacron to run the backup for home and project daily while the rest only weekly.

Since the backup is unattended I’ve used the options of duply that will take care of creating a new full backup each month (2 months for pictures and music). To do this you need to call duply as

duply <profile> backup

and specify the option MAX_FULLBKP_AGE=1M in the conf file of your profile.

I can not sleep better … somebody/something else is looking after my data

And …

something else I might add in the next few days is a script to detect the type of network I’m connected to and to launch the backup accordingly. Some like :

#!/bin/bash

$backuphost=backup.example.org

#gw="$(ip route | awk "/^default via/ {print \$3}")"
#ping -c 1 "$gw" > /dev/null

ping -c 1 "$backuphost" > /dev/null
result="$?"

dev="$(ip route | awk "/^default via/ {print \$5}")"
if [ $dev == "eth0" & $result == 0 ]; then
  # you can run your backup
  echo "wired"
  exit 0
elif [ $dev == "wlan0" & $result == 0 ]; then
  # you can run your backup ... or not
  echo "wireless"
  exit 0
else
  # you should try again another time
  echo "disconnected"
  exit 1
fi