svn db/current problem

Date Tags svn

A while ago one of my svn repositories became all the sudden inaccessible. It turned out that the file db/current got corrupted for some reason, leaving the repo in an inconsistent state. The file db/current in a svn repository (using the FSFS backend) contains the largest revision number and the next unique node and copy ids. Googling around I found this old message [1] explaining the problem, but not providing any satisfactory solution.

My first attempt was to dump the db using svnadmin and then load it back. This didn’t fix the problem. Then I tried again with svnadmin recover, to not aval.

This morning I found the solution. svnadmin recover is actually able to restore the db/current file, but only from svn version 1.5 (I was using ver 1.4). So after an update (thanks backports!) I successfully managed to recover the repo. This blog post [2] is definitely informative.

[1] http://svn.haxx.se/users/archive-2005-11/0435.shtml [2] http://www.farside.org.uk/200703/svnadmin_recover


graph sql tables

I’ve been looking for a while for a tool to generate graphs from sql schemas. This obvious operation seems implemented in many graphical design tools for database, but up until now, I didn’t manage to find a command line tool to output a simple graph. Well, the answer came from sqlfairy ( http://sqlfairy.sourceforge.net/ ), a perl library to manipulate structured data definitions.

for example, to generate a nice picture from an existing sqlite database, this is the pipeline :

echo ".schema" | sqlite3 database > schema.sql

sqlt-graph --from=SQLite -o schema.png schema.sql

UPDATE: if you actually want to display relations between your tables, you can use FOREIGN KEY / REFERENCE definitions. Sqlite3 actually parses these definitions, but does not enforce them. On the other hand sqlfairy does not accept foreign key definitions if using the sqlite parser backend. The solution is to use the MySQL parser backend instead so to generate a nice graph.

An example is attached.

UPDATE: if you use postgresql, you should definitely have a look at pg_autodoc : http://www.rbt.ca/autodoc/index.html


Apache Reverse Proxy

Date Tags apache2

What if you want to hide an internal server ? you need to configure apache to act as a reverse Proxy. This is the small snippet you need to configure a proxy pass on the public server. Everything is well known expect (for me at least) the directive ProxyPreserveHost that is needed to rewrite the HTTP_SERVER parameter for the hidden server. This way the application can ignore the fact that is behind a proxy.

<VirtualHost IP:443>
    ServerName public.domain.org
    SSLProxyEngine on
    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass / https://hidden.hiddendomain/
    ProxyPassReverse / https://hidden.hiddendomain/
    <Proxy https://hidden.hiddendomain>
            Order deny,allow
            Deny from all
            Allow from all
    </Proxy>
</VirtualHost>

sympa migration step by step

Date Tags sympa

I had to migrate a sympa installation from one domain to another. The sympa manual is certainly not of a much help. After a bit of trial and error this is my recipe.

A small warning: keep your backup available at all times. This procedure works with grep, sed and friends and it is not certainly fool proof.

  • Switch off your MTA and sympa daemon. From this moment on, you will have about 12h to finish your migration. After this time, you might start to loose incoming emails.

  • Dump the sql database. With mysql you can use mysqldump to obtain a copy of you db. Since the sympa daemon if off, it should be safe to perform a hot dump.

olddomain# mysqldump sympa > sympa.sql
  • Make a backup of your archives. On debian everything you need should be in /var/lib/sympa. Moreover you should remember to make a backup copy of /etc/sympa.
olddomain# tar zcvf sympa.tgz /var/lib/sympa
  • If you are migrating sympa to a different machine, this is a good time to copy everything there, configure the MTA to accept messages for the new domain. Since nobody knows about this new domain yet (you should give yourself a bit of time to test it !), you can switch on the MTA and sympa.

  • Now it’s time to recreate the sympa db. But before that, you need to do a bit of sed magic in the db dump file. The problem here is that the domain of all your mailing lists is in the database. If you are using a domain for your lists, then it should be easy.

newdomain# cat dump | sed -i 's/sympa.olddomain.org/sympa.newdomain.org/g' > dump.new
  • Ok then. We are ready to load the new dump in the db and start the sql server. Remeber also to change the domain in the configuration files in the /etc/sympa directory. If everything goes according to plans, you should be able to see the list of all you mailing lists.
newdomain# mysql sympa < dump.new
  • Almost done. We need to fix the config files for each list. This should do :
for i in `find -name config` ; do cp $i /var/lib/sympa/expl/sympa.newdomain.org/$i.sed ; done

for i in `find -name config` ; do sed 's/sympa\.olddomain\.org/sympa\.newdomain\.org/' < $i.sed > $i ; done
  • What is left to do it to recreate the archives. Sympa can recreate the entire web archives from the mailing lists row mboxes. To make this work, you need to change the name of all directories in /var/lib/wwsarchive. What I’ve done is something like:
newdomain# cd /var/lib/wwsarchive
newdomain# for i in `ls` ; do mv $i ${i/@sympa.olddomain.org/@sympa.newdomain.org} ; done
  • Now go to the admin interface of sympa and ask for a complete rebuild of the web archives. This should schedule the rebuild. You can check the sympa logs to make sure the procedure worked. In a few minutes, depending on the size of you archives, you should be able to access your web archives.

  • Now it’s time to add a rewrite or redirect in you old MTA to send all emails for you old domain, to the new domain. Then you should restart the old MTA, and see all the email flowing to the new sympa daemon. In order to completely remove the old domain, you should add a permanent error sometimes in the future.

NOTA (molto) BENE

I still think that sympa is a bloated and badly written piece of software. It is developed and sponsored by a consortium of French universities. If you want an efficient and manageable mailing list manager you should look somewhere else.

Update

Just for comparison… today I migrated a mailman installation to a new server. Well, the gist is (more or less) tar zcvf mailman.tgz /var/lib/mailman && scp mailman.tgz newhost: && cd /var/lib && tar zxvf mailman.tgz . Configure/Start mailman. The rest is history.

Ahhh if you change domain, there is a nice script called fix_url in the mailman directory. To easy to beat a dead horse, isn’t ?


avalaible bdd libraries

UPDATE

In the end I found ocaml bindings for the Buddy BDD library. yuppi \o/


BDDs or Binary Decision Diagrams are a method of representing boolean expressions. I searched the net for available BDD libraries (I’ve considered different BDD variants in my research). In particular I focused on OCaml implementations. My conclusion is that as today there is no viable native implementation of an efficient bdd library. It seems common knowledge (take this cum granis salis , I haven’t done any work in this direction) that the fastest bdd library is buddy, but there are not OCaml bindings to it.

Next step would be to run few tests and to evaluate the available OCaml implementations.

These are my findings. I’m sure the list is not exhaustive, also because many bdd implementations are usually anonymously embedded as part of larger projects. In particular the model checking community and the planning community do extensive use of BDDs :

First of all what is a bdd : Wikipedia .

Ocaml libraries (bindings and native) :

  • Jean-Christophe Filliâtre (ocaml implementation) Paper Code

  • bindings to the CUDD BDD library Code

  • Olivier Michel (ocaml implementation) Code

  • Xavier Leroy (part of an experimental sat solver) Code

  • John Harrison Code

  • Ocaml implementation (who is the author ?) Wiki

C/C++ Libraries

Other Languages

Relevant Mailing list Messages

The ocaml ml has several references to BDDs. These are 3 interesting threads that I’ve used as a starting point for my research.

  • In this Thread there is mention of a possible binding for Jørn Lind-Nielsen’s BDD library BuDDy. I’m wondering if this binding was ever released.

  • In this Thread Alain Frish points out that none of existing ocaml libraries implements automatic reordering of variables… And I don’t know if the state of affairs is changed at this regard.

  • In this Thread David Mentre announces a preliminary work on binding for the cudd library, but the link is broken… and there is a mention to a caml-light implementation of a robdd library that I was also not able to retrieve.