skype on amd64 (debian unstable)

to install skype on a debian unstable machine :

now we need to fix a bunch of dependencies:

run skype.

blahhhhhh . It used to be easier... Sometimes I really despise myself for using this closed source software.

Average: 1 (53 votes)

simple expat based xml parser

The other day I needed a small xml parser to convert an xml document into a different format. First I tried xml-light. This is a simple parser all written in ocaml that stores the parser xml document in an ocaml data structure. This data structure can be user to access various fields of the xml document. It does not offer a dom-like interface, but actually I consider this a feature. Unfortunately xml-light is terribly slow. To parse 30K-plus lines of xml it takes far too long to be considered for my application.

The next logic choice was to try Expat, that is a event-based parser and it is extremely fast. Since using an event based parser can be a bit cumbersome (and I already had written of bit of code using xml-light), I decided to write a small wrapper around expat to provide a xml-light interface to it.

The code is pretty simple and the main idea is taken from the cduce xml loader.

First we provide a small data structure to hold the xml document as we examine it. Nothing deep here. Notice that we use Start and String as we descend the tree and Element we we unwind the stack.

type stack =
  | Element of (Xml.xml * stack)
  | Start of (string * (string * string) list * stack)
  | String of (string * t * stack)
  | Empty
and t =
  | PCData
  | CData

Then we need to provide expat handlers to store xml fragments on the stack as we go down. Note that we have an handler for cdata, but not an handler for pcdata as it is the default.

let pcdata buff = (Xml.PCData buff)
let cdata buff = (Xml.CData buff)

let rec create_elt acc = function
  | String (s,CData, st) -> create_elt (L.push (cdata s) acc) st
  | String (s,PCData, st) -> create_elt (L.push (pcdata s) acc) st
  | Element (x,st) -> create_elt (L.push x acc) st
  | Start (tag,attrlst,st) -> stack := Element(Xml.Element(tag,attrlst,acc),st)
  | Empty -> assert false

let start_cdata_handler () = txt.cdata <- true ;;

let start_element_handler tag attrlst =
  if not (only_ws txt.buffer txt.pos) then begin
    let str = String.sub txt.buffer 0 txt.pos in
    if txt.cdata then
      stack := String (str, CData, !stack)
    else
      stack := String (str, PCData, !stack)
  end
  ;
  txt.pos <- 0;
  txt.cdata <- false;
  stack := Start (tag,attrlst,!stack)
;;

let end_element_handler _ =
  let acc =
    if only_ws txt.buffer txt.pos then L.empty
    else
      let str = String.sub txt.buffer 0 txt.pos in
      if txt.cdata then L.one (cdata str)
      else L.one (pcdata str)
  in
  txt.pos <- 0;
  txt.cdata <- false;
  create_elt acc !stack
;;

let character_data_handler = add_string txt ;;

At the end we just register all handlers with the expat parser and we return the root of the xml document.

let parse_string str =
  let p = Expat.parser_create None in
  Expat.set_start_element_handler p start_element_handler ;
  Expat.set_end_element_handler p end_element_handler ;
  Expat.set_start_cdata_handler p start_cdata_handler ;
  Expat.set_character_data_handler p character_data_handler ;
  ignore (Expat.set_param_entity_parsing p Expat.ALWAYS);
  Expat.parse p str;
  Expat.final p;
  match !stack with
  |Element (x,Empty) -> (stack := Empty; x)
  | _ -> assert false

I've copied the xml-light methods and to access the document in a different file. I've also made everything lazy to save a bit of computing time if it is only necessary to access a part of a huge xml document.

The complete code can be found here: git clone https://www.mancoosi.org/~abate/repos/xmlparser.git

UPDATE

The other that I was made aware that this parser has a serious bug when used on a 32 bit machine. The problem is that the maximal string size on a 32bit machine is equal to Sys.max_string_length that is roughly 16Mb . If we read and parse a bit document document at once with IO.read_all , we immediately get an exception. The solution is to parse the document incrementally using the new function parser_ch below that get a channel instead of a string and run the expat parser incrementally :

let parser_aux f =
  let p = Expat.parser_create None in
  Expat.set_start_element_handler p start_element_handler ;
  Expat.set_end_element_handler p end_element_handler ;

  Expat.set_start_cdata_handler p start_cdata_handler ;

  Expat.set_character_data_handler p character_data_handler ;
  ignore (Expat.set_param_entity_parsing p Expat.ALWAYS);
  f p;
  match !stack with
  |Element (x,Empty) -> (stack := Empty; x)
  | _ -> assert false

let parse_str str =
  let f p =
    Expat.parse p str;
    Expat.final p;
  in
  parser_aux f

let parse_ch ch =
  let f p =
    try while true do
      Expat.parse p (IO.nread ch 10240)
    done with IO.No_more_input -> Expat.final p ;
  in
  parser_aux f

Average: 1.1 (58 votes)

apt-get dist-upgrade

During the weekend I upgraded my laptop to sqeeze. I usually track unstable pretty closely, but in between transition I gave myself a bit of slack in order to avoid messing up with the gnome transition. The result is ok, NetworkManager Just Work !!!, the new kernel seems pretty snappy. I finally get the power status for my network card.

My laptop is a old dell latidute x200. I always had problem with the graphic card and Xorg. With this upgrade I've always motivated myself to find a solution. Not surprisingly it was quite easy. I've added these option to my xorg.conf :

Section "Device"
        Identifier "Configured Video Device"
        Driver "intel"
        Option "Tiling" "false"
        Option "FramebufferCompression" "false"
        Option  "XAANoOffscreenPixmaps" "true"
        Option  "AccelMethod" "EXA"
        Option  "ExaNoComposite" "true"
EndSection
I'm not entirely sure if I need them all. I've noticed that already my screen corruptions go away with "tiling" and "framebuffercompression" set to false. But the life changing options are the accell method (EXA seems much more stable) and the "ExaNoComposite".

What I've left to figure out is to fix the hibernate function, that is still not very reliable as it works 8 out of 10 times.

After 1.3Gb of updates, I'm happy I'm again surfing the unstable wave.

Average: 1.4 (10 votes)

gps on the freerunner

This morning I spent sometimes trying to understand how the GPS subsystem on the freerunner works. I'm using SHR unstable. These information might be incomplete or wrong, so just use them with a grain of salt.

Ok, at the beginning the clients where connecting to the GPS device either directly :

  • serial device -> client

or using gpsd

  • serial device-> gpsd-> client

so far so good. In the end an application like tangogps or navit only needed to have a fix that is not that difficult to obtain from the raw device input.

But what if I want to handle and gather information from multiple GPS devices ? the idea here is to add an additional layer of indirection to make like easier to clients. On top of it, since the freerunner used dbus to communicate and frameworkd as a communication broker we have now two different players.

  • gps -> ogpsd -> fso-gpsd -> tangogps

From [2], this is the story:

  • gpsd - this is our good old friend from http://gpsd.berlios.de/ "True gpsd" someone called it.
  • ogpsd - this is a subsystem of FSO's frameworkd. It replaces gpsd it implements the Gypsy API communicating with applications via dbus
  • fso-gpsd - is a compatibility shim to translate Gypsy messages for applications that expect gpsd
  • clients ...

What is gypsy [3] ? Gypsy is a GPS multiplexing daemon/protocol which allows multiple clients to access GPS data from multiple GPS sources concurrently.

Now, my point was to use the GPS information collected by these two fantastic projects opencellid [5] and cellhunter [6]. In order to do that I would need to add a "fake" gps device to feed ogpsd with information retrieved from the cell database.

If the architecture I've described here is correct, it should not be to difficult to add the missing bit in ogpsd [4] ...

UPDATE: It seems that there is already an implementation [8,9] of agps fetching data from agps.u-blox.com and based on gllin [7] , but you will need a data connection to use this one.

[8] http://www.opkg.org/package_127.html [9] http://lists.openmoko.org/pipermail/community/2008-June/018680.html


http://gpsd.berlios.de/NMEA.txt

[2] http://kerneltrap.org/index.php?q=mailarchive/openmoko-community/2008/9/24/3395094

[3] http://gypsy.freedesktop.org/wiki/

[4] http://git.freesmartphone.org/?p=framework.git;a=tree;f=framework/subsystems/ogpsd;hb=HEAD

[5] http://www.opencellid.org/

[6] http://78.47.116.33/~hole/cellhunter/

[7] http://wiki.openmoko.org/wiki/Gllin

Average: 1.2 (20 votes)

sharing a svn repo

Recently I had few problems with a svn repository that is shared between multiple ssh users. I followed the instructions in the svn book and to solve the problem once for all I recreated the repo from scratch. Briefly:

svnadmin dump SVN > /tmp/dump
mv SVN SVN-old
svnadmin create SVN
chmod -R g+rw SVN
find SVN -type d -exec chmod g+s {} \;
chown -R root.cvs SVN
 svnadmin load SVN < /tmp/dump

hopefully this is gonna work. hopefully, otherwise I guess I missing something that is very basic !

Average: 1.4 (11 votes)

update to drupal 6

Today I've finished the long due migration to drupal 6 . With drupal 7 almost ready was kind of important not to stay to far behind the latest version. I've to say that drupal is getting better and better. This new stable version has a lot of eye candies (web 2.0 style) and improved functionalities. The update of the drupal core modules was almost painless (apart for a stupid mistake that corrupted my database...). The module upgrade took a bit more of time, but in the end I managed to get back everything I had before (and to remove a lot of old modules).

The only thing that I want to mention is about the interference of the pearwiki filter with the geshi filter for code highlighting. This is the relevant bug that contains a patch for the pearwiki module.

This is the final list of modules that I use (and upgraded):

  • advanced_help
  • captcha_pack
  • guestbook
  • pathauto
  • spamspan
  • views
  • biblio
  • cck
  • pearwiki_filter
  • tagadelic
  • captcha
  • geshifilter
  • inline
  • spam
  • token
Average: 1.4 (21 votes)

storm, sqlite and string representation

Arghhh today I've discovered reading this bug report that specifying strings are RawStr() in strom, they are actually stored as blobs in sqlite3. The very bad side-effect is that string comparison does not work !!!

The right way to store strings with storm is to use the Unicode() data type instead and to wrap all your strings with the unicode function. If you need "utf-8", you can pass it as optional argument to it. Now string comparisons are 10 times faster !!!!!!!! Argggg

Average: 1.1 (59 votes)

add sqlite3 collation with python 2.5 and django

a while ago I wrote about enabling the sqlite3 extension with storm . This is how you do it with the Django ORM. The collation is the same and all details are in the old post. The only tricky part is to establish the connection with cursor = connection.cursor() before calling the function to enable the extension. Failing to do so, will result in an error as the connection object will be null.

    def add_collation():
        from django.db import connection
        import sqlitext
        cursor = connection.cursor()
        sqlitext.enable_extension(connection.connection,1)
        cursor.execute("SELECT load_extension('sqlite/libcollate_debian.so')")

Average: 1.3 (13 votes)

select time intervals in sql

consider the following example :

CREATE TABLE elem (
    id INTEGER,
    name VARCHAR,
    num INTEGER
);
CREATE TABLE proxy (
    elem_id INTEGER,
    time_id INTEGER
);
CREATE TABLE TIME (
    id INTEGER,
    TIMESTAMP DATESTAMP
);

INSERT INTO elem VALUES (0,"one",1);
INSERT INTO elem VALUES (1,"one",2);
INSERT INTO elem VALUES (2,"one",3);
INSERT INTO elem VALUES (3,"two",1);

INSERT INTO proxy VALUES (0,0);
INSERT INTO proxy VALUES (0,1);
INSERT INTO proxy VALUES (1,2);
INSERT INTO proxy VALUES (1,3);
INSERT INTO proxy VALUES (2,4);
INSERT INTO proxy VALUES (3,4);

INSERT INTO TIME VALUES (0,"12/12/2008");
INSERT INTO TIME VALUES (1,"13/12/2008");
INSERT INTO TIME VALUES (2,"14/12/2008");
INSERT INTO TIME VALUES (3,"15/12/2008");
INSERT INTO TIME VALUES (4,"16/12/2008");

What I want is to write a query that will group for each (name,num) the start and end edges of the interval given by the table time:

For example:

one | 1 | 12/12/2008 | 13/12/2008
one | 2 | 14/12/2008 | 15/12/2008
one | 3 | 16/12/2008 | 16/12/2008
two | 1 | 16/12/2008 | 16/12/2008

First we create a simple view to unclutter the query statement.

 CREATE temp VIEW all_t AS SELECT elem.name,elem.num,TIME.TIMESTAMP FROM elem,TIME,proxy WHERE proxy.elem_id = elem.id AND TIME.id = proxy.time_id ;

Then the sql query is pretty straightforward ...

SELECT name,num,MIN(TIMESTAMP),MAX(TIMESTAMP) FROM all_t GROUP BY name, num;
one|1|12/12/2008|13/12/2008
one|2|14/12/2008|15/12/2008
one|3|16/12/2008|16/12/2008
two|1|16/12/2008|16/12/2008

Average: 1.3 (7 votes)

dynamic forms with django

Today I started learning how to write web forms in django. My quest was to write a simple search form where I could specify multiple criteria . Django is a very nice and flexible framework written in python and it is also reasonably well documented.

I don't feel writing much today. this is the code :

The tricky part was to understand how to re-display the view and to add a new field. This is easily accomplished in django using the formset class that allows to put display together more then one form. In this case the logic is simple. First we display an empty form with two submit buttons, Add and Search. Search brings the obvious result. Add gets the data that was submitted, validates it and re-display the form with an additional field. Note also that the default validation functions are used transparently to validate the input of each sub-form.

views.py

from django.http import HttpResponse
from django.shortcuts import render_to_response
from cudfdep.depo.forms import SearchField
from django.forms.formsets import formset_factory
from django.http import Http404

def search(request):
    SearchFormSet = formset_factory(SearchField)
    data = {'formset': SearchFormSet(), 'debug' : 'default'}
    if request.method == 'POST':
        formset = SearchFormSet(request.POST)
        if formset.is_valid() :
            if request.POST.get('form-submit') == 'Add':
                data = {'formset': SearchFormSet(initial=formset.cleaned_data), 'debug' : 'add'}
            elif (request.POST.get('form-submit') == 'Search' ):
                data = {'formset': SearchFormSet(), 'debug' : 'result' }
            else :
                raise Http404('Invalid request')
        else :
            data = {'formset': formset, 'debug' : 'not valid'}

    return render_to_response('search_form.html', data)

The forms file describes the form logic to be displayed.

forms.py

from django import forms

class SearchField(forms.Form):
    date = forms.DateTimeField(label='date', widget=forms.DateTimeInput)
    arch = forms.ChoiceField([(0,'i386'),(1,'amd64')], widget=forms.Select,initial=0)
    release = forms.ChoiceField([(0,'etch'),(1,'lenny'),(2,'sid'),(3,'squeeze')],widget=forms.Select,initial=0)

This is the template :

search_form.html

<html>
<head>
    <title>Search Form</title>
</head>
<body>
    <form action="" method="post">
      {{ formset.management_form }}
      <table>
          {% for form in formset.forms %}
          {{ form.as_table }}
          {% endfor %}
      </table>
      <input name='form-submit' type="submit" value="Add">
      <input name='form-submit' type="submit" value="Search">
    </form>
    {% if debug %}
        <p style="color: red;">{{ debug }}</p>
    {% endif %}
</body>
</html>

Average: 2.7 (23 votes)
Syndicate content