command line parsing in python

Date Tags python

if a good API is meant to let you write code more concisely, I would say the argparse achieves this goal perfectly. For example, with getopt you usually write something like :

try :
    opts, cmdline = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
except getopt.GetoptError, err:
    print str(err)
    usage()
    sys.exit(1)
output = None
verbose = False
for o, a in opts:
    if o == "-v":
        verbose = True
    elif o in ("-h", "--help"):
        usage()
        sys.exit()
    elif o in ("-o", "--output"):
        output = a
    else:
        assert False, "unhandled option"
if not cmdline :
    usage()
    sys.exit(1)

This is just to add two command line options, help and verbose and without taking care of positional arguments… Now, the magic of argparse:

    parser = argparse.ArgumentParser(description='description of you program')
    parser.add_argument('-v', '--verbose')
    parser.add_argument('timestamp', type=int, nargs=1, help="a unix timestamp")
    parser.add_argument('inputfile', type=str, nargs=1, help="input file")
    args = parser.parse_args()
    parser.print_help()

tadaaaaa .

usage: import.py [-h] [-v VERBOSE] timestamp inputfile

description of you program

positional arguments:
  timestamp         a unix timestamp
  inputfile             input file

optional arguments:
  -h, --help            show this help message and exit
  -v VERBOSE, --verbose VERBOSE

And this has all the bells and whistle you want. Like checking if you pass both positional arguments, if the file exists, if the timestamp is really and integer file, etc. Very nice indeed !


expose you command line application on the web with python and fcgi

So one day you’re too lazy to write a fcgi library for your favorite language but you want nonetheless expose an application on the web… Then use python ! There are quite a few frameworks to run fcgi with python, but if you want something easy, I think that flup is for you.

The code below takes care of few aspects for you. First flup span a server talking at port 5555 on localhost. You can configure it to be multi thread is you want to. Then using the cgi module we make sure that the input is clean and ready to use. Finally we run your fantastic application as DOSOMETHING. If your application is a simple program, of course there is no reason to write a fcgi. A common cgi will pay the bill. However, if your application can benefit from some form of caching, then maybe writing the web related stuff in python and use the application as a black box can be a nice idea.

If might want to check out [flup http://trac.saddi.com/flup] and [werkzeug http://werkzeug.pocoo.org/]. I’ve not used the last one, but it seems more complete then flup.

#!/usr/bin/python
# -*- coding: UTF-8 -*-

from cgi import escape
import sys, os
from flup.server.fcgi import WSGIServer
import subprocess
import urlparse
import cgi
# expose python errors on the web
import cgitb
cgitb.enable()

def app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])
    yield '<html><head></head>'
    yield '<body>\n'

    form = cgi.FieldStorage(fp=environ['wsgi.input'], environ=environ,keep_blank_values=1)

    yield '<table border="1">'
    if form.list:
        yield '<tr><th colspan="2">Form data</th></tr>'

    for field in form.list:
        yield '<tr><td>%s</td><td>%s</td></tr>' % (field.name, field.value)

    yield '</table>'

    if form.has_key('c') and form.has_key('l'):
        cat = form.getvalue('c')
        pl = form.getlist('l')
        command = DOSOMETHING(cat,pl)
        results = subprocess.Popen(command,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        (i,e) = results.communicate()
        yield '%s\n' % i
        yield '%s\n' % e

    yield '</body>\n'
    yield '</html>\n'

WSGIServer(app,bindAddress=('localhost',5555)).run()

upload a file using httplib

I want to share a small snippet of code to upload a file to a remote server as a “multipart/form-data” . The function below gets two arguments. The server url ( ex: http://server.org/upload ) and a filename. First the filename encoded as a “form-data”, then we use httplib to POST it to the server. Since httplib wants the host + path in separate stages, we have to parse the url using urlparse.

The receiving server must accept the data and return the location of the newly created resource. There are many snippet on the web, but I felt they were all incomplete or too messy. The encode function below is actually part of a snippet I found googling around. Happy uploading.

import httplib
import urlparse

def upload(url,filename):
    def encode (file_path, fields=[]):
        BOUNDARY = '----------bundary------'
        CRLF = '\r\n'
        body = []
        # Add the metadata about the upload first
        for key, value in fields:
            body.extend(
              ['--' + BOUNDARY,
               'Content-Disposition: form-data; name="%s"' % key,
               '',
               value,
               ])
        # Now add the file itself
        file_name = os.path.basename(file_path)
        f = open(file_path, 'rb')
        file_content = f.read()
        f.close()
        body.extend(
          ['--' + BOUNDARY,
           'Content-Disposition: form-data; name="file"; filename="%s"'
           % file_name,
           # The upload server determines the mime-type, no need to set it.
           'Content-Type: application/octet-stream',
           '',
           file_content,
           ])
        # Finalize the form body
        body.extend(['--' + BOUNDARY + '--', ''])
        return 'multipart/form-data; boundary=%s' % BOUNDARY, CRLF.join(body)

    if os.path.exists(filename):
        content_type, body = encode(filename)
        headers = { 'Content-Type': content_type }
        u = urlparse.urlparse(url)
        server = httplib.HTTPConnection(u.netloc)
        server.request('POST', u.path, body, headers)
        resp = server.getresponse()
        server.close()

        if resp.status == 201:
            location = resp.getheader('Location', None)
        else :
            print resp.status, resp.reason
            location = None

        return location

Since I’m working with Django, this is the server part. Few remarks: I create the file name using uuid1(). This is an easy way to create unique identifier. A bit over killing maybe. I assume a model myfiles and a form UploadFileForm that you can easily guess. the function handle_uploaded_file is the procedure that actually saves the file on the disk. This is standard. I return a “Location” where the user can access the file. You have to create a small view to serve the file.

import uuid
from django.http import HttpResponse
import os
import datetime
from myapp.models import myfiles
from myapp.forms import UploadFileForm

def handle_uploaded_file(f,n):
    destination = open(n, 'wb+')
    for chunk in f.chunks():
        destination.write(chunk)
    destination.close()

def upload(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid():
            ip = request.META['REMOTE_ADDR']
            u = str(uuid.uuid1())
            uploaded = datetime.datetime.now()
            fname = os.path.join(baseupdir, u)
            handle_uploaded_file(request.FILES['file'],fname)
            size = os.path.getsize(fname)

            d = myfiles(fname=fname,size=size,uploaded=uploaded,ip=ip,uuid=u).save()

            response = HttpResponse(content="", status=201)
            response["Location"] = "/file?uuid=%s" % u
            return response # 10.2.2 201 Created
        else :
            return HttpResponse(status=400) # 10.4.1 400 Bad Request
    else :
        return HttpResponse(status=400) # 10.4.1 400 Bad Request

add sqlite3 collation with python 2.5 and django

a while ago I wrote about enabling the sqlite3 extension with storm . This is how you do it with the Django ORM. The collation is the same and all details are in the old post. The only tricky part is to establish the connection with cursor = connection.cursor() before calling the function to enable the extension. Failing to do so, will result in an error as the connection object will be null.

    def add_collation():
        from django.db import connection
        import sqlitext
        cursor = connection.cursor()
        sqlitext.enable_extension(connection.connection,1)
        cursor.execute("SELECT load_extension('sqlite/libcollate_debian.so')")

dynamic forms with django

Today I started learning how to write web forms in django. My quest was to write a simple search form where I could specify multiple criteria . Django is a very nice and flexible framework written in python and it is also reasonably well documented.

I don’t feel writing much today. this is the code :

The tricky part was to understand how to re-display the view and to add a new field. This is easily accomplished in django using the formset class that allows to put display together more then one form. In this case the logic is simple. First we display an empty form with two submit buttons, Add and Search. Search brings the obvious result. Add gets the data that was submitted, validates it and re-display the form with an additional field. Note also that the default validation functions are used transparently to validate the input of each sub-form.

views.py

from django.http import HttpResponse
from django.shortcuts import render_to_response
from cudfdep.depo.forms import SearchField
from django.forms.formsets import formset_factory
from django.http import Http404

def search(request):
    SearchFormSet = formset_factory(SearchField)
    data = {'formset': SearchFormSet(), 'debug' : 'default'}
    if request.method == 'POST':
        formset = SearchFormSet(request.POST)
        if formset.is_valid() :
            if request.POST.get('form-submit') == 'Add':
                data = {'formset': SearchFormSet(initial=formset.cleaned_data), 'debug' : 'add'}
            elif (request.POST.get('form-submit') == 'Search' ):
                data = {'formset': SearchFormSet(), 'debug' : 'result' }
            else :
                raise Http404('Invalid request')
        else :
            data = {'formset': formset, 'debug' : 'not valid'}

    return render_to_response('search_form.html', data)

The forms file describes the form logic to be displayed.

forms.py

from django import forms

class SearchField(forms.Form):
    date = forms.DateTimeField(label='date', widget=forms.DateTimeInput)
    arch = forms.ChoiceField([(0,'i386'),(1,'amd64')], widget=forms.Select,initial=0)
    release = forms.ChoiceField([(0,'etch'),(1,'lenny'),(2,'sid'),(3,'squeeze')],widget=forms.Select,initial=0)

This is the template :

search_form.html

<html>
<head>
    <title>Search Form</title>
</head>
<body>
    <form action="" method="post">
      {{ formset.management_form }}
      <table>
          {% for form in formset.forms %}
          {{ form.as_table }}
          {% endfor %}
      </table>
      <input name='form-submit' type="submit" value="Add">
      <input name='form-submit' type="submit" value="Search">
    </form>
    {% if debug %}
        <p style="color: red;">{{ debug }}</p>
    {% endif %}
</body>
</html>