rst_break plugin for PyBlosxom

I just scratched an itch by writing a small plugin for PyBlosxom that allows the rst (reStructured Text) and readmore plugins to work together [1]. It defines a reST "break" directive which gets transformed into the breakpoint string the readmore plugin looks out for. This allows for "Read more..." breaks to be inserted in for reST based articles.

For further information see the Code page here and at the top of the plugin itself.

[1]Yes, the audience for this plugin is probably tiny!

posted: Sat, 30 Jan 2010 15:29 | permalink | comments

Setting PYTHON_EGG_CACHE when deploying Python apps using FastCGI

I recently sorted out an issue with the IMAPClient Trac instance that's been bugging me for a while.

The problem was that whenever the web server logs were rotated logrotate would restart Lighttpd. The web server restart would in turn restart the Trac (FastCGI) processes. Unfortunately, the Trac processes would fail to start with the following error.

pkg_resources.ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg

  [Errno 13] Permission denied: '/root/.python-eggs'

The Python egg cache directory is currently set to:


Bang, no IMAPClient web site (the rest of the site was ok). To band-aid the problem when it happened (and I noticed!) I issue a sudo /etc/init.d/lighttpd restart and everything would be fine again.

After some investigation I found that running /etc/init.d/lighttpd restart as root always triggered the problem where-as restarting using sudo always worked. My guess is that restarting when logged in as root was leaving $HOME at /root even after Lighttpd had dropped to its unprivileged user account. The unprivileged user isn't allowed to write to /root so Trac blows up. setuptools seems to use $HOME instead of looking up the actual home directory of the current user.

The fix for me was to set the PYTHON_EGG_CACHE environment variable for the FastCGI processes to somewhere they are allowed to write to. This is done with the bin-environment option if you're using Lighttpd like me.

I imagine similar problems can happen with any Python app deployed using FastCGI.

posted: Thu, 19 Nov 2009 17:48 | permalink | comments

Why I chose Disqus Comments

I recently moved my blog comments away the PyBlosxom comments plugin to a hosted system. The main driver was the ability for people to subscribe to comments for an article using email or RSS. It's a pain for people to have to check back to the site to see if someone has replied to their comments. I was also keen on user-experience-enhancing features such as integration with external systems like OpenID, Twitter and Yahoo.

My criteria were:

There are a number of hosted comment systems out there. The most popular options seem to be Disqus, JS-Kit Echo and IntenseDebate.

IntenseDebate was eliminated first because it doesn't seem to provide an import mechanism for custom websites. Import only seems to be supported for well known blog platforms such as Wordpress. There's no comment API either. The approach seems to be to leave your old comment system in place and just have new comments go into IntenseDebate. Not good enough, I wanted to completely replace the existing comments system.

After some deliberation I decided on JS-Kit Echo for one tiny reason: it supports the <pre> tag. The closest Disqus supported was the <code> tag which doesn't preserve white-space (useless for Python code samples).

So I paid my US$12 (it's the only service that doesn't have a free option) and started looking at how to import my existing comments using their API ... and quickly found that it sucks. Comments can be submitted but you can't specify a timestamp so they are dated with the import date. Far from ideal. Then there's the API for retrieving comments: it returns the data as JavaScript code (no not JSON)! It's pretty clear that the API is what they use with the JavaScript for Echo itself and geared for that use only. They've just thrown it out there and documented it, warts and all.

Back to the drawing board.

The only showstopper for Disqus was the lack of <pre>. Everything else about it was great: it met all my requirements and the API was clean and comprehensive. If only there was a way to have properly formatted source code in the comments.

Light bulb moment: use a CSS hack to make <code> in comments behave like <pre>. The trick is to turn code into a block element and change how white-space is handled. The CSS snippet looks like:

.dsq-comment-message code {

Works great.

With the only blocker gone, I wrote a Python script with the help of Ian Lewis' excellent disqus-python-client package to pull in the existing comments from the old system. Within an hour or so it was ready to go.

Hopefully this article saves someone else some time if they decide to use one of these systems. Getting things running chewed up a lot more time then I had expected.

posted: Thu, 05 Nov 2009 19:49 | permalink | comments

IMAPClient has a new home

I've just (finally) finished setting up a proper website for IMAPClient. The new home for the project is

It's a Trac instance with Mercurial support that monitors the main trunk repository. All items from the TODO file in the source have been converted to tickets in the bug tracker. I've even created a hokey little logo.

Let me know me know if anything looks broken.

Time to work on some long-standing bugs...

posted: Sun, 18 Oct 2009 17:12 | permalink | comments

Problem installing Trac under Lighttpd+FastCGI

I just finished configuring a Trac instance for IMAPClient. To keep page response times tolerable I'm using FastCGI.

It turns out there's a little gotcha when you serve an app at the root (ie. "/") using FastCGI with Lighttpd. It passes the wrong SCRIPT_NAME and PATH_INFO to the app causing unexpected behaviour. The problem isn't there if the app is served from a prefix (eg. "/trac").

The problem is apparently fixed somewhere in Lighttpd 1.5 but I'm on 1.4.13 as shipped with Debian.

With Trac there is a workaround. If you modify Trac's FastCGI runner ( to correct the SCRIPT_NAME and PATH_INFO environment variables before the rest of the app sees them, the problem is solved. Here's what my now looks like (header comments removed):

import pkg_resources

from trac import __version__ as VERSION
from trac.web.main import dispatch_request

from os import environ
import _fcgi

def application(environ, start_request):
    environ['PATH_INFO'] = environ['SCRIPT_NAME'] + environ['PATH_INFO']
    environ['SCRIPT_NAME'] = ''
    return dispatch_request(environ, start_request)

def run():

if __name__ == '__main__':
    pkg_resources.require('Trac==%s' % VERSION)

Figuring this out cost me lots of time. Hopefully this information helps someone else out.

Relevant links:

posted: Fri, 28 Aug 2009 01:46 | permalink | comments

bzr repositories now available

I've just made my personal bzr repositories publically available so that anyone can easily get to them (including me!) and so I can refer to things from blog articles. The repos are available for branching using bzr under and in human browseable form. See also the links in the left sidebar and in the code section of the site.

I'm using Loggerhead to provide the web viewable form (proxied via the main lighttpd server). It was very easy to setup (using serve-branches). I just wrote a simple init.d script to ensure it stays running.

posted: Sun, 21 Jun 2009 21:41 | permalink | comments now using rest2web

I've now converted 2 popular articles [1] on so that they are generated via Michael Foord's excellent rest2web. Previously one of these articles was in plain HTML and the other was on the site wiki. I've got a few other articles in the works so I decided it was time to migrate the articles to a clean, consistent format. reStructured Text fits the bill perfectly.

I recently also started using the rst entryparser for PyBlosxom so now most of the site is generated using reStructured Text.

[1]according to the HTTP server logs

posted: Wed, 24 Sep 2008 20:52 | permalink | comments