Setting PYTHON_EGG_CACHE when deploying Python apps using FastCGI

I recently sorted out an issue with the IMAPClient Trac instance that's been bugging me for a while.

The problem was that whenever the web server logs were rotated logrotate would restart Lighttpd. The web server restart would in turn restart the Trac (FastCGI) processes. Unfortunately, the Trac processes would fail to start with the following error.

pkg_resources.ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/root/.python-eggs'

The Python egg cache directory is currently set to:

  /root/.python-eggs

Bang, no IMAPClient web site (the rest of the site was ok). To band-aid the problem when it happened (and I noticed!) I issue a sudo /etc/init.d/lighttpd restart and everything would be fine again.

After some investigation I found that running /etc/init.d/lighttpd restart as root always triggered the problem where-as restarting using sudo always worked. My guess is that restarting when logged in as root was leaving $HOME at /root even after Lighttpd had dropped to its unprivileged user account. The unprivileged user isn't allowed to write to /root so Trac blows up. setuptools seems to use $HOME instead of looking up the actual home directory of the current user.

The fix for me was to set the PYTHON_EGG_CACHE environment variable for the FastCGI processes to somewhere they are allowed to write to. This is done with the bin-environment option if you're using Lighttpd like me.

I imagine similar problems can happen with any Python app deployed using FastCGI.

posted: Thu, 19 Nov 2009 17:48 | permalink | comments

Why I chose Disqus Comments

I recently moved my blog comments away the PyBlosxom comments plugin to a hosted system. The main driver was the ability for people to subscribe to comments for an article using email or RSS. It's a pain for people to have to check back to the site to see if someone has replied to their comments. I was also keen on user-experience-enhancing features such as integration with external systems like OpenID, Twitter and Yahoo.

My criteria were:

There are a number of hosted comment systems out there. The most popular options seem to be Disqus, JS-Kit Echo and IntenseDebate.

IntenseDebate was eliminated first because it doesn't seem to provide an import mechanism for custom websites. Import only seems to be supported for well known blog platforms such as Wordpress. There's no comment API either. The approach seems to be to leave your old comment system in place and just have new comments go into IntenseDebate. Not good enough, I wanted to completely replace the existing comments system.

After some deliberation I decided on JS-Kit Echo for one tiny reason: it supports the <pre> tag. The closest Disqus supported was the <code> tag which doesn't preserve white-space (useless for Python code samples).

So I paid my US$12 (it's the only service that doesn't have a free option) and started looking at how to import my existing comments using their API ... and quickly found that it sucks. Comments can be submitted but you can't specify a timestamp so they are dated with the import date. Far from ideal. Then there's the API for retrieving comments: it returns the data as JavaScript code (no not JSON)! It's pretty clear that the API is what they use with the JavaScript for Echo itself and geared for that use only. They've just thrown it out there and documented it, warts and all.

Back to the drawing board.

The only showstopper for Disqus was the lack of <pre>. Everything else about it was great: it met all my requirements and the API was clean and comprehensive. If only there was a way to have properly formatted source code in the comments.

Light bulb moment: use a CSS hack to make <code> in comments behave like <pre>. The trick is to turn code into a block element and change how white-space is handled. The CSS snippet looks like:

.dsq-comment-message code {
  display:block;
  white-space:pre;
}

Works great.

With the only blocker gone, I wrote a Python script with the help of Ian Lewis' excellent disqus-python-client package to pull in the existing comments from the old system. Within an hour or so it was ready to go.

Hopefully this article saves someone else some time if they decide to use one of these systems. Getting things running chewed up a lot more time then I had expected.

posted: Thu, 05 Nov 2009 19:49 | permalink | comments

Problem installing Trac under Lighttpd+FastCGI

I just finished configuring a Trac instance for IMAPClient. To keep page response times tolerable I'm using FastCGI.

It turns out there's a little gotcha when you serve an app at the root (ie. "/") using FastCGI with Lighttpd. It passes the wrong SCRIPT_NAME and PATH_INFO to the app causing unexpected behaviour. The problem isn't there if the app is served from a prefix (eg. "/trac").

The problem is apparently fixed somewhere in Lighttpd 1.5 but I'm on 1.4.13 as shipped with Debian.

With Trac there is a workaround. If you modify Trac's FastCGI runner (fcgi_frontend.py) to correct the SCRIPT_NAME and PATH_INFO environment variables before the rest of the app sees them, the problem is solved. Here's what my fcgi_frontend.py now looks like (header comments removed):

import pkg_resources

from trac import __version__ as VERSION
from trac.web.main import dispatch_request

from os import environ
import _fcgi

def application(environ, start_request):
    environ['PATH_INFO'] = environ['SCRIPT_NAME'] + environ['PATH_INFO']
    environ['SCRIPT_NAME'] = ''
    return dispatch_request(environ, start_request)

def run():
    _fcgi.WSGIServer(application).run()

if __name__ == '__main__':
    pkg_resources.require('Trac==%s' % VERSION)
    run()

Figuring this out cost me lots of time. Hopefully this information helps someone else out.

Relevant links:

posted: Fri, 28 Aug 2009 01:46 | permalink | comments