Why I chose Disqus Comments

I recently moved my blog comments away the PyBlosxom comments plugin to a hosted system. The main driver was the ability for people to subscribe to comments for an article using email or RSS. It's a pain for people to have to check back to the site to see if someone has replied to their comments. I was also keen on user-experience-enhancing features such as integration with external systems like OpenID, Twitter and Yahoo.

My criteria were:

  • email subscription, RSS a bonus
  • support for pre-formatted text sections in comments (essential for code samples)
  • an import mechanism for existing comments
  • threading of comments to allow commenters to respond to each other sensibly
  • clean look with some ability to customise
  • support for a variety of authentication/profile systems

There are a number of hosted comment systems out there. The most popular options seem to be Disqus, JS-Kit Echo and IntenseDebate.

IntenseDebate was eliminated first because it doesn't seem to provide an import mechanism for custom websites. Import only seems to be supported for well known blog platforms such as Wordpress. There's no comment API either. The approach seems to be to leave your old comment system in place and just have new comments go into IntenseDebate. Not good enough, I wanted to completely replace the existing comments system.

After some deliberation I decided on JS-Kit Echo for one tiny reason: it supports the <pre> tag. The closest Disqus supported was the <code> tag which doesn't preserve white-space (useless for Python code samples).

So I paid my US$12 (it's the only service that doesn't have a free option) and started looking at how to import my existing comments using their API ... and quickly found that it sucks. Comments can be submitted but you can't specify a timestamp so they are dated with the import date. Far from ideal. Then there's the API for retrieving comments: it returns the data as JavaScript code (no not JSON)! It's pretty clear that the API is what they use with the JavaScript for Echo itself and geared for that use only. They've just thrown it out there and documented it, warts and all.

Back to the drawing board.

The only showstopper for Disqus was the lack of <pre>. Everything else about it was great: it met all my requirements and the API was clean and comprehensive. If only there was a way to have properly formatted source code in the comments.

Light bulb moment: use a CSS hack to make <code> in comments behave like <pre>. The trick is to turn code into a block element and change how white-space is handled. The CSS snippet looks like:

.dsq-comment-message code {

Works great.

With the only blocker gone, I wrote a Python script with the help of Ian Lewis' excellent disqus-python-client package to pull in the existing comments from the old system. Within an hour or so it was ready to go.

Hopefully this article saves someone else some time if they decide to use one of these systems. Getting things running chewed up a lot more time then I had expected.

IMAPClient has a new home

I've just (finally) finished setting up a proper website for IMAPClient. The new home for the project is http://imapclient.freshfoo.com/.

It's a Trac instance with Mercurial support that monitors the main trunk repository. All items from the TODO file in the source have been converted to tickets in the bug tracker. I've even created a hokey little logo.

Let me know me know if anything looks broken.

Time to work on some long-standing bugs...

Problem installing Trac under Lighttpd+FastCGI

I just finished configuring a Trac instance for IMAPClient. To keep page response times tolerable I'm using FastCGI.

It turns out there's a little gotcha when you serve an app at the root (ie. "/") using FastCGI with Lighttpd. It passes the wrong SCRIPT_NAME and PATH_INFO to the app causing unexpected behaviour. The problem isn't there if the app is served from a prefix (eg. "/trac").

The problem is apparently fixed somewhere in Lighttpd 1.5 but I'm on 1.4.13 as shipped with Debian.

With Trac there is a workaround. If you modify Trac's FastCGI runner (fcgi_frontend.py) to correct the SCRIPT_NAME and PATH_INFO environment variables before the rest of the app sees them, the problem is solved. Here's what my fcgi_frontend.py now looks like (header comments removed):

import pkg_resources

from trac import __version__ as VERSION
from trac.web.main import dispatch_request

from os import environ
import _fcgi

def application(environ, start_request):
    environ['PATH_INFO'] = environ['SCRIPT_NAME'] + environ['PATH_INFO']
    environ['SCRIPT_NAME'] = ''
    return dispatch_request(environ, start_request)

def run():

if __name__ == '__main__':
    pkg_resources.require('Trac==%s' % VERSION)

Figuring this out cost me lots of time. Hopefully this information helps someone else out.

Relevant links:

Dear Bazaar...

Our relationship has only just started but it's already time for me to move on. Although I think you're great, Mecurial has become my new best friend. Things with her are just a little easier. We just clicked.

I get along so well with her friends; I never did quite fit in with yours. They always seemed somewhat immature.

It's also hard to ignore recent developments. It seems like Mercurial is going to become quite important in the future.

No hard feelings I hope. I truly wish you all the best for the future.


Banishing trailing whitespace with Emacs

One of my pet peeves, especially with Python code, is trailing whitespace. It serves no purpose, introduces noise in diffs and wastes valuable bytes dammit (yes I'm being pedantic).

In order to see make trailing whitespace visible in Emacs you can use the show-trailing-whitespace variable. Emacs of course has a command to remove trailing whitespace: delete-trailing-whitespace.

Better yet, to get rid of trailing whitespace automatically on save you can add a function to the write-contents hook. The following snippet will cause trailing whitespace to be removed on save, but just for Python files.

;; Automatically remove trailing whitespace when saving Python file
(add-hook 'python-mode-hook
          (lambda () (add-hook 'write-contents-hooks 'delete-trailing-whitespace t)))

The one problem with doing this is that when changing an existing codebase your commits could end up with many whitespace deltas, making it difficult to see the meat of your changes. Use with care.

draftsdir plugin

I've just published a simple little PyBlosxom plugin called draftsdir. It solves a personal itch for me: I'd like to review draft blog entries on the site before they go live. The plugin lets you define a directory for draft entries which aren't shown unless you add a query arg to the URL (the default arg name is "drafts" but there's an option to change it). When you're happy with an entry you move it from the drafts directory to where your other published entries are. Simple.

For more details see the Code section of this site, the bzr repo or download it directly. There install and configuration instructions at the top of the file.

Emacs with Viper mode: editor nirvana?

I've been a long time Vim user and fan. Once you're used proficient with the Vi way of editing it's difficult go back to anything else. Other editors just feel inefficient and clunky.

That said, I've been jealous of certain Emacs features that I've seen while looking over the shoulders of my colleagues. Multiple frames (GUI windows), the clean extension system (via elisp), the tight process integration (shell, SQL, debuggers etc) and all sorts of unexpected bells and whistles; these goodies slowly ate away at my curiosity.

So 2 weeks ago I caved and decided to give Emacs another go. I hadn't really used it since university. Starting with my colleague's configuration and making some minor tweaks, I began using Emacs for serious work.

A few days in and I was enjoying exploring my new toy but it didn't feel quite right. Although I had a reasonable grasp of Emacs' editing keys and commands, most tasks took way too long, requiring convoluted hand gymnastics. My left pinky was permanently sore from constantly reaching for the Ctrl and Alt keys. I was missing those surgical, efficient Vi commands.

At the suggestion of one Emacs-using colleague I gave Viper (viper-mode) a try. It's an emulation that attempts to provide a fairly accurate Vi experience while still allowing full access to the rest of Emacs. To be honest I was expecting it to be a half-assed kludge. I was wrong. Viper is a mature Emacs extension and it does a great job of mitigating conflicts between the Emacs and Vi ways of doing things.

Viper mode proved to be the tipping point; because of it I'm sticking with Emacs. As far as I'm concerned it's the best of both worlds.

For anyone who's interested, my Emacs config is available in the repository browser here or via bzr. This is my personal configuration branch so it will update as I make changes to the setup. Note that I'm using the latest development (but rock solid) Emacs so there might be references to features in the config which only exist in this version.

Some items of note in my config:

Viper mode only emulates classic Vi. vimpulse provides a bunch of extra features which a Vim user will probably miss such as various "g" commands and visual select mode.
This is a standard feature of Emacs which isn't bound to keys by default. It gives amazing power by replacing the standard find-file and switch-buffer keystrokes with beefed up alternatives. The key features are quick directory switching and fuzzy, recursive name matching (but that's not all).
I've replaced the standard buffer list binding (C-x C-b) with ibuffer. This gives a more powerful and easier to use version of the standard buffer list and allows for crazy batch manipulation of buffers.
Mode specific template expansion. Powerful and super useful for cranking out commonly used sections of text (programming is full of them).
flymake - pyflakes integration
Flymake runs arbitrary programs over buffers on-the-fly. For Python files flymake has been configured to run pyflakes and highlight errors in code as I type. I might change this to use pylint at some stage because pylint finds a wider range of problems.

Some useful Emacs config hacking links:

bzr repositories now available

I've just made my personal bzr repositories publically available so that anyone can easily get to them (including me!) and so I can refer to things from blog articles. The repos are available for branching using bzr under http://freshfoo.com/repos/ and in human browseable form. See also the links in the left sidebar and in the code section of the site.

I'm using Loggerhead to provide the web viewable form (proxied via the main lighttpd server). It was very easy to setup (using serve-branches). I just wrote a simple init.d script to ensure it stays running.

Announcing IMAPClient 0.5

It's been a long time between releases but I've just released IMAPClient 0.5.

From the NEWS file:

Folder names are now encoded and decoded transparently if required (using modified UTF-7). This means that any methods that return folder names may return unicode objects as well as normal strings [API CHANGE]. Additionally, any method that takes a folder name now accepts unicode object too. Use the folder_encode attribute to control whether encode/decoding is performed.

Unquoted folder names in server responses are now handled correctly. Thanks to Neil Martinsen-Burrell for reporting this bug.

Fixed a bug with handling of unusual characters in folder names.

Timezones are now handled correctly for datetimes passed as input and for server responses. This fixes a number of bugs with timezones. Returned datetimes are always in the client's local timezone [API CHANGE].

Many more unit tests, some using Michael Foord's excellent mock.py. (http://www.voidspace.org.uk/python/mock/)

IMAPClient can be installed from PyPI using easy_install IMAPClient or downloaded from my Code page. As always, feedback and patches are most welcome.

Initial thoughts for PyCon 2009

I'm at PyCon 2009 in Chicago. So much awesome stuff has already been discussed and we're only a few hours in. Much of it very relevant for my employer (which is nice since they're paying).

Some random thoughts so far:

  • Windmill (the web testing framework) looks incredibly promising. If you're doing web apps and you're a Python shop you should be using this. The Python shell integration rocks.
  • Cassandra is worth exploring. It's a large-scale, distributed DB that's used (and hacked on) by Facebook with good Python bindings.
  • Twitter has added a facinating new dimension to the conference. Everyone is tweeting using the #pycon tag. There's all sorts of interesting discussion going on in real-time as talks happen. Twitterfall is a great way to watch the action.

The conference seems really well run and the space at the hotel is great. The only problem I (and many others) have noticed so far is that the wifi can be a tad unreliable. Not suprising given the huge number of wireless devices around.