Go for Pythonistas

I gave a talk last night at the monthly Christchurch Python meetup titled "Go For Pythonistas". It was an introduction to the Go programming language from the perspective of a Python developer. The talk went well, with plenty of questions and comments throughout. Thanks to all who attended for your interest!

I'm not sure how useful they'll be on their own but the slides for the talk are available here.

I mentioned Juju in the talk a few times which piqued a few people's interest so I also ended up doing a short impromptu talk about that. Juju is the amazing cloud orchestration software that I hack on at Canonical these days (written in Go). It's worth a few blog articles in itself.

Tags: ,
posted: Thu, 07 Aug 2014 22:58 | permalink | comments

IMAPClient 0.11

This is a somewhat belated announcement that IMAPClient 0.11 is out (and has been for a little over a week now). Notable changes:

Big thanks go out to John Louis del Rosario, Naveen Nathan, Brandon Rhodes and Benjamin Morrise for their contributions towards this release.

See the NEWS.rst file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Sun, 18 May 2014 11:43 | permalink | comments

IMAPClient 0.10

IMAPClient 0.10 has just been released. This is an important release because it's the first to support Python 3!

Here's the highlights:

A massive thank you to Mathieu Agopian for his massive contribution to getting the Python 3 support finished. His changes and ideas feature heavily in this release.

See the NEWS.rst file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Thu, 06 Jun 2013 00:15 | permalink | comments

Raspberry Pi: driving a VU meter using a digital-to-analog converter

As I've mentioned in previous blog articles, my wife and I have been working on driving an analog VU meter based on the sound going out the Raspberry Pi's audio outputs. This now works!

Here's a video demonstrating the result:

The music [1] is playing from a Raspberry Pi, with software running on the Pi digitally sampling the peak output audio level and writing that out to an 8-bit digital-to-analog converter (DAC). The DAC output is then used to drive the analog meter. If you're interesting in knowing how all this hangs together, keep reading.

Read more...

posted: Mon, 06 May 2013 23:40 | permalink | comments

IMAPClient 0.9.2

IMAPClient 0.9.2 was released yesterday. In this release:

See the NEWS file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

Note that the official project source repository is now on Bitbucket. http://imapclient.freshfoo.com/ is still the offical home page and is still used for project tracking. It is only the source respository that has moved.

posted: Thu, 28 Mar 2013 23:35 | permalink | comments

Monitoring Audio Levels with PulseAudio

I'm working on driving an analog VU meter from my Raspberry Pi using whatever audio is going out the Pi's sound outputs. The de facto Linux sound system, PulseAudio, allows any sound output (or "sink" in PulseAudio's nonclementure) to be monitored. In PulseAudio land, each sink has a corresponding "source" called the monitor source which can be read just like any other other PulseAudio input such as a microphone. In fact, to help with volume meter style applications, PulseAudio even allows you to ask for peak level measurements, which means you can sample the monitor sink at a low frequency, with low CPU utilisation, but still produce a useful volume display. When this feature is used, each sample read indicates the peak level since the last sample.

Read more...

posted: Sun, 10 Feb 2013 22:05 | permalink | comments

IMAPClient 0.9.1

IMAPClient 0.9.1 is out! In this release:

See the NEWS file and manual for more details.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Mon, 12 Nov 2012 22:52 | permalink | comments

Somewhat over-engineered blinking LEDs

My Raspberry Pi arrived a couple of weeks ago and I've been working on turning it into a mini-audio server to connect to the home stereo in the living room.

As part of the project I'd like to drive an analog VU meter from the sound signal.

This week my (enthusiastic!) wife and I played around with attaching some basic electronics to the GPIO ports so that we could get more comfortable the Raspberry Pi. Our electronics knowledge is more than a little rusty but surprisingly everything we tried worked first go.

Read more...

posted: Fri, 20 Jul 2012 13:52 | permalink | comments

IMAPClient 0.9 released

I'm pleased to announce version 0.9 of IMAPClient, the easy-to-use and Pythonic IMAP client library.

Highlights for this release:

The NEWS file and manual have more details on all of the above.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

The main focus of the next release (0.10) will be Python 3 support as this is easily the most requested feature. Watch this space for more news on this.

posted: Wed, 16 May 2012 16:02 | permalink | comments

IMAPClient 0.8.1 released

Version 0.8.1 of IMAPClient has just been released. This version works around a subtle bug in distutils which was preventing installation on Windows from working. Thanks to Bob Yexley for the bug report.

This release also contains a few small documentation updates and packaging fixes. The NEWS file has more details.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Wed, 11 Jan 2012 23:20 | permalink | comments

IMAPClient 0.8 released

Version 0.8 of IMAPClient is out! Although I didn't get everything into this release that I had hoped to, there's still plenty there. Thanks to Johannes Heckel and Andrew Scheller for their contributions to this release.

Highlights for 0.8:

The NEWS file and main documentation has more details on all of the above.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Wed, 09 Nov 2011 22:10 | permalink | comments

IMAPClient 0.7 released

The next version of IMAPClient was quietly released on the weekend. I've been pretty busy so I'm just getting to telling the world about it now.

This release is earlier then planned because IMAPClient is featured in the IMAP chapter of Brandon Rhodes' new edition of the book Foundations of Python Network Programming. Brandon had made several bug reports and feature requests while working on the book and some of the examples in the book relied on unreleased changes to IMAPClient. IMAPClient 0.7 works with book's examples.

What this does mean is that IMAPClient 0.7 doesn't have Python 3 support. I have been making headway with this however and with a little luck and a little more time, 0.8 should be Python 3 compatible.

Highlights for 0.7:

The NEWS document has more details on all the above.

Proper documentation is also on its way. I've been slowly making headway with Sphinx based documentation.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Wed, 09 Feb 2011 15:15 | permalink | comments

IMAPClient 0.6.2 is out

This release fixes some parsing issues when dealing with FETCH items that include square brackets (eg. "BODY[HEADER.FIELDS (From Subject)]") and includes the example when the package is installed using PyPI.

Also, IMAPClient now uses the excellent Distribute instead of setuptools.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

In the pipeline now is better (Sphinx based) documentation, cleaning up the tests, OAUTH support and Python 3 support.

posted: Fri, 17 Sep 2010 13:31 | permalink | comments

IMAPClient 0.6.1 released

I've just released IMAPClient 0.6.1.

The only functional change in the release is that it now automatically patches imaplib's IMAP4_SSL class to fix Python Issue 5949. This is a bug that's been fixed in later Python 2.6 versions and 2.7 but still exists in Python versions that are in common use. Without fix this you may experience hangs when using SSL.

The patch is only applied if the running Python version is known to be one of the affected versions. It is applied when IMAPClient is imported.

The only other change in this release is that I've now marked IMAPClient as "production ready" on PyPI and have updated the README to match. This was prompted by a request to clarify the current status of the project and seeing that all current functionality is solid and, I don't plan to change the existing APIs in backwards-incompatible ways, I've decided to indicate the project as suitable for production use.

As always, IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site. Feedback, bug reports and patches are most welcome.

posted: Thu, 02 Sep 2010 22:56 | permalink | comments

IMAPClient 0.6 released

IMAPClient 0.6 is finally out! Highlights:

Be aware that there have been several API changes with this release. See the NEWS file for further details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site. As always, feedback, bug reports and patches are most welcome.

Special thanks goes to Mark Hammond. He has contributed a significant amount of code and effort to this release. Incidentally, Mark is using IMAPClient as part of the backend for the Raindrop Mozilla Labs project.

posted: Tue, 11 May 2010 13:45 | permalink | comments

TypeError: object.__init__() takes no parameters

At my employer we are in the process of migrating from Python 2.4 to 2.6. When running some existing code under Python 2.6 we started getting DeprecationWarnings about "object.__new__() takes no parameters" and "object.__init__() takes no parameters".

A simple example that triggers the warning:

class MyClass(object):

    def __new__(cls, a, b):
        print 'MyClass.__new__', a, b
        return super(MyClass, cls).__new__(cls, a, b)

    def __init__(self, a, b):
        print 'MyClass.__init__', a, b
        super(MyClass, self).__init__(a, b)

obj = MyClass(6, 7)

This gives:

$ python2.4 simple-warning.py
MyClass.__new__ 6 7
MyClass.__init__ 6 7

$ python2.6 simple-warning.py
MyClass.__new__ 6 7
simple-warning.py:5: DeprecationWarning: object.__new__() takes no parameters
  return super(MyClass, cls).__new__(cls, a, b)
MyClass.__init__ 6 7
simple-warning.py:9: DeprecationWarning: object.__init__() takes no parameters
  super(MyClass, self).__init__(a, b)

It turns out that a change to Python for 2.6 (and 3) means that object.__new__ and object.__init__ no longer take arguments - a TypeError is raised when arguments are passed. To avoid breaking too much pre-existing code, there is a special case that will cause a DeprecationWarning instead of TypeError if both __init__ and __new__ are overridden. This is the case we were running into with our code at work.

The reason for this change seems to make enough sense: object doesn't do anything with arguments to __init__ and __new__ so it shouldn't accept them. Raising an error when arguments are passed highlights code where the code might be doing the wrong thing.

Unfortunately this change also breaks Python's multiple inheritance in a fairly serious way when cooperative super calls are used. Looking at the ticket for this change, this issue was thought about but perhaps the implications were not fully understood. Given that using super with multiple inheritance is common and "correct" practice, it would seem that this change to Python is a step backwards.

Read more...

Tags:
posted: Sat, 30 Jan 2010 19:07 | permalink | comments

rst_break plugin for PyBlosxom

I just scratched an itch by writing a small plugin for PyBlosxom that allows the rst (reStructured Text) and readmore plugins to work together [1]. It defines a reST "break" directive which gets transformed into the breakpoint string the readmore plugin looks out for. This allows for "Read more..." breaks to be inserted in for reST based articles.

For further information see the Code page here and at the top of the plugin itself.

[1]Yes, the audience for this plugin is probably tiny!

posted: Sat, 30 Jan 2010 15:29 | permalink | comments

IMAPClient 0.5.2

IMAPClient 0.5.2 has just been released. This release fixes 2 bugs (#28 and #33). Much thanks to Fergal Daly and Mark Eichin for reporting these bugs.

Install from the tarball or zip or upgrade using easy_install or pip.

posted: Sun, 24 Jan 2010 17:58 | permalink | comments

IMAPClient 0.5.1

I've just made a quick point release of IMAPClient. Mark Hammond is interested in using it for a project he's working on but the licenses (GPL and MPL) were incompatible. I was thinking about relaxing the license of IMAPClient anyway so this presented a good opportunity to make the switch.

Work on the 0.6 release is coming along. This version will fix a number issues with parsing of FETCH responses - the FETCH parsing code is being completely rewritten. This is the first time that IMAPClient will bypass most of imaplib for some functionality. It's looking like that at some point IMAPClient may not use imaplib at all.

IMAPClient can be installed from PyPI using pip install IMAPClient or easy_install IMAPClient. It can also be downloaded from the IMAPClient project page. As always, feedback and patches are most welcome.

posted: Tue, 15 Dec 2009 22:05 | permalink | comments

PyBlosxom to Disqus import script

I've had a few requests for the little hack I created to import comments from PyBlosxom into Disqus. A cleaned up version of disqus-import.py is now on the Code page. There's some docs at the top of the file.

posted: Wed, 25 Nov 2009 15:58 | permalink | comments

Setting PYTHON_EGG_CACHE when deploying Python apps using FastCGI

I recently sorted out an issue with the IMAPClient Trac instance that's been bugging me for a while.

The problem was that whenever the web server logs were rotated logrotate would restart Lighttpd. The web server restart would in turn restart the Trac (FastCGI) processes. Unfortunately, the Trac processes would fail to start with the following error.

pkg_resources.ExtractionError: Can't extract file(s) to egg cache

The following error occurred while trying to extract file(s) to the Python egg
cache:

  [Errno 13] Permission denied: '/root/.python-eggs'

The Python egg cache directory is currently set to:

  /root/.python-eggs

Bang, no IMAPClient web site (the rest of the site was ok). To band-aid the problem when it happened (and I noticed!) I issue a sudo /etc/init.d/lighttpd restart and everything would be fine again.

After some investigation I found that running /etc/init.d/lighttpd restart as root always triggered the problem where-as restarting using sudo always worked. My guess is that restarting when logged in as root was leaving $HOME at /root even after Lighttpd had dropped to its unprivileged user account. The unprivileged user isn't allowed to write to /root so Trac blows up. setuptools seems to use $HOME instead of looking up the actual home directory of the current user.

The fix for me was to set the PYTHON_EGG_CACHE environment variable for the FastCGI processes to somewhere they are allowed to write to. This is done with the bin-environment option if you're using Lighttpd like me.

I imagine similar problems can happen with any Python app deployed using FastCGI.

posted: Thu, 19 Nov 2009 17:48 | permalink | comments

Why I chose Disqus Comments

I recently moved my blog comments away the PyBlosxom comments plugin to a hosted system. The main driver was the ability for people to subscribe to comments for an article using email or RSS. It's a pain for people to have to check back to the site to see if someone has replied to their comments. I was also keen on user-experience-enhancing features such as integration with external systems like OpenID, Twitter and Yahoo.

My criteria were:

There are a number of hosted comment systems out there. The most popular options seem to be Disqus, JS-Kit Echo and IntenseDebate.

IntenseDebate was eliminated first because it doesn't seem to provide an import mechanism for custom websites. Import only seems to be supported for well known blog platforms such as Wordpress. There's no comment API either. The approach seems to be to leave your old comment system in place and just have new comments go into IntenseDebate. Not good enough, I wanted to completely replace the existing comments system.

After some deliberation I decided on JS-Kit Echo for one tiny reason: it supports the <pre> tag. The closest Disqus supported was the <code> tag which doesn't preserve white-space (useless for Python code samples).

So I paid my US$12 (it's the only service that doesn't have a free option) and started looking at how to import my existing comments using their API ... and quickly found that it sucks. Comments can be submitted but you can't specify a timestamp so they are dated with the import date. Far from ideal. Then there's the API for retrieving comments: it returns the data as JavaScript code (no not JSON)! It's pretty clear that the API is what they use with the JavaScript for Echo itself and geared for that use only. They've just thrown it out there and documented it, warts and all.

Back to the drawing board.

The only showstopper for Disqus was the lack of <pre>. Everything else about it was great: it met all my requirements and the API was clean and comprehensive. If only there was a way to have properly formatted source code in the comments.

Light bulb moment: use a CSS hack to make <code> in comments behave like <pre>. The trick is to turn code into a block element and change how white-space is handled. The CSS snippet looks like:

.dsq-comment-message code {
  display:block;
  white-space:pre;
}

Works great.

With the only blocker gone, I wrote a Python script with the help of Ian Lewis' excellent disqus-python-client package to pull in the existing comments from the old system. Within an hour or so it was ready to go.

Hopefully this article saves someone else some time if they decide to use one of these systems. Getting things running chewed up a lot more time then I had expected.

posted: Thu, 05 Nov 2009 19:49 | permalink | comments

IMAPClient has a new home

I've just (finally) finished setting up a proper website for IMAPClient. The new home for the project is http://imapclient.freshfoo.com/.

It's a Trac instance with Mercurial support that monitors the main trunk repository. All items from the TODO file in the source have been converted to tickets in the bug tracker. I've even created a hokey little logo.

Let me know me know if anything looks broken.

Time to work on some long-standing bugs...

posted: Sun, 18 Oct 2009 17:12 | permalink | comments

itertools.groupby

A relatively unknown part of the Python standard library that I find myself using fairly regularly at work these days is the groupby function in the itertools module. In a nutshell, groupby takes an iterator and breaks it up into sub-iterators based on changes in the "key" of the main iterator. This is of course done without reading the entire source iterator into memory.

The "key" is almost always based on some part of the items returned by the iterator. It is defined by a "key function", much like the sorted builtin function. groupby probably works best when the data is grouped by the key but this isn't strictly necessary. It depends on the use case.

I've successfully used groupby for splitting up the results of large database queries or the contents of large data files. The resulting code ends up being clean and small.

Here's an example:

from itertools import groupby
from operator import itemgetter

things = [('2009-09-02', 11),
          ('2009-09-02', 3),
          ('2009-09-03', 10),
          ('2009-09-03', 4),
          ('2009-09-03', 22),
          ('2009-09-06', 33)]

for key, items in groupby(things, itemgetter(0)):
    print key
    for subitem in items:
        print subitem
    print '-' * 20

Here the dummy data in the "things" list is grouped by the first item of each element (that is, the key is the first element). For each key, the key is printed followed by the items returned by each sub-iterator.

The output looks like:

2009-09-02
('2009-09-02', 11)
('2009-09-02', 3)
--------------------
2009-09-03
('2009-09-03', 10)
('2009-09-03', 4)
('2009-09-03', 22)
--------------------
2009-09-06
('2009-09-06', 33)
-------------------

The "things" list is a contrived example. In a real world situation this could be a database cursor object or a CSV reader object. Any iterable object can be used.

Here's a closer look at what groupby is doing using the Python interactive shell:

>>> iterator = groupby(things, itemgetter(0))
>>> iterator
<itertools.groupby object at 0x95d3acc>
>>> iterator.next()
('2009-09-02', <itertools._grouper object at 0x95e0d0c>)
>>> iterator.next()
('2009-09-03', <itertools._grouper object at 0x95e0aec>)

You can see how a key and sub-iterator are returned for each pass through the groupby iterator.

groupby is a handy tool to have under your belt. Think of it whenever you need to split up a dataset by some criteria.

Tags:
posted: Sun, 18 Oct 2009 14:04 | permalink | comments

Problem installing Trac under Lighttpd+FastCGI

I just finished configuring a Trac instance for IMAPClient. To keep page response times tolerable I'm using FastCGI.

It turns out there's a little gotcha when you serve an app at the root (ie. "/") using FastCGI with Lighttpd. It passes the wrong SCRIPT_NAME and PATH_INFO to the app causing unexpected behaviour. The problem isn't there if the app is served from a prefix (eg. "/trac").

The problem is apparently fixed somewhere in Lighttpd 1.5 but I'm on 1.4.13 as shipped with Debian.

With Trac there is a workaround. If you modify Trac's FastCGI runner (fcgi_frontend.py) to correct the SCRIPT_NAME and PATH_INFO environment variables before the rest of the app sees them, the problem is solved. Here's what my fcgi_frontend.py now looks like (header comments removed):

import pkg_resources

from trac import __version__ as VERSION
from trac.web.main import dispatch_request

from os import environ
import _fcgi

def application(environ, start_request):
    environ['PATH_INFO'] = environ['SCRIPT_NAME'] + environ['PATH_INFO']
    environ['SCRIPT_NAME'] = ''
    return dispatch_request(environ, start_request)

def run():
    _fcgi.WSGIServer(application).run()

if __name__ == '__main__':
    pkg_resources.require('Trac==%s' % VERSION)
    run()

Figuring this out cost me lots of time. Hopefully this information helps someone else out.

Relevant links:

posted: Fri, 28 Aug 2009 01:46 | permalink | comments

Dear Bazaar...

Our relationship has only just started but it's already time for me to move on. Although I think you're great, Mecurial has become my new best friend. Things with her are just a little easier. We just clicked.

I get along so well with her friends; I never did quite fit in with yours. They always seemed somewhat immature.

It's also hard to ignore recent developments. It seems like Mercurial is going to become quite important in the future.

No hard feelings I hope. I truly wish you all the best for the future.

Menno

posted: Wed, 29 Jul 2009 01:03 | permalink | comments

Banishing trailing whitespace with Emacs

One of my pet peeves, especially with Python code, is trailing whitespace. It serves no purpose, introduces noise in diffs and wastes valuable bytes dammit (yes I'm being pedantic).

In order to see make trailing whitespace visible in Emacs you can use the show-trailing-whitespace variable. Emacs of course has a command to remove trailing whitespace: delete-trailing-whitespace.

Better yet, to get rid of trailing whitespace automatically on save you can add a function to the write-contents hook. The following snippet will cause trailing whitespace to be removed on save, but just for Python files.

;; Automatically remove trailing whitespace when saving Python file
(add-hook 'python-mode-hook
          (lambda () (add-hook 'write-contents-hooks 'delete-trailing-whitespace t)))

The one problem with doing this is that when changing an existing codebase your commits could end up with many whitespace deltas, making it difficult to see the meat of your changes. Use with care.

posted: Fri, 03 Jul 2009 12:06 | permalink | comments

draftsdir plugin

I've just published a simple little PyBlosxom plugin called draftsdir. It solves a personal itch for me: I'd like to review draft blog entries on the site before they go live. The plugin lets you define a directory for draft entries which aren't shown unless you add a query arg to the URL (the default arg name is "drafts" but there's an option to change it). When you're happy with an entry you move it from the drafts directory to where your other published entries are. Simple.

For more details see the Code section of this site, the bzr repo or download it directly. There install and configuration instructions at the top of the file.

Tags:
posted: Fri, 26 Jun 2009 00:20 | permalink | comments

Announcing IMAPClient 0.5

It's been a long time between releases but I've just released IMAPClient 0.5.

From the NEWS file:

Folder names are now encoded and decoded transparently if required (using modified UTF-7). This means that any methods that return folder names may return unicode objects as well as normal strings [API CHANGE]. Additionally, any method that takes a folder name now accepts unicode object too. Use the folder_encode attribute to control whether encode/decoding is performed.

Unquoted folder names in server responses are now handled correctly. Thanks to Neil Martinsen-Burrell for reporting this bug.

Fixed a bug with handling of unusual characters in folder names.

Timezones are now handled correctly for datetimes passed as input and for server responses. This fixes a number of bugs with timezones. Returned datetimes are always in the client's local timezone [API CHANGE].

Many more unit tests, some using Michael Foord's excellent mock.py. (http://www.voidspace.org.uk/python/mock/)

IMAPClient can be installed from PyPI using easy_install IMAPClient or downloaded from my Code page. As always, feedback and patches are most welcome.

posted: Wed, 29 Apr 2009 00:54 | permalink | comments

Initial thoughts for PyCon 2009

I'm at PyCon 2009 in Chicago. So much awesome stuff has already been discussed and we're only a few hours in. Much of it very relevant for my employer (which is nice since they're paying).

Some random thoughts so far:

The conference seems really well run and the space at the hotel is great. The only problem I (and many others) have noticed so far is that the wifi can be a tad unreliable. Not suprising given the huge number of wireless devices around.

Tags:
posted: Fri, 27 Mar 2009 16:54 | permalink | comments

All contacts in sync (+1 for the iPhone)

So new employer has given me an iPhone and despite my dislike of vendor lock-in I really like it.

One thing I've now managed to achieve is a single, central set of contact data. Like most geeks I had people's email addresses, phone numbers and postal addresses scattered across various devices and email profiles. In my case this included a Nokia 6130, my laptop's Thunderbird profile, various work Thunderbird profiles, my Gmail account and my trusty old Palm Tungsten E2. None of these stores of contact data were being synced. Contacts were added haphazardly as required.

No longer. The mess has been solved through the use of a variety of tools.

Getting an iPhone was the tipping point. Suddenly it was possible to sync contact (and calendar) data over the air, without needing to connect to a PC. This is very attractive to me. I like being able to sync even when away from the computer where iTunes is installed. I'm using the excellent (and free) NeuvaSync service for this. It hijacks the iPhone's Microsoft Exchange support to push changes to contact data to and from Google Contacts.

I've chosen Google Contacts as the central storage point for this data because it's reliable, I already have a populated gmail account and (mainly) because of the large number of tools that can interface with it.

Syncing to the various Thunderbird instances I use is done using the Zindus extension. This syncs Thunderbird's address book with Google Contacts. It works fairly well as long as you remember to stick to Google's rules about contact entries (eg. email addresses must be unique across all contacts). It will warn you if there's a problem.

So, that covers syncing contact data to everywhere I need it: iPhone, Gmail and Thunderbird.

The final challenge was to pull together all the contact entries I had around the place and merge them into Google Contacts. This was a long and messy process that was helped by some adhoc Python hacking. To get the data out of my Palm I used the pilot-address tool that ships with the pilot-link package. The excellent Gnokii tool pulled the contacts out of my Nokia (over Bluetooth, no less!). I wrote some Python scripts to help merge the highly imperfect sets of data, trying to make intelligent guesses where possible. Finally, OpenOffice Calc was used to manually fix the CSV before importing into Google Contacts.

It took a long time and many editing iterations but I now have a clean, centralised set of contacts. Say what you like about me needing to get out more, but I think this is awesome (and it was a worthwhile learning process).

Tags: ,
posted: Sat, 25 Oct 2008 19:35 | permalink | comments

markup.py

Glenn and I discovered markup.py today. It's a lightweight, simple Python module for generating HTML and XML. Tags are generated using Python objects, with calls to methods on those objects dynamically creating child tags (using __getattr__ trickery). A useful feature is that repeated tags of the same type can be blatted out by passing sequences of values.

The module provides a natural way to create correct HTML/XML from Python and suited our needs perfectly. My only complaint is that the output isn't indented so it can be harder to read than it needs to be. It would probably be straightforward to add such an option. Also, it would be nice if the output could be streamed to a file-like object. Currently it builds the output in memory which could be a potential problem for large pages.

markup.py is worth adding to your toolbox if you ever have to generate bits of XML from Python. For simple jobs, it beats using a complex XML library and is less error prone than plain old string interpolation.

Tags:
posted: Wed, 17 Sep 2008 22:14 | permalink | comments

Slides online for Python on the Openmoko Neo Freerunner

PyCon UK was fantastic. Many thanks to the organisers for such a great event.

As promised at the end of the talk, I've just put the slides and code online for my Python on Openmoko presentation. I'm hoping this stuff is useful to anyone looking to develop Python apps for Openmoko phone.

posted: Mon, 15 Sep 2008 21:52 | permalink | comments

PyCon UK 2008

PyCon UK 2008 is approaching fast. If you're a Python programmer in the UK (or are just Py-curious) then you really should be going. The talk schedule looks hugely exciting and the in-the-hallways and at-the-pub action will undoubtably be fun and engaging. I guarantee you'll learn something about Python and come away feeling inspired.

Disclaimer: I'm presenting :) I'll be doing a talk on Python on the Openmoko Freerunner as well as helping out Michael and Christian with a 4 hour tutorial: Developing with IronPython

posted: Fri, 29 Aug 2008 10:15 | permalink | comments

Neo Freerunner: first impressions

I finally got my hands on my Neo Freerunner two weeks ago and have been playing with it when time allows (so much so that I haven't given myself time to blog about it).

Overall, the hardware is great. The first thing you notice is that the unit feels very solid and the quality of the display is excellent; bright and high resolution. I've had success with wifi, GPS, the SD card slot and basic GSM usage. I haven't had a change to try out the accelerometers yet, mainly due to a (surprising) lack of software that uses them.

Read more...

posted: Tue, 29 Jul 2008 22:46 | permalink | comments

Catching Unhandled Exceptions in .NET

At Resolver, we've been looking at better ways of dealing with unhandled exceptions that occur during test runs. Apart from the need to log that a problem occurred it is important that the dialog boxes that Windows generates don't block the test run (ideally they wouldn't appear at all). We had a hack in place to deal with these dialogs that I won't go into here. Let's just say we've been finding our hack inadequate.

In the .NET world there's 2 APIs that your program can use to be notified about unhandled exceptions. Each covers exceptions that happen in different parts of your code. In order to be comprehensive about catching unhandled exceptions you really need to use both APIs.

Read more...

posted: Thu, 22 May 2008 15:30 | permalink | comments

PyCon Italia presentation materials are online

As promised at the end of my talk, I just uploaded my slides and sample code from PyCon Italia. Included are the S5 slides, a simple Resolver One sample, the IronPython shell example and code for the demo program, mp3stats.

Thanks to Michael Foord for the basis of much of the slide content.

Tags:
posted: Fri, 16 May 2008 11:20 | permalink | comments

PyCon Italia Due

I'm enjoying the wonderful weather in Florence this weekend while attending PyCon Italy 2. Yesterday's highlight was Richard Stallman's thought-provoking keynote titled Free Software in ethics and in practice held in the jaw-dropping Palazzo Vecchio. Stallman's alter-ego Saint IGNUcious (of the church of Emacs) even made an appearance.

My presentation on Sunday covers Application Development in IronPython. It's mainly an introduction to IronPython for Python programmers. Being very much an an Italian language conference, there's real-time translation of English presentations to Italian (mine included). Conversely there's translation from Italian to English in one stream.

The conferences organisers and attendees are being patient with my lack of Italian language skills. I feel very lucky that many Italians can speak English. It's easy to be complacent about learning other languages when you already know English.

I've been fortunate to have met some great people including Raymond Hettinger (core Python team), Arkadiusz Wahlig (Skype4Py) and some of the organisers Simone, Giovanni and Lawrence. It's always great to be able talk shop, exchange perspectives and be inspired. (Resolver One has been getting plugged too!) I'm looking forward to more conversations as the conference continues. The best stuff at conferences always happens outside of the lecture theatres.

ps. The food rocks! The conference lunch today was amazing and last night's Florentine style steak was super-tasty.

Tags:
posted: Sat, 10 May 2008 15:02 | permalink | comments

Resolver One 1.0 Released

As mentioned by several others, Resolver One 1.0 was released on Wednesday!

For those of you who don't know, Resolver One is a unique spreadsheet application written in IronPython, that allows you to easily add functionality using the Python language. The entire spreadsheet is represented as a IronPython program which can be eyeballed and extended. Changes to the spreadsheet grid are reflected in the Python code and changes to the code are reflected on the spreadsheet grid. It's really neat. Resolver is great for people who want to develop complex spreadsheets in a clean, powerful and testable way. It's also useful (and fun!) for programmers when prototyping.

One of the Resolver developers Andrzej, has written a nice article describing 5 Reasons To Try Resolver One. Of course there are more than just 5 reasons :) If you do try out Resolver One, make sure to check out the Resolver Hacks site too.

The 1.0 release of Resolver One is the result of over 2 years of work. Having joined the team very recently I've only been part of the very last bit of that. One thing I've found really interesting is the time leading up to the release date. Compared to other commercial software projects I've worked on the atmosphere felt under-control and (almost) relaxed. It seems that the XP development practices used by the development team are really paying off. No horrible integration issues right before the release, no unexpected bugs on release day, no huge schedule blow-outs. Such a nice difference.

posted: Fri, 18 Jan 2008 18:18 | permalink | comments

Announcing IMAPClient 0.4

I've just released IMAPClient 0.4. Changes are:

Many thanks to Helder Guerreiro for contributing patches for some of the features in this release.

IMAPClient can be installed from PyPI using easy_install IMAPClient or downloaded from my Code page. As always, feedback and patches are most welcome

posted: Fri, 11 Jan 2008 22:58 | permalink | comments

spamquestion 0.2

I released version 0.2 of the spamquestion plugin last night. It now conforms to the requirements for a PyBlosxom plugin. The code is much cleaner too. Available from the usual place.

Tags: ,
posted: Tue, 20 Nov 2007 15:53 | permalink | comments

Making Python Do the Hard Stuff

At work we use a Python decorator to restrict access to certain web controller methods based on the current user's permissions. This was done using by simplying ANDing the permissions together. For example:

@permissions('read', 'write')
def some_method(...):
	...
The above means that a user would need both the 'read' and 'write' permission in order to be able to call the method. The permissions decorator only adds an attribute to the function being decorated. The actual permission enforcement is done in the base controller class as the request comes in.

This approach worked well for a while but then requirements started appearing for more complex relationships between permissions. What if a method required that a user has "read" and "write" or just "admin"? What about negation?

Read more...

posted: Sun, 11 Nov 2007 13:21 | permalink | comments

Blog spam update

Well, two comment spams have made it past the spamquestion plugin. This makes me wonder if either the submissions were done manually or whether the software the spammers use is at least human assisted. I guess it's also possible that the spam software is so good that it can automatically work out my simpler arithmetic questions.

The web server logs give some clues. There's literally hundreds of obviously automated POST attempts to various pages on my blog. The requested related to the two comments that made it through however seem far more human however. Here's one example:

68.187.226.250 freshfoo.com - [03/Nov/2007:01:41:57 +0000] "GET /blog/Holland_photos_online.1024px HTTP/1.1" 200 11367 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
68.187.226.250 freshfoo.com - [03/Nov/2007:01:42:06 +0000] "POST /blog/Holland_photos_online#comment_anchor HTTP/1.1" 200 14928 "http://freshfoo.com/blog/Holland_photos_online.1024px" "Mozilla/4.0 (compatible; MSIE 6.0; Windows
NT 5.1; SV1)"

These are the only two HTTP requests made for the first spam that made it through; no dumb, repeated automatic requests like some of the other attempts in the logs. Notice how the parent page was visited first and then 9 seconds later the POST was made. That's pretty quick for someone to fill out the form manually but it's possible, especially if the spam body was ready in the clipboard. If their system is partially automated then the short delay is even more plausible.

To test whether some spambots are actually capable of doing simple arithmetic by themselves, I've removed all the addition and subtraction questions from my spamquestion configuration and have added more questions that are harder to answer programmatically. If the spam continues, then I'm going to conclude that there's definitely some human assistance going on. If it stops, then it's more likely that the spambot software was actually able to solve some of my arithmetic questions itself.

I also need to look at is short-term blocking of spamming IPs. When examining my logs I found there had been almost 500 comment spam attempts for just today! I'd rather not be dealing with that bandwidth on my server. Dropping all packets from a spammer's IP for a few hours would slow them right down.

Fun fun fun...

Tags: ,
posted: Mon, 05 Nov 2007 22:49 | permalink | comments

Comments renabled, announcing spamquestion

Due to my recent comment spam issues I've created a new PyBlosxom plugin called spamquestion. It is similar to the existing Magic Word plugin but instead of using just the one question for any comment entry on the blog, it randomly selects a question from a larger set of configured questions. This makes it much harder for spammers to get past the comment form using automated software. Unlike CAPTCHA systems this scheme doesn't disadvantage visually impaired people or those on text based browsers.

The spamquestion plugin can be downloaded from my Code page.

Comments are now re-enabled on my site, with spamquestion enabled. It'll be interesting to see how the scheme holds up. I also plan to install the Akismet plugin as a second line of defense.

Tags: ,
posted: Wed, 31 Oct 2007 23:17 | permalink | comments

Announcing IMAPClient 0.3

I've just made a new release of IMAPClient (0.3). Changes are:

IMAPClient can be installed from PyPI using easy_install IMAPClient or downloaded from my Code page. As always, feedback and patches are most welcome

posted: Tue, 16 Oct 2007 15:20 | permalink | comments

Converted to PyBlosxom

I've converted the blog part of this site to use PyBlosxom. It used to be powered by Blogger using their FTP-upload-to-your-own-host feature but I found various parts of the Blogger system to be inflexible and buggy.

PyBlosxom is an old-school CGI requires a certain amount of expertise to get set up but it's solid and has a very powerful plugin system. Most of the features on this page from the archive list to the tag list are provided via plugins. Because PyBlosxom is written in Python it's easy to extend, tweak or write new plugins.

I'm not completely happy with the styling of the pages but it will do for now. I also need to import the comments from the old blog. Please let me know if you see any problems.

Tags:
posted: Sat, 15 Sep 2007 22:46 | permalink | comments

Introducing imapclient

Today I released the first versions of imapclient, an IMAP4 client library I've been working on. From the README:

imapclient aims to be a easy-to-use, Pythonic and complete IMAP client library with no dependencies outside the Python standard library. Features:
  • Arguments and return values are natural Python types.
  • IMAP server responses are fully parsed and readily useable.
  • IMAP unique message IDs (UIDs) are handled transparently. There is no need to call different methods to use UIDs.
  • Convenience methods are provided for commonly used functionality.
  • Exceptions are raised when errors occur.

imapclient makes IMAP useable from Python with little effort. If you've used the imaplib module from the standard library, you'll know that a lot of supporting code is needed just to get simple things done. I'd wager that everyone who has used imaplib has their own fragile regexes and management code to go with it. I hope that imapclient will put an end to this.

imapclient can be downloaded from my Code page. Feedback and patches most welcome!

posted: Wed, 02 May 2007 21:08 | permalink | comments

cert2rss.py retired

A long time ago, I wrote a little script called cert2rss.py that takes the URLs from the US-CERT Summary Bulletin Feed, parsed the HTML summary pages and generated a feed of containing each item individually. I needed something like this for work as I need to keep track security vulnerabilities for all things Linux. The script has been broken for some time because the HTML of summary pages has changed.

Today I finally got around to looking at the problem and found that the HTML of the summary pages is now so bad that it is very difficult to extract useful data from. The HTML looks fine when rendered but it is full of incorrectly escaped text, missing tags and bizarre formatting. Whoever is responsible for generating that HTML shouldn't have their jobs.

So, I'm retiring cert2rss.py. I've updated my Code page to reflect this.

As an alternative, there's some reasonable RSS feeds from the National Vulnerability Database that provide similar functionality.

Tags: ,
posted: Wed, 21 Mar 2007 16:59 | permalink | comments

Hours and hours and hours....

Perhaps I've been living under a rock but I just discovered Best Tech Videos On the Net today. It's a site that indexes all the best technology related videos on the web. There's so many good screencasts and conference sessions. Especially exciting is that there's sessions I wish I'd gone to at conferences I've attended. I could spend hours on this site (and probably will!).

posted: Fri, 16 Mar 2007 15:32 | permalink | comments

londonpython.org.uk updates

As requested by a few people at the last London Python meeting, I've added a button for the Atom feed for londonpython.org.uk. I've also cleaned up the sidebar, added a link to the announce site and fixed a few other minor problems.

Please let know if you see any breakage or would like something changed.

Tags: ,
posted: Tue, 27 Feb 2007 15:21 | permalink | comments

ssh-agent-applet 0.1

I've finally managed to do the first release of ssh-agent-applet. I've actually been using it for some time on my own PC but it needed a bit of love and documentation to make it easier to install.

From the README:

ssh-agent-applet is a Gnome applet that allows you to conveniently keep your ssh key(s) on external media. This means that if your computer is cracked or stolen, the attacker will not have a copy of private ssh key(s).
Using ssh-agent-applet, your keys are loaded into ssh-agent as soon as you insert your "key drive" into a USB port. The drive is automatically unmounted once the key loaded so you can remove it from the USB port immediately.

For developers, ssh-agent-applet is a reasonable example of how to write Gnome applets in Python and how to interface with HAL from Python using DBUS. The HAL/DBUS interactions are non-trivial and go beyond what most documentation covers. It took a fair amount of experimentation to get some of this stuff working. Hopefully the applet or at least its' code is of use to other people.

Downloads can be found on my Code page.

The next thing to do is organise a better build/install system, probably using autoconf. As discussed previously, the current system is a little fragile.

posted: Tue, 13 Feb 2007 14:33 | permalink | comments

distutils is no good for distributing applications

I've been trying to use distutils to create an installation framework for a little Gnome applet written in Python that I've been working on. After struggling with it for a few hours I've decided that it's really not going to do what I want.

The main problem I have is that I need to substitute parts of some files with the final installation paths for various data files. distutils doesn't provide such a capability and bolting it on yourself in a reliable way is rather difficult. I've tried all sorts of ways of extending the distutils core but everything I come up with is fragile, breaking if the user specifies a custom --prefix. It seems that I'm not alone.

I suppose I'm pushing the boundaries of distutils a little too far. Its really intended to distribute Python modules not Python applications. Still, modules that require external data files (outside of site-packages) would run in to similar problems.

I may have to learn how to use autotools or perhaps just roll my own scripts and Makefile. I'd love to know if there's some other alternatives.

Tags:
posted: Fri, 05 Jan 2007 20:27 | permalink | comments

Starting with Ruby

This excellent read from Arto Bendiken has inspired me to do 2 things:

  1. start learning Ruby
  2. keep going with learning Scheme which I haven't touched much since I started teaching myself on the plane to Australia
I hunted around for some Python to Ruby guides and comparisons and found the following useful links: Now I just have to make some time for all this. I still want to keep contributing to Yum and starting doing a Dutch language course as well (my Dutch isn't too bad but I want to get a let more proficient). Yoga is on the cards as well. I'm sure it can be done with a bit of planning and enthusiasm.

posted: Fri, 05 Jan 2007 16:22 | permalink | comments

GStreamer and Python experimentation

I recently played with GStreamer for the first time. I'd heard plenty of good stuff about the framework but had never spent any time developing with it.

In a nutshell, it's very powerful and very cool. Complex multimedia apps can be created with very little code. You link together small blocks to form a useful chain of functionality. The Python API is quite nice because the Python bindings are well supported by the developers.

The big problem is lack of documentation. The best starting point I found was Jono Bacon's excellent Getting Started with GStreamer with Python. After that, the C based docs and the gst-inspect tool are probably your best friends. They make sense once you get your head around the general architecture.

In the interest of expanding the amount of reference information available for GStreamer I've put the results of a few hours of messing around online. It's a simple program that illustrates some interesting stuff using GStreamer. Despite the low line-count, what it does is somewhat impressive; it takes 2 arbitrary audio files and plays them over the top of each mixing in real-time. More details and the download link can be found on my code page.

Tags:
posted: Fri, 27 Oct 2006 20:49 | permalink | comments

Planet London Python is online

I've just installed the Planet London Python site. Planet took a small amount of fiddling but it's all quite logical. A little more customisation is probably required. Some more feeds wouldn't go astray either!

Thanks to Remi from WebFaction for the hosting space.

Tags:
posted: Mon, 09 Oct 2006 13:32 | permalink | comments