Menno's Musings

Banishing trailing whitespace with Emacs

2009-07-03T11:06:22Z

One of my pet peeves, especially with Python code, is trailing whitespace. It serves no purpose, introduces noise in diffs and wastes valuable bytes dammit (yes I'm being pedantic).

In order to see make trailing whitespace visible in Emacs you can use the show-trailing-whitespace variable. Emacs of course has a command to remove trailing whitespace: delete-trailing-whitespace.

Better yet, to get rid of trailing whitespace automatically on save you can add a function to the write-contents hook. The following snippet will cause trailing whitespace to be removed on save, but just for Python files.

;; Automatically remove trailing whitespace when saving Python file
(add-hook 'python-mode-hook
          (lambda () (add-hook 'write-contents-hooks 'delete-trailing-whitespace t)))

The one problem with doing this is that when changing an existing codebase your commits could end up with many whitespace deltas, making it difficult to see the meat of your changes. Use with care.

draftsdir plugin

2009-06-25T23:20:16Z

I've just published a simple little PyBlosxom plugin called draftsdir. It solves a personal itch for me: I'd like to review draft blog entries on the site before they go live. The plugin lets you define a directory for draft entries which aren't shown unless you add a query arg to the URL (the default arg name is "drafts" but there's an option to change it). When you're happy with an entry you move it from the drafts directory to where your other published entries are. Simple.

For more details see the Code section of this site, the bzr repo or download it directly. There install and configuration instructions at the top of the file.

Emacs with Viper mode: editor nirvana?

2009-06-22T20:36:53Z

I've been a long time Vim user and fan. Once you're used proficient with the Vi way of editing it's difficult go back to anything else. Other editors just feel inefficient and clunky.

That said, I've been jealous of certain Emacs features that I've seen while looking over the shoulders of my colleagues. Multiple frames (GUI windows), the clean extension system (via elisp), the tight process integration (shell, SQL, debuggers etc) and all sorts of unexpected bells and whistles; these goodies slowly ate away at my curiosity.

So 2 weeks ago I caved and decided to give Emacs another go. I hadn't really used it since university. Starting with my colleague's configuration and making some minor tweaks, I began using Emacs for serious work.

A few days in and I was enjoying exploring my new toy but it didn't feel quite right. Although I had a reasonable grasp of Emacs' editing keys and commands, most tasks took way too long, requiring convoluted hand gymnastics. My left pinky was permanently sore from constantly reaching for the Ctrl and Alt keys. I was missing those surgical, efficient Vi commands.

At the suggestion of one Emacs-using colleague I gave Viper (viper-mode) a try. It's an emulation that attempts to provide a fairly accurate Vi experience while still allowing full access to the rest of Emacs. To be honest I was expecting it to be a half-assed kludge. I was wrong. Viper is a mature Emacs extension and it does a great job of mitigating conflicts between the Emacs and Vi ways of doing things.

Viper mode proved to be the tipping point; because of it I'm sticking with Emacs. As far as I'm concerned it's the best of both worlds.

For anyone who's interested, my Emacs config is available in the repository browser here or via bzr. This is my personal configuration branch so it will update as I make changes to the setup. Note that I'm using the latest development (but rock solid) Emacs so there might be references to features in the config which only exist in this version.

Some items of note in my config:

vimpulse: Viper mode only emulates classic Vi. vimpulse provides a bunch of extra features which a Vim user will probably miss such as various "g" commands and visual select mode.
ido-mode: This is a standard feature of Emacs which isn't bound to keys by default. It gives amazing power by replacing the standard find-file and switch-buffer keystrokes with beefed up alternatives. The key features are quick directory switching and fuzzy, recursive name matching (but that's not all).
ibuffer: I've replaced the standard buffer list binding (C-x C-b) with ibuffer. This gives a more powerful and easier to use version of the standard buffer list and allows for crazy batch manipulation of buffers.
yasnippet: Mode specific template expansion. Powerful and super useful for cranking out commonly used sections of text (programming is full of them).
flymake - pyflakes integration: Flymake runs arbitrary programs over buffers on-the-fly. For Python files flymake has been configured to run pyflakes and highlight errors in code as I type. I might change this to use pylint at some stage because pylint finds a wider range of problems.

Some useful Emacs config hacking links:

http://steve.yegge.googlepages.com/effective-emacs

http://www.emacswiki.org/

http://xahlee.org/emacs/elisp_idioms.html

bzr repositories now available

2009-06-21T20:41:36Z

I've just made my personal bzr repositories publically available so that anyone can easily get to them (including me!) and so I can refer to things from blog articles. The repos are available for branching using bzr under http://freshfoo.com/repos/ and in human browseable form. See also the links in the left sidebar and in the code section of the site.

I'm using Loggerhead to provide the web viewable form (proxied via the main lighttpd server). It was very easy to setup (using serve-branches). I just wrote a simple init.d script to ensure it stays running.

Announcing IMAPClient 0.5

2009-04-28T23:54:07Z

It's been a long time between releases but I've just released IMAPClient 0.5.

From the NEWS file:

Folder names are now encoded and decoded transparently if required (using modified UTF-7). This means that any methods that return folder names may return unicode objects as well as normal strings [API CHANGE]. Additionally, any method that takes a folder name now accepts unicode object too. Use the folder_encode attribute to control whether encode/decoding is performed.

Unquoted folder names in server responses are now handled correctly. Thanks to Neil Martinsen-Burrell for reporting this bug.

Fixed a bug with handling of unusual characters in folder names.

Timezones are now handled correctly for datetimes passed as input and for server responses. This fixes a number of bugs with timezones. Returned datetimes are always in the client's local timezone [API CHANGE].

Many more unit tests, some using Michael Foord's excellent mock.py. (http://www.voidspace.org.uk/python/mock/)

IMAPClient can be installed from PyPI using easy_install IMAPClient or downloaded from my Code page. As always, feedback and patches are most welcome.

Initial thoughts for PyCon 2009

2009-03-27T16:54:21Z

I'm at PyCon 2009 in Chicago. So much awesome stuff has already been discussed and we're only a few hours in. Much of it very relevant for my employer (which is nice since they're paying).

Some random thoughts so far:

Windmill (the web testing framework) looks incredibly promising. If you're doing web apps and you're a Python shop you should be using this. The Python shell integration rocks.
Cassandra is worth exploring. It's a large-scale, distributed DB that's used (and hacked on) by Facebook with good Python bindings.
Twitter has added a facinating new dimension to the conference. Everyone is tweeting using the #pycon tag. There's all sorts of interesting discussion going on in real-time as talks happen. Twitterfall is a great way to watch the action.

The conference seems really well run and the space at the hotel is great. The only problem I (and many others) have noticed so far is that the wifi can be a tad unreliable. Not suprising given the huge number of wireless devices around.

Vaio BZ11 article updated

2009-01-11T18:11:44Z

I've just updated the Linux on the Vaio BZ11 article on this site. It now includes details for Ubuntu Intrepid Ibex: the webcam now just works, wireless is easy to get functioning. I've also clarfied the information on suspend and resume.

Upgrading iPhone firmware using VMWare on Linux

2008-12-31T11:07:56Z

Well that was a bit scary. I've just updated the firmware on my iPhone to 2.2 (the horrors!). Actually, this normally shouldn't be too much of a trial. The difference here is that I did it from Windows running inside VMWare on a Linux host.

I initially tried the naive approach of applying the update normally using iTunes (but running inside VMWware). Don't do this! The host (Linux) USB subsystem gets in the way leaving you with a somewhat useless iPhone in "recovery mode". It seems that the iPhone firmware upgrade procedure does something tricky with USB that doesn't play nicely with VMWare running on Linux.

To workaround the issue, some USB modules have to not be loaded on the Linux host during the upgrade. Extending the tips on Stephen Laniel's blog I created a blacklist file for modprobe that prevents certain USB modules from loading. Just unloading the modules manually isn't enough. The iPhone reconnects to the USB bus at least once during the upgrade process causing udev to reload the modules you've manually unloaded.

So before attempting the upgrade, create a file named something like /etc/modprobe.d/blacklist-usb with the following contents:

blacklist snd_usb_audio
blacklist usbhid
blacklist ehci_hcd

I'm not sure if ehci_hcd needs to be disabled, but I did this in my case.

Reload udev so that it knows about the new modprobe config file:

sudo /etc/init.d/udev reload

Now make sure these modules aren't already loaded:

sudo /sbin/modprobe -r snd_usb_audio
sudo /sbin/modprobe -r usbhid
sudo /sbin/modprobe -r usbhid

Now go back to VMWare, cross your fingers and attempt the firmware upgrade using iTunes. Things should progress along slowly.

You might find that at one point during the upgrade that iTunes sits there with a dialog saying "Preparing iPhone for upgrade". If this goes on for several minutes it could be that the iPhone isn't connected to VMWare anymore. iTunes is just waiting for the device to appear. If this happens, reattach the iPhone to VMWare using the "Removable Devices" option on VMWare's VM menu. It's a good idea to occasionally check that the iPhone is connected to VMWare during the upgrade.

Once the upgrade is complete you can remove the modprobe config file and reload VMWare:

sudo rm /etc/modprobe.d/blacklist-usb
sudo /etc/init.d/udev reload

For the record, this was done using iTunes 8 running on Vista running inside VMware Workstation 6.5.0 build-118166 on Ubuntu Intrepid (8.10).

All contacts in sync (+1 for the iPhone)

2008-10-25T18:35:30Z

So new employer has given me an iPhone and despite my dislike of vendor lock-in I really like it.

One thing I've now managed to achieve is a single, central set of contact data. Like most geeks I had people's email addresses, phone numbers and postal addresses scattered across various devices and email profiles. In my case this included a Nokia 6130, my laptop's Thunderbird profile, various work Thunderbird profiles, my Gmail account and my trusty old Palm Tungsten E2. None of these stores of contact data were being synced. Contacts were added haphazardly as required.

No longer. The mess has been solved through the use of a variety of tools.

Getting an iPhone was the tipping point. Suddenly it was possible to sync contact (and calendar) data over the air, without needing to connect to a PC. This is very attractive to me. I like being able to sync even when away from the computer where iTunes is installed. I'm using the excellent (and free) NeuvaSync service for this. It hijacks the iPhone's Microsoft Exchange support to push changes to contact data to and from Google Contacts.

I've chosen Google Contacts as the central storage point for this data because it's reliable, I already have a populated gmail account and (mainly) because of the large number of tools that can interface with it.

Syncing to the various Thunderbird instances I use is done using the Zindus extension. This syncs Thunderbird's address book with Google Contacts. It works fairly well as long as you remember to stick to Google's rules about contact entries (eg. email addresses must be unique across all contacts). It will warn you if there's a problem.

So, that covers syncing contact data to everywhere I need it: iPhone, Gmail and Thunderbird.

The final challenge was to pull together all the contact entries I had around the place and merge them into Google Contacts. This was a long and messy process that was helped by some adhoc Python hacking. To get the data out of my Palm I used the pilot-address tool that ships with the pilot-link package. The excellent Gnokii tool pulled the contacts out of my Nokia (over Bluetooth, no less!). I wrote some Python scripts to help merge the highly imperfect sets of data, trying to make intelligent guesses where possible. Finally, OpenOffice Calc was used to manually fix the CSV before importing into Google Contacts.

It took a long time and many editing iterations but I now have a clean, centralised set of contacts. Say what you like about me needing to get out more, but I think this is awesome (and it was a worthwhile learning process).

Linux on the Sony Vaio VGN-BZ11

2008-09-24T22:13:38Z

I've just posted an article on my experiences with Linux on my shiny new Sony Vaio VGN-BZ11XN notebook. Hopefully it's useful to other Linux users who own or are considering purchasing this laptop.

Questions, updates and feedback most definitely welcome.