Nikola

Those paying attention may have noticed that this site has changed. Up until recently, freshfoo.com was generated using a combination of PyBlosxom and rest2web but it's now using the wonderful Nikola static site generator. The change was prompted by a desire to have more flexibility about freshfoo.com's hosting situation - moving a static site around is far easier than a dynamically generated one. A static site also provides a snappier experience for the end user and there really wasn't a need for the site to be dynamically generated anyway.

Nikola also provides some nice tooling for the site author to help with the writing process. The nikola auto command which serves up the site locally, automatically rebuilding as you work on it and automatically refreshing browser tabs which are viewing the site is especially useful.

Read more…

IMAPClient mailing list moved to Groups.io

The IMAPClient mailing list has been hosted at Librelist for a while. It worked OK initially but the site doesn't seem to be getting much love these days. People also tend to find the "send an email to subscribe" model confusing.

After looking at a number of options, I've decided to shift the mailing list to Groups.io. It's a well run mailing list service which allows interaction via both the web and email. There's lots of nice bells and whistles but the main benefit I see is that it seems to be well thought out and efficient.

To join the new list or to find out more, take a look at https://groups.io/g/imapclient.

The change to the new mailing list is effective immediately. Please use the new one from now on. I'm going to send out invites to people who were active on the old list. IMAPClient's documentation has already been updated to reflect the change.

To ensure they aren't lost, the historical messages from the old list on librelist.com will be imported into new one.

Please email me or comment here if you have any feedback about the change.

IMAPClient 1.0.0

IMAPClient 1.0.0 is finally here! This is a monster release, bursting with new features and fixes.

Here's the highlights:

Enhanced TLS support: The way that IMAPClient establishes TLS connections has been completely reworked. By default, IMAPClient will attempt certificate verification and certificate hostname checking, and will not use known-insecure TLS settings and protocols. In addition, TLS parameters are now highly configurable.

This change necessitates that backwards compatibility has been broken, and also means that IMAPClient has a bunch of new dependencies. Please see the earlier blog article about the TLS changes as well as the release notes for more information.

STARTTLS support: When the server supports it, IMAPClient can now establish an encrypted connection after initially starting with an unencrypted connection using the STARTTLS command. The starttls method takes an SSL context object for controlling the parameters of the TLS negotiation.

Many thanks to Chris Arndt for his extensive initial work on this.

Robust search criteria handling: IMAPClient's methods that accept search criteria have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly.

Unfortunately these changes also mean that small changes may be required to existing code that uses IMAPClient. Please see the earlier blog article about the search changes as well as the release notes for more information.

Socket timeout support: IMAPClient now accepts a timeout at creation time. The timeout applies while establishing the connection and for all operations on the socket connected to the IMAP server.

Semantic versioning: In order to better indicate version compatibility to IMAPClient's users, the project will now strictly adhere to the Semantic Versioning scheme.

Performance optimisation for parsing message id lists: A short circuit is now used when parsing a list of message ids which greatly speeds up parsing time.

Installation via wheels: In addition to .zip and .tar.gz files, IMAPClient releases are now also available as universal wheels.

There have also been many other smaller fixes and improvements. See the release notes and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded via IMAPClient site.

This release couldn't have been possible with the amazing support of Nylas. If you're developing software that needs to deal with email, save yourself a whole lot of pain and check out their email platform. If you're after a modern, extensible, cross-platform email client check out N1.

Search Changes in IMAPClient 1.0

Following on from last week's post about the upcoming IMAPClient 1.0 release, I'd like to draw attention to some significant, compatibility breaking changes related to the handling of search criteria.

IMAPClient's methods that accept search criteria (search, sort, thread, gmail_search) have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly but unfortunately also mean that small changes may be required to existing code that uses IMAPClient.

Search criteria

The preferred way to specify criteria now is as a list of strings, ints and dates (where relevant). The list should be flat with all the criteria parts run together. Where a criteria takes an argument, just provide it as the next element in the list.

Some valid examples:

c.search(['DELETED'])
c.search(['NOT', 'DELETED'])
c.search(['FLAGGED', 'SUBJECT', 'foo', 'BODY', 'hello world'])
c.search(['NOT', 'DELETED', 'SMALLER', 1000])
c.search(['SINCE', date(2006, 5, 3)])

IMAPClient will perform all required conversion, quoting and encoding. Callers do not need to and should not attempt to do this themselves. IMAPClient will automatically send criteria parts as IMAP literals when required (i.e. when the encoded part is 8-bit).

Some previously accepted ways of passing search criteria will not work as they did in previous versions of IMAPClient. Small changes will be required in these cases. Here are some examples of how to update code written against older versions of IMAPClient:

c.search(['NOT DELETED'])    # Before
c.search(['NOT', 'DELETED']) # After

c.search(['TEXT "foo"'])     # Before
c.search(['TEXT', 'foo'])    # After (IMAPClient will add the quotes)

c.search(['DELETED', 'TEXT "foo"'])    # Before
c.search(['DELETED', 'TEXT', 'foo'])   # After

c.search(['SMALLER 1000'])    # Before
c.search(['SMALLER', 1000])   # After

It is also possible to pass a single string as the search criteria. IMAPClient will not attempt quoting in this case, allowing the caller to specify search criteria at a lower level. Specifying criteria using a sequence of strings is preferable however. The following examples (equivalent to those further above) are valid:

c.search('DELETED')
c.search('NOT DELETED')
c.search('FLAGGED SUBJECT "foo" BODY "hello world"')
c.search('NOT DELETED SMALLER 1000')
c.search('SINCE 03-May-2006')

Search charset

The way that the search charset argument is handled has also changed.

Any unicode criteria arguments will now be encoded by IMAPClient using the supplied charset. The charset must refer to an encoding that is capable of handling the criteria's characters or an error will occur. The charset must obviously also be one that the server supports! (UTF-8 is common)

Any criteria given as bytes will not be changed by IMAPClient, but the provided charset will still be passed to the IMAP server. This allows already encoding criteria to be passed through as-is. The encoding referred to by charset should match the actual encoding used for the criteria.

The following are valid examples:

c.search(['TEXT', u'\u263a'], 'utf-8')         # IMAPClient will apply UTF-8 encoding
c.search([b'TEXT', b'\xe2\x98\xba'], 'utf-8')  # Caller has already applied UTF-8 encoding

The documentation and tests for search, gmail_search, sort and thread has updated to account for these changes and have also been generally improved.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback on the changes and/or documentation would be hugely appreciated.

TLS Changes in IMAPClient 1.0

IMAPClient 1.0 is really close to being done now and it's going to be one of the biggest releases in its history (thanks largely to the support of the good people at Nylas).

The headline feature of this release is the completely revamped TLS support. With 1.0, IMAPClient will perform certificate verification, certificate hostname checking, and will not use known-insecure TLS settings and protocols - by default. In order to work around Python historically patchy TLS support [*], IMAPClient uses backports.ssl and pyOpenSSL to provide consistent TLS functionality across all supported Python versions (2.6, 2.7, 3.3 and 3.4).

All this goodness doesn't come for free however. There were some compromises and compatibility breaks required to make it work:

  1. Due to lack of support in some of the new dependencies, IMAPClient no longer supports Python 3.2.
  2. The keyword arguments that were accepted by the IMAPClient constructor to pass through to imaplib's IMAP4 constructor are no longer supported. This was in place to provide access to imaplib's SSL arguments which are no longer relevant. Please pass a SSL context object instead.
  3. When using the default SSL context that IMAPClient creates (recommended), certificate verification is enabled and various bad TLS settings are disabled. This means that IMAPClient may fail to establish TLS connections to servers that used to work with previous versions of IMAPClient (especially if a self-signed certificate is used by the server). Fortunately it's not difficult to supply your own CA certificates or disable verification if required.

There's a new section in the manual which has more details and includes examples of how to tweak the SSL context for some common scenarios.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback would be hugely appreciated.

[*] Note that due to the hard work of various folks, TLS support is much better in Python 3.4 and 2.7.9.

IMAPClient 0.13

I'm chuffed to announce that IMAPClient 0.13 is out!

Here's what's new:

  • Added support for the ID command (as per RFC2971). Many thanks to Eben Freeman from Nylas.
  • Fixed exception with NIL address in envelope address list. Thomas Steinacher gets a big thank you for this one.
  • Fixed a regression in the handling of NIL/None SEARCH responses. Thanks again to Thomas Steinacher.
  • Don't traceback when an unparseable date is seen in ENVELOPE or INTERNALDATE responses. None is now returned instead.
  • Extended timestamp parsing support to allow for quirky timestamp strings which use dots for the time separator.
  • Replaced the horrible INTERNALDATE parsing code
  • The datetime_to_imap top-level function has been moved to the datetime_util module and is now called datetime_to_INTERNALDATE. This will only affect you in the unlikely case that you were importing this function out of the IMAPClient package.
  • The docs for various IMAPClient methods, and the HACKING.rst file have been improved.
  • CONDSTORE live test is now more reliable (especially when running against Gmail)

See the NEWS.rst file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

I'm also excited to announce that Nylas (formerly Inbox) has now employed me to work on IMAPClient part time. There should be a significant uptick in the development of IMAPClient.

The next major version of IMAPClient will be 1.0.0, and will be primarily focussed on enhancing TLS/SSL support.

IMAPClient 0.12

I'm very happy to announce that IMAPClient 0.12 is out!

This is a big release. Some highlights:

  • Unicode handling has been fixed. Some bad decisions were made during the Python 3 port (v0.10) and this release fixes that. Bytes are now returned in most places (instead of unicode strings).
  • MODSEQ parts in SEARCH responses are now handled correctly. A crash has been fixed when MODSEQ queries (part of the CONDSTORE extension) are made with the search method. The returned MODSEQ value is now available via the "modseq" attribute on the returned list of ids.
  • Extra __init__ keyword args are passed through. This allows access to SSL options that the underlying imaplib library might support (Python version dependent).
  • Python 3.4 is now officially supported.
  • More control over OAUTH2 parameters.
  • The deprecated get_folder_delimiter() method has been removed.

See the NEWS.rst file and manual for more details.

Many thanks go to Inbox for sponsoring the significant unicode changes in this release.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

The next major version of IMAPClient will be 1.0.0, and will be primarily focussed on enhancing TLS/SSL support.

IMAPClient introduction

I gave a presentation introducting IMAPClient at the monthly Christchurch Python meetup on Thursday. It included a brief introduction to the IMAP protocol, the motivation for creating the IMAPClient package, some examples of how it compares to using imaplib from the standard library, and some discussion of future plans. There also may have been some silly pictures. The talk seemed to be well received. Thanks to everyone who attended.

I'm not sure how useful they'll be without the vocal part but the slides are now available online.

Apart from my talk, a wide range of topics came up including installing and managing multiple versions of Python side-by-side, TLS (SSL) and Python and the suckiness (or not) of JavaScript. As always, lots of fun. There's always a great bunch of people who make it along.

IMAPClient now all at Bitbucket

I've been wanting to do this for a while: IMAPClient is now completely hosted on Bitbucket. The Trac instance is no more and all tickets have been migrated to the Bitbucket issue tracker. http://imapclient.freshfoo.com now redirects to the Bitbucket site.

The primary motivation for this change is that it makes it possible for anyone to easily interact with IMAPClient's tickets using the Bitbucket account they probably already have. Due to spam issues, I had to turn off public write access to the Trac bug tracker a long time ago meaning I was the only one creating and updating tickets. This change also means less maintenance overhead for me, so more time to spend on IMAPClient.

Please let me know if you see anything related to the migration that doesn't look right.

Go for Pythonistas

I gave a talk last night at the monthly Christchurch Python meetup titled "Go For Pythonistas". It was an introduction to the Go programming language from the perspective of a Python developer. The talk went well, with plenty of questions and comments throughout. Thanks to all who attended for your interest!

I'm not sure how useful they'll be on their own but the slides for the talk are available here.

I mentioned Juju in the talk a few times which piqued a few people's interest so I also ended up doing a short impromptu talk about that. Juju is the amazing cloud orchestration software that I hack on at Canonical these days (written in Go). It's worth a few blog articles in itself.