IMAPClient 1.0.0

IMAPClient 1.0.0 is finally here! This is a monster release, bursting with new features and fixes.

Here's the highlights:

Enhanced TLS support: The way that IMAPClient establishes TLS connections has been completely reworked. By default, IMAPClient will attempt certificate verification and certificate hostname checking, and will not use known-insecure TLS settings and protocols. In addition, TLS parameters are now highly configurable.

This change necessitates that backwards compatibility has been broken, and also means that IMAPClient has a bunch of new dependencies. Please see the earlier blog article about the TLS changes as well as the release notes for more information.

STARTTLS support: When the server supports it, IMAPClient can now establish an encrypted connection after initially starting with an unencrypted connection using the STARTTLS command. The starttls method takes an SSL context object for controlling the parameters of the TLS negotiation.

Many thanks to Chris Arndt for his extensive initial work on this.

Robust search criteria handling: IMAPClient's methods that accept search criteria have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly.

Unfortunately these changes also mean that small changes may be required to existing code that uses IMAPClient. Please see the earlier blog article about the search changes as well as the release notes for more information.

Socket timeout support: IMAPClient now accepts a timeout at creation time. The timeout applies while establishing the connection and for all operations on the socket connected to the IMAP server.

Semantic versioning: In order to better indicate version compatibility to IMAPClient's users, the project will now strictly adhere to the Semantic Versioning scheme.

Performance optimisation for parsing message id lists: A short circuit is now used when parsing a list of message ids which greatly speeds up parsing time.

Installation via wheels: In addition to .zip and .tar.gz files, IMAPClient releases are now also available as universal wheels.

There have also been many other smaller fixes and improvements. See the release notes and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded via IMAPClient site.

This release couldn't have been possible with the amazing support of Nylas. If you're developing software that needs to deal with email, save yourself a whole lot of pain and check out their email platform. If you're after a modern, extensible, cross-platform email client check out N1.

posted: Tue, 24 Nov 2015 04:45 | permalink | comments

Search Changes in IMAPClient 1.0

Following on from last week's post about the upcoming IMAPClient 1.0 release, I'd like to draw attention to some significant, compatibility breaking changes related to the handling of search criteria.

IMAPClient's methods that accept search criteria (search, sort, thread, gmail_search) have been changed to provide take criteria in a more straightforward and robust way. In addition, the way the charset argument interacts with search criteria has been improved. These changes make it easier to pass search criteria and have them handled correctly but unfortunately also mean that small changes may be required to existing code that uses IMAPClient.

Search criteria

The preferred way to specify criteria now is as a list of strings, ints and dates (where relevant). The list should be flat with all the criteria parts run together. Where a criteria takes an argument, just provide it as the next element in the list.

Some valid examples:

c.search(['DELETED'])
c.search(['NOT', 'DELETED'])
c.search(['FLAGGED', 'SUBJECT', 'foo', 'BODY', 'hello world'])
c.search(['NOT', 'DELETED', 'SMALLER', 1000])
c.search(['SINCE', date(2006, 5, 3)])

IMAPClient will perform all required conversion, quoting and encoding. Callers do not need to and should not attempt to do this themselves. IMAPClient will automatically send criteria parts as IMAP literals when required (i.e. when the encoded part is 8-bit).

Some previously accepted ways of passing search criteria will not work as they did in previous versions of IMAPClient. Small changes will be required in these cases. Here are some examples of how to update code written against older versions of IMAPClient:

c.search(['NOT DELETED'])    # Before
c.search(['NOT', 'DELETED']) # After

c.search(['TEXT "foo"'])     # Before
c.search(['TEXT', 'foo'])    # After (IMAPClient will add the quotes)

c.search(['DELETED', 'TEXT "foo"'])    # Before
c.search(['DELETED', 'TEXT', 'foo'])   # After

c.search(['SMALLER 1000'])    # Before
c.search(['SMALLER', 1000])   # After

It is also possible to pass a single string as the search criteria. IMAPClient will not attempt quoting in this case, allowing the caller to specify search criteria at a lower level. Specifying criteria using a sequence of strings is preferable however. The following examples (equivalent to those further above) are valid:

c.search('DELETED')
c.search('NOT DELETED')
c.search('FLAGGED SUBJECT "foo" BODY "hello world"')
c.search('NOT DELETED SMALLER 1000')
c.search('SINCE 03-May-2006')

Search charset

The way that the search charset argument is handled has also changed.

Any unicode criteria arguments will now be encoded by IMAPClient using the supplied charset. The charset must refer to an encoding that is capable of handling the criteria's characters or an error will occur. The charset must obviously also be one that the server supports! (UTF-8 is common)

Any criteria given as bytes will not be changed by IMAPClient, but the provided charset will still be passed to the IMAP server. This allows already encoding criteria to be passed through as-is. The encoding referred to by charset should match the actual encoding used for the criteria.

The following are valid examples:

c.search(['TEXT', u'\u263a'], 'utf-8')         # IMAPClient will apply UTF-8 encoding
c.search([b'TEXT', b'\xe2\x98\xba'], 'utf-8')  # Caller has already applied UTF-8 encoding

The documentation and tests for search, gmail_search, sort and thread has updated to account for these changes and have also been generally improved.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback on the changes and/or documentation would be hugely appreciated.

posted: Tue, 15 Sep 2015 04:46 | permalink | comments

TLS Changes in IMAPClient 1.0

IMAPClient 1.0 is really close to being done now and it's going to be one of the biggest releases in its history (thanks largely to the support of the good people at Nylas).

The headline feature of this release is the completely revamped TLS support. With 1.0, IMAPClient will perform certificate verification, certificate hostname checking, and will not use known-insecure TLS settings and protocols - by default. In order to work around Python historically patchy TLS support [*], IMAPClient uses backports.ssl and pyOpenSSL to provide consistent TLS functionality across all supported Python versions (2.6, 2.7, 3.3 and 3.4).

All this goodness doesn't come for free however. There were some compromises and compatibility breaks required to make it work:

  1. Due to lack of support in some of the new dependencies, IMAPClient no longer supports Python 3.2.
  2. The keyword arguments that were accepted by the IMAPClient constructor to pass through to imaplib's IMAP4 constructor are no longer supported. This was in place to provide access to imaplib's SSL arguments which are no longer relevant. Please pass a SSL context object instead.
  3. When using the default SSL context that IMAPClient creates (recommended), certificate verification is enabled and various bad TLS settings are disabled. This means that IMAPClient may fail to establish TLS connections to servers that used to work with previous versions of IMAPClient (especially if a self-signed certificate is used by the server). Fortunately it's not difficult to supply your own CA certificates or disable verification if required.

There's a new section in the manual which has more details and includes examples of how to tweak the SSL context for some common scenarios.

For those willing to try out the changes now please install from IMAPClient's tip. Any feedback would be hugely appreciated.

[*]Note that due to the hard work of various folks, TLS support is much better in Python 3.4 and 2.7.9.

posted: Tue, 08 Sep 2015 11:46 | permalink | comments

IMAPClient 0.13

I'm chuffed to announce that IMAPClient 0.13 is out!

Here's what's new:

See the NEWS.rst file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

I'm also excited to announce that Nylas (formerly Inbox) has now employed me to work on IMAPClient part time. There should be a significant uptick in the development of IMAPClient.

The next major version of IMAPClient will be 1.0.0, and will be primarily focussed on enhancing TLS/SSL support.

posted: Tue, 30 Jun 2015 11:38 | permalink | comments

IMAPClient introduction

I gave a presentation introducting IMAPClient at the monthly Christchurch Python meetup on Thursday. It included a brief introduction to the IMAP protocol, the motivation for creating the IMAPClient package, some examples of how it compares to using imaplib from the standard library, and some discussion of future plans. There also may have been some silly pictures. The talk seemed to be well received. Thanks to everyone who attended.

I'm not sure how useful they'll be without the vocal part but the slides are now available online.

Apart from my talk, a wide range of topics came up including installing and managing multiple versions of Python side-by-side, TLS (SSL) and Python and the suckiness (or not) of JavaScript. As always, lots of fun. There's always a great bunch of people who make it along.

posted: Sun, 08 Mar 2015 08:59 | permalink | comments

IMAPClient now all at Bitbucket

I've been wanting to do this for a while: IMAPClient is now completely hosted on Bitbucket. The Trac instance is no more and all tickets have been migrated to the Bitbucket issue tracker. http://imapclient.freshfoo.com now redirects to the Bitbucket site.

The primary motivation for this change is that it makes it possible for anyone to easily interact with IMAPClient's tickets using the Bitbucket account they probably already have. Due to spam issues, I had to turn off public write access to the Trac bug tracker a long time ago meaning I was the only one creating and updating tickets. This change also means less maintenance overhead for me, so more time to spend on IMAPClient.

Please let me know if you see anything related to the migration that doesn't look right.

posted: Sat, 28 Feb 2015 11:32 | permalink | comments

IMAPClient 0.12

I'm very happy to announce that IMAPClient 0.12 is out!

This is a big release. Some highlights:

See the NEWS.rst file and manual for more details.

Many thanks go to Inbox for sponsoring the significant unicode changes in this release.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

The next major version of IMAPClient will be 1.0.0, and will be primarily focussed on enhancing TLS/SSL support.

posted: Mon, 12 Jan 2015 12:14 | permalink | comments

Go for Pythonistas

I gave a talk last night at the monthly Christchurch Python meetup titled "Go For Pythonistas". It was an introduction to the Go programming language from the perspective of a Python developer. The talk went well, with plenty of questions and comments throughout. Thanks to all who attended for your interest!

I'm not sure how useful they'll be on their own but the slides for the talk are available here.

I mentioned Juju in the talk a few times which piqued a few people's interest so I also ended up doing a short impromptu talk about that. Juju is the amazing cloud orchestration software that I hack on at Canonical these days (written in Go). It's worth a few blog articles in itself.

Tags: ,
posted: Thu, 07 Aug 2014 22:58 | permalink | comments

Inbox is sponsoring IMAPClient development

Now that they have officially launched I can happily announce that the good folks at Inbox are sponsoring the development of certain features and fixes for IMAPClient. Inbox have just released the initial version of their open source email sync engine which provides a clean REST API for dealing with email - hiding all the archaic intricacies of protocols such as IMAP and MIME. IMAPClient is used by the Inbox engine to interact with IMAP servers.

The sponsorship of IMAPClient by Inbox will help to increase the speed of IMAPClient development and all improvements will be open source, feeding directly in to trunk so that all IMAPClient users benefit. Thanks Inbox!

The first request from Inbox is to fix some unicode/bytes handling issues that crept in as part of the addition of Python 3 support. It's a non-trivial amount of work but things are going well. Watch this space...

posted: Wed, 16 Jul 2014 12:10 | permalink | comments

IMAPClient 0.11

This is a somewhat belated announcement that IMAPClient 0.11 is out (and has been for a little over a week now). Notable changes:

Big thanks go out to John Louis del Rosario, Naveen Nathan, Brandon Rhodes and Benjamin Morrise for their contributions towards this release.

See the NEWS.rst file and manual for more details.

IMAPClient can be installed from PyPI (pip install imapclient) or downloaded from the IMAPClient site.

posted: Sun, 18 May 2014 11:43 | permalink | comments