IMAP IDLE support is not implemented in the current imaplib. A "drop-in" replacement called imaplib2 exists (), but uses internally managed threads - a heavy solution that is not always appropriate (e.g. when handling many IMAP accounts an asynchronous approach would be more efficient)

I am about to start implementation of an asynchronous select()-compatible approach, and was wondering if there has been any discussion over IDLE, any specific reasons it hasn't been implemented and if eventual integration into imaplib would be a desirable thing.

Proposed approach:
* Addition of a new state 'IDLE'
* Addition of an idle() method to class IMAP4, which issues the IDLE command to the server and returns immediately. At this point we enter the IDLE state, in which no normal IMAP commands may be issued.
* Users can now select() or poll() the socket as they wish
* A method can be called to retrieve any untagged responses (e.g. EXISTS) that have arrived since entering the IDLE state. The function returns immediately and does not modify the state.
* To end the IDLE state, the user calls a method (done()?) which resumes the previous state.

Would appreciate any sort of feedback...

imaplib has no particular maintainer and I know little about it.
Doc says it implements 'a large subset of the IMAP4rev1 client protocol as defined in RFC 2060." I do not remember any discussion on pydev, over the last several years, about imaplib. I presume just the subset was chosen because of some combination of necessity and feasibility, as judged by the implementors.  Hence the complement, the unimplemented subset, would be 'not done' rather than 'not wanted'. If your proposed new feature, an IDLE command, is part of this complement, then I would assume that a patch would, in principle, be acceptable.

I cannot comment on your particular proposal, but I hope the above helps as far as it goes.

I just wound up doing a bit of research on this for other reasons.  Piers Lauder was the original author of the imaplib module, and he is (as far as I can tell) currently maintaining an imaplib2 module that does support IDLE (but not, I think, python3).  But it does IDLE (and other things) via threads, and in the email I found announcing it he didn't think it was suitable for stdlib inclusion (because of the threading).  Piers hasn't contributed to core in quite a while as far as I can tell, but he was active in a bug report back in 2008 according to google, so I thought I'd add him to nosy and see if he has time for an opinion.

We have implemented this functionality according to RFC 2177.
We actually implemented a synchronous idle function that blocks until a timeout occurs or the server sent some event.

This is not the most flexible way, however it will provide a basic functionality that enables user to use imap idle based notifications.
Besides, every other solution would require threads or regular polling.

See attached patch file.

To fully answer the original question that opened this issue: contributions will be welcomed.  While I don't currently have time to work on imaplib myself, I have an interest and will review and if appropriate commit patches.

I like Shay's proposal, but absent a patch along those lines having blocking IMAP support will definitely be an improvement.  An application needing to monitor more than one imap connection could do its own threading.

Thanks for proposing the patch.  Could you please submit a contributor agreement?  I probably won't have time to fully consider the proposed patch for a bit, but I've put it on my todo list.  

test_imaplib does have a testing framework now, do you think you could write tests for the new feature?

I got the confirmation for my agreement.

I'm not quite sure about the tests, as I'm not really familiar with the way this is done in cpython. The test_imaplib.py seems to cover all ways to connect to some server, but none of the actual imap commands. The patch only implements another commands (whose behaviour is highly/only dependent of other events on the server).

Hence, I don't see a way to create a meaningfull test case other than just calling the command...

Yeah writing a good test case for this is a bit tricky, since we'll need some infrastructure (an Event?) so we can prove that the call has blocked without delaying the test for very long.

I'm not sure when I'll be able to get to this...I'm not going to check it in without a test, so I'll have to find enough time to be able to write the tests.

I stumbled about this issue again and would really like to see it fixed.

I see the possibility to create a test case in combination with the first test sequence which creates a temporary mail. Would it be enough, that we just call IDLE in some folder, create a temporary mail in this folder and check if it returns?

Unfortuantely, I have not been able to write code for such a test case yet, as the whole test routine fails with "[PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections". This is using 3.2.3, but I guess it will not be any different with the current release... (as it is the same with 2.7.3)

What do you mean by the whole test routine failing?  The test suite is currently passing on the buildbots, so are you speaking of the new test you are trying to write?

Hmm.  Looking at this again, it appears as though there's no way to interrupt IDLE if you want to, say, send an email.  If you are actually using this in code, how are you handling that situation?

So, let's resurrect this one.

For the project that lead to the old patch, we did not need this feature.
However, we now needed are more complete implementation of IDLE.
Hence, we extended this to return after sending idle() and support polling, leaving idle mode or wait until something happens (like before).

IMAP polling hurts, just merge imaplib2 into standard library as imaplib.

Piers Lauder authored imaplib IMAP4 client, part of python standard library, back in December 1997 based on RFC 2060. In 2003 RFC 2060 was made obsolete by RFC 3501 adding important features and Piers released imaplib2 which receives feature updates since.
Last feature updates to the standard library imaplib were before Piers retired from Sydney University a decade ago.

imaplib2 presents an almost identical API as that provided by the standard library imaplib, the main difference being that imaplib2 allows parallel execution of commands on the IMAP4 server, and implements the IDLE extension, so NO POLLING IS REQUIRED. IMAP server will push new mail notifications to the client. Imaplib2 also supports COMPRESS, ID, better timeout handling etc. There is 975 more lines of code all doing useful things a modern IMAP client needs.

imaplib2 can be substituted for imaplib in existing clients with no changes in the code apart from required logout call to shutdown the threads.

Old imaplib was ported to Python 3 with the rest of the standard library. I am working to port imaplib2 to py3, stuck on receiving bytes v strings.

References:

imaplib2 code and docs
http://sourceforge.net/p/imaplib2/code/ci/master/tree/
also http://sydney.edu.au/engineering/it/~piers/python/imaplib2.html

imaplib
https://hg.python.org/cpython/file/3.4/Lib/imaplib.py

Ruby stdlib support for idle (not that it hurts python performance, just my pride)
http://ruby-doc.org/stdlib-2.0.0/libdoc/net/imap/rdoc/Net/IMAP.html#method-i-idle

Imaplib2 now supports Python 3. Piers and me propose to merge imaplib2 into standard library as imaplib.

Excerpt from our conversation:

Piers: ...Thanks for bringing it (this thread) to my attention. I entirely agree with your comments.

Me: ...I found the criticism of the "threads - a heavy solution"? counterproductive. Not that I know anything about threads...

Piers: I'm not sure what the whole anti-threads thing was about all those years ago since I always loved using them. Maybe early implementations were
slow, or, more likely, early adopters were clumsy ("giving threads to
a novice is like giving a blow torch to a baby" to paraphrade a quote :-)

Are you volunteering to be maintainer, and/or is Piers?  If he's changed his mind about the threading, that's good enough for me (and by now he has a lot more experience with the library in actual use).

The biggest barrier to inclusion, IMO, is tests and backward compatibility.  There have been enough changes that making sure we don't break backward compatibility will be important, and almost certainly requires more tests than we have now.  Does imaplib2 have a test suite?

We would need to get approval from python-dev, though.  We have ongoing problems with packages that are maintained outside the stdlib...but updating to imaplib2 may be better than leaving it without a maintainer at all.

Can we get Piers involved in this conversation directly?

I am in for my part and I emailed Piers to come and join us and he surely
will when the bug tracker is responsive again.

Imaplib2 does have an up to date test suite and compatibility wise imaplib2
can be substituted for imaplib in existing clients with no changes in the
code.

On top of that I have a private IDLE test suite for common tasks such as
- instant bounce processing,
- email photo upload to web service,
- unsubscribe processing via list-unsubscribe headers and feedback loops of
major email providers.

I am looking to make it part of an external project
https://github.com/fmalina/emails, but need to extract much of the recipes
first out of a working application in a reusable manner as I need it in
other projects and will do for years to come.

This is great.

When you say it is fully compatible, though, is that testing against imaplib in python2 or python3?  It is the python3 decisions about string/bytes handling where the discrepancies are most likely to arise, unless the python3 port was modeled on the stdlib version.

I just wen’t through my repo looking at relevant commits to double check and I didn’t have to change a line in my user level code when upgrading from python2 to 3. There was only one way to do it.

Do you have any tests that use non-ascii passwords?  I think that was the most significant bug.

I don’t have a test for it, neither has stdlib imaplib. We just need to port over the encode fix.

Copy over the fixed version of _CRAM_MD5_AUTH.
from line 599 in python3.5 imaplib
https://github.com/python/cpython/blob/master/Lib/imaplib.py#L599 <https://github.com/python/cpython/blob/master/Lib/imaplib.py#L599>
corresponding to line 884 in imaplib2
https://github.com/bcoe/imaplib2/blob/master/imaplib2/imaplib2.py#L884 <https://github.com/bcoe/imaplib2/blob/master/imaplib2/imaplib2.py#L884>

Hi, apologies for not responding to the "pierslauder" pings, but i don't own that login, or at least have forgotten all about it, and its email address is invalid (or there is another pierslauder out there).

I maintain imaplib2 on sourceforge (as piersrlauder) at https://sourceforge.net/projects/imaplib2/
and that version has just been modified to incorporate the CRAM_MD5_AUTH change from python3.6. It is regularly updated with bug fixes and it also has built-in tests for the IDLE function.

I originally intended for imaplib2 to be incorporated into pythonlib, leaving the original module in place (a la urllib/2). Then people wouldn't be forced into a switch using threads except by choice.

Anyway, happy to help.

Thanks, Piers!

Sorry for dropping off the map on this, I've been busy.

I'll post to python-dev about this and see how the community would like to proceed.

By the way, the pierslauder id points to 'pierslauder@users.sourceforge.net'.

Before merging `imaplib2` please consider making proper use of the Python's standard `logging` module.

Hi, I'm new to python but I had a go at implementing this for imaplib(1) using a different approach. It works but it has a couple issues (see patch), I would appreciate any thoughts/improvements.