classification
Title: IMAP4 missing support for starttls
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: giampaolo.rodola, gregory.p.smith, janssen, lcatucci, pitrou
Priority: normal Keywords: patch

Created on 2008-11-30 16:33 by lcatucci, last changed 2010-11-12 18:49 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
imaplib.rst.patch lcatucci, 2008-11-30 16:45
imaplib_01_SSL_refactor.diff lcatucci, 2008-12-02 15:41
imaplib_02_shutdown.diff lcatucci, 2008-12-02 15:42
imaplib_03_starttls.diff lcatucci, 2008-12-02 15:42
test_imapnet.py lcatucci, 2009-02-07 00:22
imaptls.patch pitrou, 2010-11-09 23:29
Messages (16)
msg76641 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2008-11-30 16:33
In the enclosed patch, there are three changes:
1. Support starttls on IMAP4 connections
2. Rework of the IMAP_SSL, to replace home-grown file-like
   methods with proper ones from ssl module's makefile();
3. Properly shutdown sockets at close() time to avoid server-side pile-up
msg76644 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2008-11-30 16:45
the needed changes to library documentation if the patch is accepted
msg76714 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-12-01 23:05
Same comments than issue #4473:
- you might split your patch into smaller patches
- Do you really need to keep a reference to the "raw" socket?
- I don't understand what is sock.shutdown(SHUT_RDWR). When is it 
needed? Does it change the behaviour?

Oh, another comment:
- When I fixed poplib/imaplib in py3k, I created a _create_socket() 
method which to factorize the "classic" class and the SSL class. The 
classic class uses socket.create_connection() which supports IPv4 and 
IPv6 (and maybe other protocols) whereas the SSL version reimplements 
create_connection(): "for ... in getaddrinfo...". So you may reuse 
this idea for in your SSL refactoring (for POP3 and IMAP4).
msg76718 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2008-12-02 00:02
As in #4473: if needed, I'll redo the patch into a small
series. I've cut and pasted the following.

As for the shutdown before close, it's needed to let the server know
we are leaving, instead of waiting until socket timeout. This is the
reason I need to keep the reference to the wrapped socket.

You don't usually configure maildrop servers to limit the number/rate of
connects as you do on smtp servers; still, you could get into problems
with stateful firewalls or the like.

As for the last comment, I'll gladly look at your changes and try to
copy^H^H^H^Hbackport them.
msg76752 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2008-12-02 15:41
As requested, I've split the patch into three parts: the first one does
just refactor IMAP4_SSL, the second is really a one liner for shutting
down the socket before closing it, and the thirs does introduce the
starttls method in IMAP4.
msg80960 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2009-02-02 17:08
Lorenzo, do we have test cases for this?  I think you should try to add
some test cases.  We may need to set up some test mail servers on
python.org to accommodate such tests.
msg80972 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2009-02-02 18:06
Thanks for following-up, Bill.

While I fully understand the need for unit-testing, I don't have the
guts to start writing a dummy imap server from scratch.

I tested my changes on a couple of servers I manage, one running
uw-imapd and the other running cyrus imapd; still, I don't think unit
tests could rely on a "real" server. On the other hand, I don't think a
python.org hosted test server would be a very wise choice: even if we
found some dummy all-data-in-memory server, I fear the ssl/tls load on
the server.
msg80974 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2009-02-02 18:19
For network tests like this where it is useful to test against external 
servers, could we just pick few known external servers that are unlikely 
to every go away?

imap.gmail.com:993 for instance?  (i don't know enough about imap to 
know if it supports what we need to test, I'll leave that up to you to 
determine :)

Picking another host or two from other big ISPs would also be wise in 
order to test multiple server implementations.

Once that is done, the tests that connect to external servers should be 
put in their own file marked as requiring the network resource.  Similar 
to how its done in Lib/test/test_urllib2net.py.
msg80981 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2009-02-02 18:43
I just found out that the gmail servers don't support connections on
the standard pop3/imapv4 ports, but only on the SSL wrapped ones.

I'm unsure if cmu.edu anonymous imap server admin's would be happy with
their server becoming python's official imap/pop testing playground;
still, I just verified it does support TLS upgrades both for pop3 and imap
and anonymous (read-only) logins.

I think I could cook-up something.
msg81326 - (view) Author: Lorenzo M. Catucci (lcatucci) * Date: 2009-02-07 00:22
As the tests are new, I hope sending the real file is the right way to
proceed.
msg82984 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2009-03-01 20:26
Why can't we use python.org for tests?  Do we need IMAP/POP servers
running?  Let's send some mail to pydotorg to get that set up.
msg83079 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2009-03-03 18:15
I brought this up on pydotorg, and Barry suggests that someone put
together a Twisted environment which could be downloaded and run locally
on the test machine.  It would provide IMAP and POP servers, perhaps
NNTP and others as well.  Now, all we need is someone to make that
happen :-).
msg120902 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-09 23:02
I've committed some of the remote tests in r86380, which also helped me fix a bug in login() in 3.x.
msg120903 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-09 23:17
The shutdown change was committed in r86383.
msg120904 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-09 23:29
Here is an updated STARTTLS patch for py3k.
msg121053 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-12 18:49
The starttls patch has been committed in r86431. Thank you very much for writing the initial patch.
History
Date User Action Args
2011-06-20 03:03:26r.david.murraylinkissue6734 superseder
2010-11-12 18:49:46pitrousetstatus: open -> closed
resolution: fixed
messages: + msg121053

stage: patch review -> resolved
2010-11-09 23:29:57pitrousetfiles: + imaptls.patch
versions: + Python 3.2, - Python 2.6
messages: + msg120904

type: enhancement
stage: patch review
2010-11-09 23:17:41pitrousetmessages: + msg120903
2010-11-09 23:02:48pitrousetnosy: + pitrou
messages: + msg120902
2009-03-03 18:15:19janssensetmessages: + msg83079
2009-03-01 20:26:49janssensetmessages: + msg82984
2009-02-08 22:01:22hayposetnosy: - haypo
2009-02-07 00:22:55lcatuccisetfiles: + test_imapnet.py
messages: + msg81326
2009-02-02 18:43:29lcatuccisetmessages: + msg80981
2009-02-02 18:19:52gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg80974
2009-02-02 18:06:03lcatuccisetmessages: + msg80972
2009-02-02 17:08:33janssensetnosy: + janssen
messages: + msg80960
2008-12-02 15:42:49lcatuccisetfiles: - imaplib.py.patch
2008-12-02 15:42:41lcatuccisetfiles: + imaplib_03_starttls.diff
2008-12-02 15:42:26lcatuccisetfiles: + imaplib_02_shutdown.diff
2008-12-02 15:41:48lcatuccisetfiles: + imaplib_01_SSL_refactor.diff
messages: + msg76752
2008-12-02 00:02:56lcatuccisetmessages: + msg76718
2008-12-01 23:05:56hayposetnosy: + haypo
messages: + msg76714
2008-12-01 21:44:05giampaolo.rodolasetnosy: + giampaolo.rodola
2008-11-30 16:45:18lcatuccisetfiles: + imaplib.rst.patch
messages: + msg76644
2008-11-30 16:33:15lcatuccicreate