In the enclosed patch, there are three changes:

1. Support starttls on IMAP4 connections
2. Rework of the IMAP_SSL, to replace home-grown file-like
   methods with proper ones from ssl module's makefile();
3. Properly shutdown sockets at close() time to avoid server-side pile-up

the needed changes to library documentation if the patch is accepted

Same comments than issue bpo-4473:

• you might split your patch into smaller patches
• Do you really need to keep a reference to the "raw" socket?
• I don't understand what is sock.shutdown(SHUT_RDWR). When is it
  needed? Does it change the behaviour?

Oh, another comment:

• When I fixed poplib/imaplib in py3k, I created a _create_socket()
  method which to factorize the "classic" class and the SSL class. The
  classic class uses socket.create_connection() which supports IPv4 and
  IPv6 (and maybe other protocols) whereas the SSL version reimplements
  create_connection(): "for ... in getaddrinfo...". So you may reuse
  this idea for in your SSL refactoring (for POP3 and IMAP4).

As in bpo-4473: if needed, I'll redo the patch into a small
series. I've cut and pasted the following.

As for the shutdown before close, it's needed to let the server know
we are leaving, instead of waiting until socket timeout. This is the
reason I need to keep the reference to the wrapped socket.

You don't usually configure maildrop servers to limit the number/rate of
connects as you do on smtp servers; still, you could get into problems
with stateful firewalls or the like.

As for the last comment, I'll gladly look at your changes and try to
copy^H^H^H^Hbackport them.

As requested, I've split the patch into three parts: the first one does
just refactor IMAP4_SSL, the second is really a one liner for shutting
down the socket before closing it, and the thirs does introduce the
starttls method in IMAP4.

Lorenzo, do we have test cases for this? I think you should try to add
some test cases. We may need to set up some test mail servers on
python.org to accommodate such tests.

Thanks for following-up, Bill.

While I fully understand the need for unit-testing, I don't have the
guts to start writing a dummy imap server from scratch.

I tested my changes on a couple of servers I manage, one running
uw-imapd and the other running cyrus imapd; still, I don't think unit
tests could rely on a "real" server. On the other hand, I don't think a
python.org hosted test server would be a very wise choice: even if we
found some dummy all-data-in-memory server, I fear the ssl/tls load on
the server.

For network tests like this where it is useful to test against external
servers, could we just pick few known external servers that are unlikely
to every go away?

imap.gmail.com:993 for instance? (i don't know enough about imap to
know if it supports what we need to test, I'll leave that up to you to
determine :)

Picking another host or two from other big ISPs would also be wise in
order to test multiple server implementations.

Once that is done, the tests that connect to external servers should be
put in their own file marked as requiring the network resource. Similar
to how its done in Lib/test/test_urllib2net.py.

I just found out that the gmail servers don't support connections on
the standard pop3/imapv4 ports, but only on the SSL wrapped ones.

I'm unsure if cmu.edu anonymous imap server admin's would be happy with
their server becoming python's official imap/pop testing playground;
still, I just verified it does support TLS upgrades both for pop3 and imap
and anonymous (read-only) logins.

I think I could cook-up something.

As the tests are new, I hope sending the real file is the right way to
proceed.

Why can't we use python.org for tests? Do we need IMAP/POP servers
running? Let's send some mail to pydotorg to get that set up.

I brought this up on pydotorg, and Barry suggests that someone put
together a Twisted environment which could be downloaded and run locally
on the test machine. It would provide IMAP and POP servers, perhaps
NNTP and others as well. Now, all we need is someone to make that
happen :-).

I've committed some of the remote tests in r86380, which also helped me fix a bug in login() in 3.x.

The shutdown change was committed in r86383.

Here is an updated STARTTLS patch for py3k.

The starttls patch has been committed in r86431. Thank you very much for writing the initial patch.