The enclosed patch does three things:

1. enables SMTP_SSL working: the _get_socket method was setting
   self.sock instead of returning the socket to the caller, which
   did reset self.sock to None
2. replace home-grown SSLFakeFile() with calls to ssl.socket's makefile()
   calls both in the starttls and in the SMTP_SSL cases
3. shutdown sockets before closing them, to avoid server-side piling and
   connection refused on connection-limited servers
   The last change is just a cosmetical refactoring, but it really helps
   the SMTP_SSL case: default_port should really be a class attribute,
   instead of being set at __init__ time.

I've reworked the patch into a series, like haypo requested for
poplib and imaplib.

With the closure of 4066 all the tests in the test patch pass, so I'm
lowering the piority of this ticket.

I haven't reviewed the other patches, but the tests in the test patch
appear to be doing tests in the setup method, which doesn't seem like a
good idea.

I still have to apply Catucci's patch (or a modification of) after every Ubuntu installation or upgrade. Otherwise, I get

...
File "/usr/lib/python2.7/smtplib.py", line 752, in __init__
SMTP.__init__(self, host, port, local_hostname, timeout)
File "/usr/lib/python2.7/smtplib.py", line 239, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib/python2.7/smtplib.py", line 295, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python2.7/smtplib.py", line 757, in _get_socket
new_socket = socket.create_connection((host, port), timeout)
File "/usr/lib/python2.7/socket.py", line 571, in create_connection
raise err
socket.error: [Errno 111] Connection refused

But isn't it that bpo-4066 should have solved the critical part this issue pair 4066/4470?

Torsten, can you provide a clear, failing unittest for this?

No, I don't know how to do that. All I can provide is a minimal version of my code that triggers the above mentioned traceback. It is:

import smtplib

s = smtplib.SMTP_SSL("relay-auth.rwth-aachen.de")
s.login("***", "***")
s.sendmail("bronger@physik.rwth-aachen.de", "bronger.randys@googlemail.com"], "Hello")

I hope it helps to understand what I mean.

Sorry, it must be:

import smtplib

s = smtplib.SMTP_SSL("relay-auth.rwth-aachen.de")
s.login("***", "***")
s.sendmail("bronger@physik.rwth-aachen.de", ["bronger.randys@googlemail.com"], "Hello")

(A bracket was missing.)

According to your traceback you should be seeing the error in the first line (the creation of the SMTP_SSL object). If I run that line at the python prompt of python2.7.1, I get your connection failure. If I run it using 2.7 tip (or 3.3 tip), the connection succeeds. I know there have been other fixes in this area, but I don't know which one makes the difference.

Torsten, can you test with 2.7.2 and/or 2.7 tip?

I'm not sure what is left to do in this issue. Do you have an opinion, Lorenzo?

On Fri, 17 Jun 2011, R. David Murray wrote:

RDM>
RDM> R. David Murray <rdmurray@bitdance.com> added the comment:
RDM>
RDM> According to your traceback you should be seeing the error in the
RDM> first line (the creation of the SMTP_SSL object). If I run that line
RDM> at the python prompt of python2.7.1, I get your connection failure.
RDM> If I run it using 2.7 tip (or 3.3 tip), the connection succeeds. I
RDM> know there have been other fixes in this area, but I don't know which
RDM> one makes the difference.
RDM>
RDM> Torsten, can you test with 2.7.2 and/or 2.7 tip?
RDM>
RDM> I'm not sure what is left to do in this issue. Do you have an
RDM> opinion, Lorenzo?
RDM>

Torsten, would you mind letting us know both the python and the .deb
version by running

$ python2.7 --version

and

$ dpkg -l python2.7

At least in debian's python2.7_2.7.1-8, default_port is still an instance
attribute instead of a class attribute.

If the ubuntu situation is the same, the missing patch is
smtplib_01_default_port.diff , and you could work-around the problem by
explicitly calling s = smtplib.SMTP_SSL("relay-auth.rwth-aachen.de", 465).

The patch got in with:

changeset: 69931:bcf04ced5ef1
branch: 2.7
parent: 69915:7c3a20b5943a
user: Antoine Pitrou <solipsis@pitrou.net>
date: Sat May 07 19:59:33 2011 +0200
summary: Issue bpo-11927: SMTP_SSL now uses port 465 by default as
documented. Patch by Kasun Herath.

The last hunk, which fixes LMTP is still missing.

@@ -776,8 +777,9 @@
authentication, but your mileage might vary."""

     ehlo_msg = "lhlo"
+    default_port = LMTP_PORT

-    def __init__(self, host = '', port = LMTP_PORT, local_hostname = 
None):
+    def __init__(self, host = '', port = 0, local_hostname = None):
         """Initialize a new instance."""
         SMTP.__init__(self, host, port, local_hostname)

Have a nice week-end, yours

lorenzo

+-------------------------+----------------------------------------------+
| Lorenzo M. Catucci | Centro di Calcolo e Documentazione |
| catucci@ccd.uniroma2.it | Università degli Studi di Roma "Tor Vergata" |
| | Via O. Raimondo 18 ** I-00173 ROMA ** ITALY |
| Tel. +39 06 7259 2255 | Fax. +39 06 7259 2125 |
+-------------------------+----------------------------------------------+

Most of the problems in this issue were solved already so it could almost be closed:

• patch 1 was addressed in bpo-11927
• patch 2 was addressed in bpo-4066
• patches 3 and 4 were addressed in bpo-11893

Torsten's problem was addressed by bcf04ced5ef1.

I'm not sure what is left to do in this issue.

The only patch remaining is patch 5. I attached an updated version against tip of default branch. My patch mimics shutdown in imaplib.py in that it silences ENOTCONN. However I don't have a test that fails without the shutdown and I don't know if checking ENOTCONN is really needed. I tried to get shutdown to produce ENOTCONN by using Postfix as a server with smtpd_timeout=5s, connecting to it and waiting idle for more than 5 seconds before doing close(). In the Postfix logs I see that Postfix disconnects after 5 seconds of inactivity but doing shutdown afterwards doesn't trigger any exception so the ENOTCONN part remains unexercised.

My patch also adds shutdown method and SHUT_RDWR constant to mock_socket.py since otherwise test_smtplib fails.

(Added Antoine to nosy because he reviewed the patches for bpo-11927 and bpo-11893)

My Python version is "Python 2.7.1+" and the package is called "python2.7 2.7.1-5ubuntu2" (Ubuntu Natty).

Just finished testing both 2.7 and default branches' socket close behaviour, and it seems 05 is not strictly needed.

I'd still prefer if smtplib_05_shutdown_socket_v2.patch since, this way the REMOTE socket close will be unconditionally correct, instead of being dependent on GC artifacts.

I'd still prefer if smtplib_05_shutdown_socket_v2.patch could get in,
^^^^^^^^^^^^^^
since, this way the REMOTE socket close will be unconditionally correct,
instead of being dependent on GC artifacts.

For five Ubuntu releases now, I apply this patch. In contrast to Catalin's statement, it is not solved for me unless the upstream changes of two years ago haven't yet found their way into Ubuntu. I'm currently using Python 2.7.4 on Ubuntu 13.04. That said, the patch solves my issue every time.

When you say "I apply this patch", you mean smtplib_05_shutdown_socket_v2.patch, right?

In any case, I'm growing wary of bugfix regressions right now, so I would only apply the patch on the default branch.

Sorry, after having had another look at it, I realised that I have a different SSMTP issue now, non-Python-related. So for me, Ubuntu 13.04 indeed solves my old issue.

Lorenzo, any chance you could supply a unit test that fails without smtplib_05_shutdown_socket.diff applied?

On Sun, 19 May 2013, R. David Murray wrote:

RDM>
RDM> R. David Murray added the comment:
RDM>
RDM> Lorenzo, any chance you could supply a unit test that fails without
RDM> smtplib_05_shutdown_socket.diff applied?
RDM>

It would really be a functional test, since the problem with half-open
connection pile-up stems from smtp server's access control rules.

To test we should setup a fake smtp server, which forbids having multiple
connections from the same IP address, and connect twice in a row to the
fake server. I'm not sure I'm able to implement both an smtpd.py server
serving more than one connection and the client code in a race-free way in
the same "unit" test. Will try in the next week.

Thank you very much,

lorenzo

The bug is 8 years old and hasn't seen activity for three years. Is SMTP over SSL still broken for you?

This issue has been in pending status for over 18 months following Christian's question to the OP, so closing as out of date.