classification
Title: ssl makefile never closes socket
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: janssen Nosy List: JonathansCorner.com, csapuntz, dugan, giampaolo.rodola, janssen, marcin.bachry, pitrou
Priority: high Keywords: patch

Created on 2009-02-13 00:34 by dugan, last changed 2010-04-23 23:37 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
ssl.py.patch dugan, 2009-02-13 00:43 One line patch
test.py JonathansCorner.com, 2009-05-11 20:47
test-simple.py marcin.bachry, 2009-05-11 23:45
Messages (10)
msg81842 - (view) Author: David Christian (dugan) Date: 2009-02-13 00:34
The ssl.py makefile function returns a socket._fileobject object with a
reference to itself, and also increments the makefile_refs variable.

However, the _fileobject is created with the parameter close=False,
which means that when you call _fileobject.close, it does not call close
on the ssl socket!  

>>> import socket, ssl
>>> s = socket.create_connection(('www.rpath.com', 443))
>>> sslSocket = ssl.wrap_socket(s)
>>> f1 = sslSocket.makefile()
>>> f2 = sslSocket.makefile()
>>> f3 = sslSocket.makefile()
>>> sslSocket._makefile_refs
3
>>> sslSocket._sock
<socket object, fd=3, family=2, type=1, protocol=6>
>>> sslSocket.close()
>>> f1.close()
>>> f2.close()
>>> f3.close()
>>> sslSocket._makefile_refs
2

The quick fix is to add close=True on the _fileobject call in ssl.py. 
Note that this close=True is _not_ needed in the socket.py makefile call
as that makefile does not do reference counting.
msg82980 - (view) Author: Bill Janssen (janssen) * (Python committer) Date: 2009-03-01 20:13
I'd recommend running the whole suite of tests here.  The issue is
mainly with httplib, as I recall it, which "closes" the socket before it
finishes reading from it.
msg83000 - (view) Author: David Christian (dugan) Date: 2009-03-02 05:19
I actually discovered this issue when using httplib over ssl. Closing
the httplib connection was not closing the socket - the socket would
only be closed after garbage collection, due to this bug.  That's what
caused me to investigate and find this flaw.

I ran the regression tests and didn't run into any issues.
msg86850 - (view) Author: Constantine Sapuntzakis (csapuntz) Date: 2009-04-30 18:34
I ran into this problem when trying to use wrapsocket with httplib.py
and came up with the same fix.

The problem turns out to be even simpler than a ref counting issue.

In the current tree, the _fileobject constructor is called without the
close = True argument, As a result, _fileobject._close gets set to False
and _fileobject.close() method never propagates the close to
SSLSocket.close(). See line 269 of socket.py.
msg87466 - (view) Author: Jonathan Hayward (JonathansCorner.com) Date: 2009-05-08 19:51
Is there a workaround to close a TLS socket and its underlying socket?

I was making something to use https for a simple operation, and it the
browser acted as if the socket never closed. If I followed the close of
the ssl socket by a close of the underlying socket, I didn't get errors,
but the browser throbber acted as if the connection was still open.

Jonathan, http://JonathansCorner.com/
msg87468 - (view) Author: Constantine Sapuntzakis (csapuntz) Date: 2009-05-08 20:30
Here is the workaround I'm using until the code gets fixed:

import ssl

# Work around python bug #5328
def SSLSocket_makefile_fixed(self, mode='r', bufsize=-1):
    from socket import _fileobject

    self._makefile_refs += 1
    return _fileobject(self, mode, bufsize, True)

ssl.SSLSocket.makefile = SSLSocket_makefile_fixed

An alternate way to fix it is to reach in to the _fileobject wrapper and
close the underlying
implementation:

In the do_GET() method of my web server I called:

self.rfile._sock.close()
self.wfile._sock.close()

-Costa

On Fri, May 8, 2009 at 12:51 PM, Jonathan Hayward <report@bugs.python.org>wrote:

>
> Jonathan Hayward <jonathan.hayward@pobox.com> added the comment:
>
> Is there a workaround to close a TLS socket and its underlying socket?
>
> I was making something to use https for a simple operation, and it the
> browser acted as if the socket never closed. If I followed the close of
> the ssl socket by a close of the underlying socket, I didn't get errors,
> but the browser throbber acted as if the connection was still open.
>
> Jonathan, http://JonathansCorner.com/
>
> ----------
> nosy: +JonathansCorner.com
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue5238>
> _______________________________________
>
msg87545 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-10 20:56
Bill, can you check if it's a real bug? Unclosed sockets are pretty bad.
msg87585 - (view) Author: Jonathan Hayward (JonathansCorner.com) Date: 2009-05-11 20:47
Constantine Sapuntzakis wrote:
> import ssl
> 
> # Work around python bug #5328
> def SSLSocket_makefile_fixed(self, mode='r', bufsize=-1):
>     from socket import _fileobject
> 
>     self._makefile_refs += 1
>     return _fileobject(self, mode, bufsize, True)
> 
> ssl.SSLSocket.makefile = SSLSocket_makefile_fixed

Is it possible this workaround has a bug?

In my production code the socket remains open both after this
monkeypatch and after manually closing the underlying socket. In the
attached test case (TLS certificate and keyfile referenced but not
included), a server does the following:

1: listens to a single HTTPS request on port 8443.
2: Serves a "Hello, world!" page.
3: Closes the connection.
4: Sleeps for five seconds.
5: Exits the process.

The server incorporates the quoted patch, and the behavior from within
Firefox is that it serves up a "Hello world!" page via
https://localhost:8443/ and the connection remains open for five seconds
until the server process exits, apparently indicating a connection that
remains open as long as the process is running.

Closing the underlying connection manually seems to work.
msg87596 - (view) Author: Marcin Bachry (marcin.bachry) Date: 2009-05-11 23:45
I think there are two bugs spotted here. The first is the mentioned
missing "close" param to _fileobject in makefile() method.

The second one was noticed by Jonathan. He doesn't use makefile() at
all, yet his socket isn't closed either. Here's what I think is going on:

The raw filesystem socket seems to be closed only upon garbage
collection, when reference count of socket._realsocket drops to zero
(ie. it's not closed explicitly anywhere). _realsocket is wrapped by
socket._socketobject and all its close() method does is dropping
reference to _realsocket object it wraps.

The bug goes like this:
1. plain = socket.create_connection(...)
   it creates and wraps a native _realsocket (plain._sock) with
reference count = 1; "plain" is an instance of _socketobject
2. secure = ssl.wrap_socket(plain)
   because SSLSocket inherits from _socketobject, it also wraps
_realsocket plain._sock and the reference count is 2 now
3. secure.close()
   it calls in turn it's parent method close() which simply drops raw
_realsocket reference count to 1
4. native socket is still referenced by plain._sock and filesystem
descriptor isn't closed unless plain.close() is called too and refcount
drops to zero

I attach the simplest possible test to prove the bug.
msg104062 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-23 23:37
The makefile issue is fixed in r80428 (trunk) and r80431 (2.6). Also ported the additional test to py3k and 3.1.

The other issue pointed out by Marcin Bachry doesn't seem fixable without breaking backwards compatibility, for applications which close() the SSL object but expect the underlying socket to still be usable for clear-text communications. Therefore I prefer to close this issue.
History
Date User Action Args
2010-04-23 23:37:40pitrousetstatus: open -> closed
resolution: fixed
messages: + msg104062

stage: resolved
2010-04-20 20:46:50pitrousetnosy: + giampaolo.rodola

versions: + Python 3.2, - Python 3.0
2010-04-20 20:45:55pitroulinkissue7927 superseder
2009-05-11 23:45:09marcin.bachrysetfiles: + test-simple.py
nosy: + marcin.bachry
messages: + msg87596

2009-05-11 20:47:14JonathansCorner.comsetfiles: + test.py

messages: + msg87585
2009-05-10 20:56:10pitrousetpriority: high
versions: + Python 3.0, Python 3.1, Python 2.7
nosy: + pitrou

messages: + msg87545
2009-05-10 20:55:42pitrousetfiles: - unnamed
2009-05-08 20:30:09csapuntzsetfiles: + unnamed

messages: + msg87468
2009-05-08 19:51:22JonathansCorner.comsetnosy: + JonathansCorner.com
messages: + msg87466
2009-04-30 18:34:36csapuntzsetnosy: + csapuntz
messages: + msg86850
2009-03-02 05:19:58dugansetmessages: + msg83000
2009-03-01 20:13:33janssensetmessages: + msg82980
2009-02-28 18:14:06benjamin.petersonsetassignee: janssen
nosy: + janssen
2009-02-13 00:43:39dugansetfiles: + ssl.py.patch
keywords: + patch
type: resource usage
components: + Library (Lib)
2009-02-13 00:34:37dugancreate