classification
Title: Socket is closed prematurely in httplib, if server sends response before request body has been sent
Type: Stage:
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jhylton Nosy List: ABR, guy.kloss, jasondavies, jhylton, orsenthil, vstinner
Priority: normal Keywords:

Created on 2009-03-23 09:29 by jasondavies, last changed 2012-09-27 15:50 by ABR. This issue is now closed.

Messages (9)
msg84000 - (view) Author: Jason Davies (jasondavies) Date: 2009-03-23 09:29
I came across this bug when trying to use CouchDB-Python to PUT an
attachment using HTTP authentication (using the httplib2 library). 
Essentially what httplib2 does is this:

1. Attempt PUT with no auth credentials
2. If a 401 response is received, try again with credentials

However, CouchDB sends a 401 response as soon as it has consumed the
request headers, and before the request body has been fully sent by the
client.

This triggers a bug in both httplib and httplib2, whereby they raise
exceptions due to a broken pipe being encountered when trying to finish
sending the request body.

It seems that Python's httplib checks for broken pipe errors and closes
the connection entirely when they occur when making a request.  This is
unhelpful if a legitimate response was sent and the socket was closed
early by the server.

The offending try/except handler is in httplib.HttpConnection.send and
has a comment saying:

# send the data to the server. if we get a broken pipe, then close
# the socket. we want to reconnect when somebody tries to send again.

This is wrong, as someone might want to read the response before closing
the socket.

For reference, the CouchDB-Python issue is:
http://code.google.com/p/couchdb-python/issues/detail?id=68

This bug may also be related to: http://bugs.python.org/issue3566
msg84001 - (view) Author: Jason Davies (jasondavies) Date: 2009-03-23 09:41
Note: in case I didn't make it clear, the fix is to remove the offending
try/except handler and let the broken pipe error propagate.
msg84290 - (view) Author: Jeremy Hylton (jhylton) (Python triager) Date: 2009-03-27 23:31
Wow!  Old issue.  This behavior was present in Greg's original version
of the code.
msg84298 - (view) Author: Jeremy Hylton (jhylton) (Python triager) Date: 2009-03-28 04:34
I think it makes sense to leave the socket open in this case.  (In
general, I think httplib is too aggressive about closing the socket.) 
I'm checking in a version for py3k, and will get around to backporting
it later.

Committed revision 70643.
msg111421 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-07-24 02:47
jhylton> Committed revision 70643.

I backported the fix to 2.7 (r83122). It's required to fix #9032 (XML-RPC bugĂ  in 2.7.

I don't think that it should be backported to 2.6 or 3.1, so I close the issue.
msg129247 - (view) Author: Guy Kloss (guy.kloss) Date: 2011-02-24 03:45
I know this issue is marked as fixed, and won't be backported to 2.6. But the fix is simple enough to self perform on 2.6. Doing that I have discovered an issue that might still be present with the fix, as it was not discussed here, yet, but is still related.

Whenever one does the same thing, but uses an SSL connection (HTTPS), the socket.error is raised from within ssl.py (line 174).

Even though the send() call in httplib does not misbehave now, when using an SSL encrypted connection the ssl module still throws you in front of a bus.
msg129693 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-02-28 12:24
Guy, if the issue you describe in relevant to py3.x or 2.7, can you open a new issue?
msg129731 - (view) Author: Guy Kloss (guy.kloss) Date: 2011-02-28 23:27
I would open a new issue if I had it verified. But I can't verify it right now, as I cannot install the dependencies for a Py2.7 (or 3.x) for running my failing code on the server (running Ubuntu Lucid/LTS).

Unless you're saying I should open an issue for it anyway, without having it verified on 2.7 or 3.x ...
msg171377 - (view) Author: ABR (ABR) Date: 2012-09-27 15:50
I can report that this occurs on Python 2.7.3, when using urllib[x] wrapped in 'requests'.  Trying to access a site using basic auth over https will not work with certain servers, whereas it will work over plain http.

I tried searching to see if a corresponding https issue exists, but this came up instead.  :-)
History
Date User Action Args
2012-09-27 15:50:22ABRsetnosy: + ABR
messages: + msg171377
2011-02-28 23:27:38guy.klosssetnosy: jhylton, orsenthil, vstinner, jasondavies, guy.kloss
messages: + msg129731
2011-02-28 12:24:27orsenthilsetnosy: + orsenthil
messages: + msg129693
2011-02-24 03:45:38guy.klosssetnosy: + guy.kloss
messages: + msg129247
2010-07-24 02:47:41vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg111421

resolution: accepted -> fixed
2009-03-28 04:34:34jhyltonsetresolution: accepted
messages: + msg84298
2009-03-27 23:31:16jhyltonsetmessages: + msg84290
2009-03-27 23:16:02jhyltonsetassignee: jhylton

nosy: + jhylton
2009-03-23 09:41:31jasondaviessetmessages: + msg84001
2009-03-23 09:29:09jasondaviescreate