classification
Title: HTTPResponse.close() should consume all remaining data in body if any
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: Jacky, iritkatriel, martin.panter
Priority: normal Keywords:

Created on 2016-01-28 10:43 by Jacky, last changed 2021-06-18 13:38 by iritkatriel. This issue is now closed.

Files
File name Uploaded Description Edit
zzz.py Jacky, 2016-01-28 10:43 A http server and a http client for reproducing the issue
Messages (4)
msg259122 - (view) Author: Jacky (Jacky) Date: 2016-01-28 10:43
HTTPResponse.close() in httplib should consume all remaining data in body if any. Or the followed request would get the unread content from the previous request, which result in the wrong result.

The following code shown that the second request will get a wrong status code if calling HTTPResponse.close() on the first response.

The whole code consists of a HTTP server and a client. The server will always return 403 for testing.


def client():
    conn = httplib.HTTPConnection(HOST, PORT)
    conn.request('GET', PATH, None, headers={})
    rsp = conn.getresponse()
    print rsp.status
    rsp.close()  # close response

    conn.request('GET', PATH, None, headers={})
    rsp2 = conn.getresponse()
    print rsp2.status  # --> should be 403

The full version see the attached file (The server used Tornado)
msg259124 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-28 10:59
The documentation already says you have to read the whole response before sending a new request: <https://docs.python.org/2/library/httplib.html#httplib.HTTPConnection.getresponse>. So I think this is more like a new feature than a bug fix, and would have to go into 3.6.

But I am not sure it is a good idea. What if the response is a large multi-megabyte download? In that case, it might make more sense to shut down the connection and start a new one.

It might be possible to add extra error checking to prevent reading a second response until the first is complete. I am not sure if that is practical or worthwhile though.
msg259127 - (view) Author: Jacky (Jacky) Date: 2016-01-28 13:19
In my opinion, HTTPResponse.close() should do really close work. Not only releasing the underlying file obj but also need to consume the remaining data to make sure the request complete.

If close() does not consume the remaining data, the user would have to do it after invoking close(), regardless of how large the data is. But how do do it, the HTTPResponse has been closed. So we have to use HTTPConnection.sock to do read work. However, this looks somewhat weird.
msg396064 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-06-18 13:38
> If close() does not consume the remaining data, the user would have to do it after invoking close(), regardless of how large the data is.


Martin's point was that you can abandon the connection and create a new one if you choose not to consume all the data.

>  But how do do it, the HTTPResponse has been closed. 

Right, if you want to consume the data you should do it before calling close().
History
Date User Action Args
2021-06-18 13:38:09iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg396064

resolution: rejected
stage: resolved
2016-01-28 13:19:55Jackysetmessages: + msg259127
2016-01-28 10:59:57martin.pantersetnosy: + martin.panter
messages: + msg259124
2016-01-28 10:43:17Jackycreate