classification
Title: Client support for HTTP 1.1 persistent connections throughout the standard library
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: amaury.forgeotdarc, ipatrol, loewis, martin.panter, orsenthil, pitrou, trent, vklogin
Priority: normal Keywords:

Created on 2010-09-02 11:56 by ipatrol, last changed 2015-04-11 02:53 by martin.panter.

Messages (11)
msg115365 - (view) Author: (ipatrol) Date: 2010-09-02 11:56
HTTP 1.1 introduced persistent connections nearly six years ago. Yet this resource saving and speed improving option is not available in the standard library. Can this be added?
msg115367 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-09-02 12:08
It seems that httplib is exactly what you need:
http://docs.python.org/library/httplib.html#examples
msg115368 - (view) Author: (ipatrol) Date: 2010-09-02 12:18
No, httplib actually creates a second connection with the same object. Neither is their support in urllib, urllib2, nor in any of the HTTP servers. This would be really useful for a bot connected to an API.
msg115369 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-02 12:24
Do you think you could provide a patch?
msg116641 - (view) Author: (ipatrol) Date: 2010-09-16 23:48
Possibly, but I don't really have expertise in the underbelly of the HTTP system.
msg201875 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2013-11-01 03:39
I wrote a basic “urllib.request” handler class that I have been using for HTTP persistent connections. It is called PersistentConnectionHandler; see

https://github.com/vadmium/python-iview/blob/80dc1b4/iview/hds.py#L442

I am happy for this to be used as the basis for a patch for Python, or elsewhere.

It manages a single connection. You pass an instance to urllib.request.build_opener(), and then it connects to whatever host is specified when open() is called. Any old connection is closed when you ask for a URL on a new host.
msg221299 - (view) Author: V. Kumar (vklogin) Date: 2014-06-22 19:42
Verified that Persistent Connections per HTTP 1.1 spec is already implemented and working correctly.

See:
http://hg.python.org/cpython/file/3f3de8c47ff8/Lib/http/client.py#l432

Also, tested this in packet sniffer to verify that connection is being reused.

But the server.py has in issue where the keep-alive can be used in HTTP/1.0:
http://hg.python.org/cpython/file/7417d8854f93/Lib/http/server.py#l331
So, the and in this line needs to be an 'or'.
msg221304 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-06-22 20:05
Mr. Kumar: the "and" is intentional. The server will use keep-alive messages only if it operates in HTTP/1.1 mode itself (protocol_version). By default, it operates in HTTP/1.0 mode, and does not enable persistent connections there. The initial implementation did, but it failed since it often would not send content-length headers, which are mandatory for persistent connections.
msg234609 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-01-24 12:59
See Issue 3566 about tweaking the “http.client” module’s BadStatusLine handling to be more helpful when implementing persistent connections. I am dumping some thoughts here about persistent connections with the “http.client” module, gained by working on that bug.

* Lib/xmlrpc/client.py appears to have persistent connection support, so may be useful for this bug.

* RFC 7230 §6.5 <https://tools.ietf.org/html/rfc7230#section-6.5> mentions monitoring for connection closure. This could be be partly implemented inside HTTPConnection by polling for closure before sending a request, but to fully implement might require the co-operation of the user calling into the module to check for closure at other times using select() or similar.

* Current “http.client” assumes that each socket.makefile() object will not buffer any data from a subsequent response. Unsure if this is a problem in the real world, but I ran into it implementing test cases. E.g. if the server anticipates the first few bytes of the subsequent response:

c.send(b"HTTP/1.1 200 Okay\r\nContent-Length: 0\r\n\r\n" b"HTTP/")

then the client misses the “HTTP/” and raises BadStatusLine("1.1 200 Okay\r\n").
msg235999 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-02-14 22:21
Opened Issue 23377 about losing the buffer at the end of a response
msg240459 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-11 02:53
Tweaking the title to exclude servers. Persistent connections have apparently been supported in the low-level server for ages, since Issue 430706, and now that the close_connection flag is documented (Issue 23410), someone implementing a server should be able to use that to implement HTTP 1.0 keep-alive connections, etc as well.

For the client aspect, this bug is rather vague about exactly what should be added. Issue 3566 has been committed and closed, meaning now we should have an easier way to detect when a persistent connection has been dropped by the server. Here is a list of other things that come to mind:

1. Allow a way of polling for an unsolicited disconnection or 408 response without sending a request, either by:
  * exposing the sock attribute or adding a fileno() method, so that select() can be called, or
  * adding a new poll_response() or similar method
2. Poll if disconnected before sending a request, to avoid in 99% of cases the problem with non-idempotent requests (like POST) where you do not know if they were accepted or not
3. Change urlopen() to reuse the same connection when retrying a request (e.g. for redirects, authentication)
4. A urllib.request.PersistentConnectionHandler class or other urlopen() level API where the caller can reuse the same connection for multiple requests to the same host. Like HTTPConnection mainly does, but with the extra redirect, error, etc handling of urlopen().

I think the first point is important to do, because I have had to work around this by relying on the undocumented “HTTPConnection.sock” attribute. Once point 1 is done, point 2 might be easy to do in user code.

The last two points I currently don’t have so much personal interest in seeing in the standard library, unless others also want something like them.
History
Date User Action Args
2015-04-11 02:53:59martin.pantersetmessages: + msg240459
title: Support for HTTP 1.1 persistent connections throughout the standard library -> Client support for HTTP 1.1 persistent connections throughout the standard library
2015-02-14 22:21:55martin.pantersetmessages: + msg235999
2015-01-24 12:59:30martin.pantersetmessages: + msg234609
2014-06-22 20:05:33loewissetnosy: + loewis
messages: + msg221304
2014-06-22 19:42:27vkloginsetnosy: + trent, vklogin
messages: + msg221299
2013-11-01 03:39:36martin.pantersetmessages: + msg201875
2013-06-18 06:47:41martin.pantersetnosy: + martin.panter
2011-03-20 13:44:07merwoksetnosy: amaury.forgeotdarc, orsenthil, pitrou, ipatrol
versions: + Python 3.3, - Python 3.2
2011-03-19 22:32:51orsenthilsetassignee: orsenthil
nosy: amaury.forgeotdarc, orsenthil, pitrou, ipatrol
2010-09-16 23:48:10ipatrolsetmessages: + msg116641
2010-09-02 21:13:25merwoksetversions: - Python 3.1, Python 2.7
2010-09-02 12:24:21pitrousetversions: - Python 3.3
nosy: + orsenthil, pitrou

messages: + msg115369

stage: needs patch
2010-09-02 12:18:07ipatrolsetstatus: pending -> open
resolution: works for me ->
messages: + msg115368
2010-09-02 12:08:52amaury.forgeotdarcsetstatus: open -> pending

nosy: + amaury.forgeotdarc
messages: + msg115367

resolution: works for me
2010-09-02 11:56:50ipatrolcreate