HTTP 1.1 introduced persistent connections nearly six years ago. Yet this resource saving and speed improving option is not available in the standard library. Can this be added?

It seems that httplib is exactly what you need:
http://docs.python.org/library/httplib.html#examples

No, httplib actually creates a second connection with the same object. Neither is their support in urllib, urllib2, nor in any of the HTTP servers. This would be really useful for a bot connected to an API.

Do you think you could provide a patch?

Possibly, but I don't really have expertise in the underbelly of the HTTP system.

I wrote a basic “urllib.request” handler class that I have been using for HTTP persistent connections. It is called PersistentConnectionHandler; see

https://github.com/vadmium/python-iview/blob/80dc1b4/iview/hds.py#L442

I am happy for this to be used as the basis for a patch for Python, or elsewhere.

It manages a single connection. You pass an instance to urllib.request.build_opener(), and then it connects to whatever host is specified when open() is called. Any old connection is closed when you ask for a URL on a new host.

Verified that Persistent Connections per HTTP 1.1 spec is already implemented and working correctly.

See:
http://hg.python.org/cpython/file/3f3de8c47ff8/Lib/http/client.py#l432

Also, tested this in packet sniffer to verify that connection is being reused.

But the server.py has in issue where the keep-alive can be used in HTTP/1.0:
http://hg.python.org/cpython/file/7417d8854f93/Lib/http/server.py#l331
So, the and in this line needs to be an 'or'.

Mr. Kumar: the "and" is intentional. The server will use keep-alive messages only if it operates in HTTP/1.1 mode itself (protocol_version). By default, it operates in HTTP/1.0 mode, and does not enable persistent connections there. The initial implementation did, but it failed since it often would not send content-length headers, which are mandatory for persistent connections.

See bpo-3566 about tweaking the “http.client” module’s BadStatusLine handling to be more helpful when implementing persistent connections. I am dumping some thoughts here about persistent connections with the “http.client” module, gained by working on that bug.

• Lib/xmlrpc/client.py appears to have persistent connection support, so may be useful for this bug.

• RFC 7230 §6.5 <https://tools.ietf.org/html/rfc7230#section-6.5\> mentions monitoring for connection closure. This could be be partly implemented inside HTTPConnection by polling for closure before sending a request, but to fully implement might require the co-operation of the user calling into the module to check for closure at other times using select() or similar.

• Current “http.client” assumes that each socket.makefile() object will not buffer any data from a subsequent response. Unsure if this is a problem in the real world, but I ran into it implementing test cases. E.g. if the server anticipates the first few bytes of the subsequent response:

c.send(b"HTTP/1.1 200 Okay\r\nContent-Length: 0\r\n\r\n" b"HTTP/")

then the client misses the “HTTP/” and raises BadStatusLine("1.1 200 Okay\r\n").

Opened bpo-23377 about losing the buffer at the end of a response

Tweaking the title to exclude servers. Persistent connections have apparently been supported in the low-level server for ages, since bpo-430706, and now that the close_connection flag is documented (bpo-23410), someone implementing a server should be able to use that to implement HTTP 1.0 keep-alive connections, etc as well.

For the client aspect, this bug is rather vague about exactly what should be added. bpo-3566 has been committed and closed, meaning now we should have an easier way to detect when a persistent connection has been dropped by the server. Here is a list of other things that come to mind:

1. Allow a way of polling for an unsolicited disconnection or 408 response without sending a request, either by:

• exposing the sock attribute or adding a fileno() method, so that select() can be called, or
• adding a new poll_response() or similar method

2. Poll if disconnected before sending a request, to avoid in 99% of cases the problem with non-idempotent requests (like POST) where you do not know if they were accepted or not
3. Change urlopen() to reuse the same connection when retrying a request (e.g. for redirects, authentication)
4. A urllib.request.PersistentConnectionHandler class or other urlopen() level API where the caller can reuse the same connection for multiple requests to the same host. Like HTTPConnection mainly does, but with the extra redirect, error, etc handling of urlopen().

I think the first point is important to do, because I have had to work around this by relying on the undocumented “HTTPConnection.sock” attribute. Once point 1 is done, point 2 might be easy to do in user code.

The last two points I currently don’t have so much personal interest in seeing in the standard library, unless others also want something like them.