Author Lukasa
Recipients Lukasa
Date 2015-03-03.16:56:12
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1425401773.41.0.539386605796.issue23576@psf.upfronthosting.co.za>
In-reply-to
Content
Initially reported on the requests bug list at https://github.com/kennethreitz/requests/issues/2467

In cases when a remote web server sends a non-chunked response that does not have a content length, it is possible to get http.client to hang on a read. To hit it requires a specific set of circumstances:

- Python 3 (this bug is not seen on Python 2 in my testing)
- HTTPS (using plaintext HTTP resolves the problem)
- A socket timeout must be set (leaving it unset, i.e. leaving the socket in blocking mode, resolves this problem)
- The server must not send a content-length or set Transfer-Coding: chunked.
- The reads must not be an even divisor of the content's length.
- The timeout must be longer than the time the server is willing to keep the connection open (otherwise an exception is thrown).

The following code can be used as a sample to demonstrate the bug:

import http.client
import time

def test(connection_class=http.client.HTTPSConnection, timeout=10, read_size=7):
    start = time.time()
    c = connection_class('sleepy-reaches-6892.herokuapp.com')
    c.connect()
    if timeout is not None:
        c.sock.settimeout(timeout)
    c.request('GET', '/test')
    r = c.getresponse()
    while True:
        data = r.read(read_size)
        if not data:
            break
    print('Finished in {}'.format(time.time() - start))


Below are the results from several different runs:

test(): Finished in 4.8294830322265625
test(connection_class=http.client.HTTPConnection): Finished in 0.3060309886932373
test(timeout=None): Finished in 0.6070599555969238
test(read_size=2): Finished in 0.6600658893585205

As you can see, only this particular set of features causes the bug. Note that if you change the URL to one that does return a Content-Length (e.g. http2bin.org/get), this bug also goes away.

This is a really weird edge case bug. There's some extensive investigation over at the requests bug report, but I've not been able to convincingly point at the problem yet.
History
Date User Action Args
2015-03-03 16:56:13Lukasasetrecipients: + Lukasa
2015-03-03 16:56:13Lukasasetmessageid: <1425401773.41.0.539386605796.issue23576@psf.upfronthosting.co.za>
2015-03-03 16:56:13Lukasalinkissue23576 messages
2015-03-03 16:56:12Lukasacreate