Using python's httplib, there is no way to set a timeout for the full request cycle.  A timeout can be given, which will apply to the connection attempt and each blocking socket operation.  However, a malicious (or poorly performing) server can keep a connection open for an indefinite amount of time by slowly sending data.  Consider this server process:

https://gist.github.com/vegitron/bc883ddc88fe9253adc3e0bccea6445e

and this client:

https://gist.github.com/vegitron/4ee269b6492ff80d350e108363689d5c

With a timeout of 0.5, the client takes 0.501363039017 seconds.  With a timeout of 2.5, it takes 10.0041370392 seconds.

This is explained in the documentation, but it's a problem.  A commonly suggested solution is to use SIGALRM to set a timeout, but that doesn't work in a multi-threaded environment.  Moving to multi-process introduces big slow downs as I can't use connection pools, and data needs to be serialized and deserialized for the parent process.

I would like to propose an addition to httplib that would add a hook to httplib.HTTPResponse._read_chunked (python 2) and http.client.HTTPResponse._readall_chunked (python 3) that would either:

1) support an overall read timeout, by tracking a per-response start time and raising a timeout exception if that chunk read finishes after the given timeout
2) support a per-chunk callback function, so a client of httplib/http.client could define that logic for themselves.


Current possible timeouts, and where they can happen:

  connect  | read chunk | read chunk | read chunk
[ timeout ] [ timeout ]  [ timeout ]  [ timeout ]

Proposed addition:

  connect  | read chunk | read chunk | read chunk
[ timeout ] [ timeout ]  [ timeout ]  [ timeout ]
            [        total read time out        ]