classification
Title: add socket independent timeout to httplib/http.client read
Type: enhancement Stage:
Components: Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Patrick Michaud
Priority: normal Keywords:

Created on 2017-02-02 16:51 by Patrick Michaud, last changed 2021-06-22 21:34 by iritkatriel.

Messages (1)
msg286807 - (view) Author: Patrick Michaud (Patrick Michaud) Date: 2017-02-02 16:51
Using python's httplib, there is no way to set a timeout for the full request cycle.  A timeout can be given, which will apply to the connection attempt and each blocking socket operation.  However, a malicious (or poorly performing) server can keep a connection open for an indefinite amount of time by slowly sending data.  Consider this server process:

https://gist.github.com/vegitron/bc883ddc88fe9253adc3e0bccea6445e

and this client:

https://gist.github.com/vegitron/4ee269b6492ff80d350e108363689d5c

With a timeout of 0.5, the client takes 0.501363039017 seconds.  With a timeout of 2.5, it takes 10.0041370392 seconds.

This is explained in the documentation, but it's a problem.  A commonly suggested solution is to use SIGALRM to set a timeout, but that doesn't work in a multi-threaded environment.  Moving to multi-process introduces big slow downs as I can't use connection pools, and data needs to be serialized and deserialized for the parent process.

I would like to propose an addition to httplib that would add a hook to httplib.HTTPResponse._read_chunked (python 2) and http.client.HTTPResponse._readall_chunked (python 3) that would either:

1) support an overall read timeout, by tracking a per-response start time and raising a timeout exception if that chunk read finishes after the given timeout
2) support a per-chunk callback function, so a client of httplib/http.client could define that logic for themselves.


Current possible timeouts, and where they can happen:

  connect  | read chunk | read chunk | read chunk
[ timeout ] [ timeout ]  [ timeout ]  [ timeout ]

Proposed addition:

  connect  | read chunk | read chunk | read chunk
[ timeout ] [ timeout ]  [ timeout ]  [ timeout ]
            [        total read time out        ]
History
Date User Action Args
2021-06-22 21:34:27iritkatrielsetversions: + Python 3.11, - Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7
2017-02-02 16:51:34Patrick Michaudcreate