classification
Title: http/client.py block indefinitely on line 308 in _read_status
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: Julien.Palard, martin.panter
Priority: normal Keywords:

Created on 2015-06-22 13:15 by Julien.Palard, last changed 2015-06-24 12:55 by martin.panter. This issue is now closed.

Messages (6)
msg245626 - (view) Author: Julien Palard (Julien.Palard) Date: 2015-06-22 13:15
Requesting HTTP using `requests`, which uses `http.client` which use `socket`, sometimes, my program get stuck like this:

```
File "/usr/lib/python3.2/socket.py", line 287 in readinto
File "/usr/lib/python3.2/http/client.py", line 308 in _read_status
File "/usr/lib/python3.2/http/client.py", line 346 in begin
File "/usr/lib/python3.2/http/client.py", line 1052 in getresponse
... in python requests ...
```

The line is the first of _read_status::

```
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
```

At a lower level, the program is stuck in a `recvfrom()` syscall.

So obviously, an error at the network level caused the server to not reply, but the client recvfrom() the network in a blocking socket without checking first if data is available, so it get stuck indefinitely :-(
msg245632 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-22 14:39
You did not mention if the HTTP connection has a timeout set. If no timeout is set, it will block until the server sends or closes the connection. The timeout sets how long the socket spends “checking if data is available” before giving up.
msg245669 - (view) Author: Julien Palard (Julien.Palard) Date: 2015-06-23 07:43
I only have a `socket.setdefaulttimeout(10)` just after my imports...
msg245686 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-23 12:49
The closest I have is Python 3.3, but this times out properly for me:

>>> from http.client import HTTPConnection
>>> import socket
>>> socket.setdefaulttimeout(10)
>>> h = HTTPConnection("192.168.1.84")
>>> h.request("GET", "/")
>>> h.getresponse()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python33\lib\http\client.py", line 1147, in getresponse
    response.begin()
  File "C:\Python33\lib\http\client.py", line 358, in begin
    version, status, reason = self._read_status()
  File "C:\Python33\lib\http\client.py", line 320, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Python33\lib\socket.py", line 297, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

I suggest you either try to come up with a recipe to reproduce this with plain http.client, or look into whether the Requests package supports using the default socket timeout setting.
msg245735 - (view) Author: Julien Palard (Julien.Palard) Date: 2015-06-24 12:46
OK, so, requests have a `timeout` and take it into account, and it solves my problem.

Yet I don't understand one little thing:

With both requests `timeout` parameter set or unset, the exact same http.client.py:_read_status call the same socket.readinto. With a `request` `timeout`, the socket.readinto uses the recvfrom syscall, but with a `request` `timeout`, readinto uses a `poll` syscall with the given timeout.

The root of the problem is that urllib3 ignores the `socket` `defaulttimeout`, I opened a ticket on this:

https://github.com/shazow/urllib3/issues/655#issuecomment-114835279

So this ticket can be considered closed.
msg245736 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-06-24 12:55
I assume you meant _without_ an explicit timeout setting in Requests, you see a recvfrom() system call, and _with_ an explicit timeout you see poll(). I guess that would be because Requests is using a blocking socket in the first case, and in the second case the timeout is implemented by waiting for data to be available before reading it.
History
Date User Action Args
2015-06-24 12:55:58martin.pantersetresolution: third party
messages: + msg245736
stage: resolved
2015-06-24 12:46:28Julien.Palardsetstatus: open -> closed
2015-06-24 12:46:20Julien.Palardsetmessages: + msg245735
2015-06-23 12:49:50martin.pantersetmessages: + msg245686
2015-06-23 07:43:54Julien.Palardsetmessages: + msg245669
2015-06-22 14:39:29martin.pantersettype: behavior

messages: + msg245632
nosy: + martin.panter
2015-06-22 13:15:11Julien.Palardcreate