classification
Title: httplib/ftplib: timeout parameter not applied correctly
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 3.3, Python 2.7, Python 2.6
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asandvig, giampaolo.rodola
Priority: normal Keywords:

Created on 2010-08-19 11:29 by asandvig, last changed 2010-08-21 19:47 by giampaolo.rodola. This issue is now closed.

Messages (3)
msg114343 - (view) Author: Anders Sandvig (asandvig) Date: 2010-08-19 11:29
From <http://mail.python.org/pipermail/python-dev/2010-July/101266.html>:

Consider the following code for retreieving a web page using httplib:

   def get_url(hostname, port, url, timeout=5):
       con = httplib.HTTPConnection(hostname, port, timeout)
       con.request("GET", url)
       res = con.getresponse()
       data = res.read()
       return res, data

As expected, this will raise a socket.error if the client is unable to
connect before the timeout has expired. However, once the connection
has been made, the timeout parameter no longer has any effect and
con.getresponse() will block forever if the server does not send any
data.

I think the reason for this is that the socket object created in
HTTPConnection.connect() has a default timeout of 0 (i.e. it is never
set explicitly):

   def connect(self):
       """Connect to the host and port specified in __init__."""
       self.sock = socket.create_connection((self.host,self.port),
                                            self.timeout)


For now I have been able to work around this by manually setting the
timeout of the (private) socket object after calling con.request(),
like this:

       ...
       con.request("GET", url)
       con.sock.settimeout(timeout)
       res = con.getresponse()
       ...

However, I don't think this is a very good solution as it relies on
knowledge about the inner workings of httplib (and I had to read the
library source code to come up with it).

>From the top of my head, I can come up with three (four) ways of
properly solving the issue:

1) Documenting the timeout behavior and describing the above hack in
the httplib documentation.

2) Modify HTTPConnection.connect() to set the timeout on the socket
object after it has been  created (using the same timeout as given on
the HTTPConnection constructor).

3) Adding (optional) timeout parameters to
HTTPConnection.getresponse() and HTTPResponse.read() (and possibly
other functions with the same problem).

4) A combination of 2) and 3).

[...]
msg114347 - (view) Author: Anders Sandvig (asandvig) Date: 2010-08-19 11:46
The best (and simplest) solution seems to be option 2).

Affected methods are found to be HTTPConnection.connect() and
HTTPSConnection.connect() in Lib/httplib.py (Lib/http/client.py for 3.x) and
FTP.connect() and FTP.ntransfercmd() in Lib/ftplib.py.

It appears the issue can be fixed by simply adding a call to settimeout() on
socket objects returned by socket.create_connection(), but this should of course
be verified by appropriate tests.
msg114363 - (view) Author: Anders Sandvig (asandvig) Date: 2010-08-19 13:52
socket.create_connection() does in fact set the timeout of the resulting socket object, so the issue is not an issue after all.

The problems I experienced was a result of sending the timeout as the third parameter to the HTTPConnection() constructor instead of a named parameter, i.e.:

  con = httplib.HTTPConnection(hostname, port, timeout)

should have been:

  con = httplib.HTTPConnection(hostname, port, timeout=timeout)
History
Date User Action Args
2010-08-21 19:47:21giampaolo.rodolasetnosy: + giampaolo.rodola
2010-08-19 13:52:24asandvigsetstatus: open -> closed

messages: + msg114363
2010-08-19 11:46:37asandvigsetmessages: + msg114347
2010-08-19 11:29:28asandvigcreate