Hi there,

Recent urlopen with timeout did not work. Below is the back trace. After digging into the Python lib, the root cause is found - within the socket.py, self._sock.recv(), under a 'while True' loop, tried to retrieve sth from the under-layer SSL socket. However, the under-layer PySSL_SSLread() neither returned anything useful nor reported an exception, such as timeout. Because of this 'while True' loop, urlopen got stuck. My temp fix is to add timeout within this issued loop of the socket.py. However, there are other similar loops ranging from urllib2, httplib to socket...and I do not think anything wrong with these loops considering the system socket programming practice. Instead, I am thinking if we could add a new timeout value into the socket object. This new timeout starts with the same value as the user-defined timeout but decrease for the same socket operation. For the case I have encountered:

while True:
    some_sock_recv() # decrease the timeout value within the function call

This should guarantee the timeout timely even within the 'while True' loop.


Thanks,
Dave

http://davejingtian.org




>>> obj=urllib2.urlopen(url, timeout=20)
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 441, in begin
    self.msg = HTTPMessage(self.fp, 0)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/mimetools.py", line 25, in __init__
    rfc822.Message.__init__(self, fp, seekable)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/rfc822.py", line 108, in __init__
    self.readheaders()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 280, in readheaders
    line = self.fp.readline(_MAXLINE + 1)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
KeyboardInterrupt
>>>

It sounds like the bug is that PySSL_SSLread didn't raise the timeout?  Any idea if this is still a problem in python3?  (Could possibly have changed on trunk as well, as SSL is being updated in 2.7.9.)

Hi David,

Thanks for your quick response. I have tried Python 3.4.2 using urllib.request.urlopen() - still not working. Below is the backtrace. I am not sure if this is a bug of PySSL_SSLread, which returns nothing yet without timeout. If you want me to dig into this deeper, just let me know.

Thanks,

Dave Tian
dave.jing.tian@gmail.com

>>> obj=urllib.request.urlopen(url, timeout=20)
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 153, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 461, in open
    response = meth(req, response)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 571, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 493, in error
    result = self._call_chain(*args)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 676, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 455, in open
    response = self._open(req, data)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 473, in _open
    '_open', req)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 1217, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Users/daveti/Python-3.4.2/Lib/urllib/request.py", line 1177, in do_open
    r = h.getresponse()
  File "/Users/daveti/Python-3.4.2/Lib/http/client.py", line 1172, in getresponse
    response.begin()
  File "/Users/daveti/Python-3.4.2/Lib/http/client.py", line 375, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "/Users/daveti/Python-3.4.2/Lib/http/client.py", line 261, in parse_headers
    line = fp.readline(_MAXLINE + 1)
  File "/Users/daveti/Python-3.4.2/Lib/socket.py", line 371, in readinto
    return self._sock.recv_into(b)
  File "/Users/daveti/Python-3.4.2/Lib/ssl.py", line 746, in recv_into
    return self.read(nbytes, buffer)
  File "/Users/daveti/Python-3.4.2/Lib/ssl.py", line 618, in read
    v = self._sslobj.read(len, buffer)
KeyboardInterrupt
>>> 

> On Nov 15, 2014, at 3:36 AM, R. David Murray <report@bugs.python.org> wrote:
> 
> 
> R. David Murray added the comment:
> 
> It sounds like the bug is that PySSL_SSLread didn't raise the timeout?  Any idea if this is still a problem in python3?  (Could possibly have changed on trunk as well, as SSL is being updated in 2.7.9.)
> 
> ----------
> nosy: +r.david.murray
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22870>
> _______________________________________

I won't be the one, as I'm not conversant with the ssl C code.  What would be helpful right now would be a recipe for reproducing the problem.

Alright. The issued URL of my case is here: www.5giay.vn
Nor am I a ssl dev...Here is what happens after further debugging: PySSL_SSLread() returns 1/2 bytes without any error/timeout per call. readline() in socket.py keeps looping. Occasionally, it may break out because of a newline recv’d. However, readheaders() in httplib.py repeats the readline() again because of another while True loop. I have no idea why I’ve got only 1/2 bytes recv’d from SSLread(). I am using my Mac OS for testing. Hope this is clear for further investigation if anyone would like to dig in.

Dave Tian
dave.jing.tian@gmail.com

> On Nov 16, 2014, at 2:38 AM, R. David Murray <report@bugs.python.org> wrote:
> 
> 
> R. David Murray added the comment:
> 
> I won't be the one, as I'm not conversant with the ssl C code.  What would be helpful right now would be a recipe for reproducing the problem.
> 
> ----------
> nosy: +alex, pitrou
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22870>
> _______________________________________

Is this bug still relevant for you?