When running urllib2 and getting a BadStatus from an http server, this error is raised:

  File "/var/www/pinax-env/pline/apps/page/models.py", line 303, in render
    content = urllib2.urlopen(self.url,timeout=10).read()
  File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 389, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 407, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1146, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1119, in do_open
    r = h.getresponse()
  File "/usr/lib/python2.6/httplib.py", line 974, in getresponse
    response.begin()
  File "/usr/lib/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.6/httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine

Because urllib2 doesn't catch it with this:

lines 1116-1120

try:
    h.request(req.get_method(), req.get_selector(), req.data, headers)
    r = h.getresponse()
except socket.error, err: # XXX what error?
    raise URLError(err)

It is not caught anywhere else and the call blows up unless I make a special exception for all httplib exceptions.  The specific url that blew this up is:

http://phoenix.untd.com/oasx/rqst/type=sx/rdb=8203014d740000555355533a415a2d2d2d2d2d2d2d2d2d2d0100001d494a0901000000000000/version=3/origin=uol/isp=et_cau

urllib2 is currently catching the socket.error exceptions and presenting the reason as custom URLError exception. To address this issue, the module should catch the httplib raised exceptions also and present it.

I've made a small change to urllib2 to catch the httplib.BadStatusLine and raise as URLError. This exception should rarely happen as it means the server is returning invalid responses. Nevertheless, I've added a test and hopefully fixed the issue.

Patch is attached. I will check if this needs to be added to newer versions of Python.

By the way, the issue can be recreated by running the following:

netcat -l -p 9999 -e "echo HTTP/1.1 1000 OK" &
python -c "import urllib2; urllib2.urlopen('http://localhost:9999')"

This happens on 2.6, 2.7 and 3 by the looks of it.

See also Issue 22797, about documenting that non-URLError exceptions may be raised.

Issue 13736 also proposes to wrap HTTP client exceptions, although I personally don’t really endorse this.

The other option is to fix the documentation: Issue 25633, Issue 22797.