Title: urllib2.HTTPError does not have 'reason' attribute.
Type: Stage: needs patch
Components: Documentation, Library (Lib) Versions: Python 3.4, Python 3.3, Python 3.2, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arfrever, docs@python, jason.coombs, orsenthil, petri.lehtinen, pitrou, python-dev
Priority: normal Keywords: easy, needs review, patch

Created on 2011-10-18 13:06 by jason.coombs, last changed 2012-12-09 21:53 by python-dev. This issue is now closed.

File name Uploaded Description Edit
fffeff7721c0.diff jason.coombs, 2011-11-07 16:15 review
Repositories containing patches
Messages (17)
msg145805 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-10-18 13:06
The urllib2 docs indicate that HTTPError is a subclass of URLError and that URLError has an attribute of 'reason', but HTTPError does not have this attribute. The docs should be updated to reflect this deviance.

It appears the Python 3.2 docs no longer include documentation for URLError

Python 2.7.2
>>> try:urllib2.urlopen('')
... except urllib2.HTTPError as exc:
...   print(dir(exc))
['_HTTPError__super_init', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', 'args', 'close', 'code', 'errno', 'filename', 'fileno', 'fp', 'getcode', 'geturl', 'hdrs', 'headers', 'info', 'message', 'msg', 'next', 'read', 'readline', 'readlines', 'strerror', 'url']

The same issue exists in Python 3.2.2.

Here's what I propose:
  - For Python 3.2 and 3.3, update HTTPError to supply a @property, aliasing .msg (will .msg always be a suitable .reason?).
  - For Python 2.7, document the deviance, such as by adding the following wording to the HTTPError docs: "Unlike URLError, HTTPError does not supply a reason attribute. The reason can be retrieved through the msg attribute."
msg146416 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-10-26 06:03
> It appears the Python 3.2 docs no longer include documentation for URLError

Both URLError and HTTPError are documented in 3.2 and 3.3:

It needs to be investigated whether each point raising a HTTPError sets a good msg. If this is the case, reason can be aliased to msg.

This could be done for 2.7, too, because this is a bug (deviation from what the documentation says) rather than a new feature.
msg146426 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-10-26 09:04
I scanned through the libs for Python 2.7, 3.2, and 3.3 and there is no construction of HTTPError that does not pass a string for msg. I believe it would be reasonable to alias reason to msg. I'll put together the changesets.
msg147233 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-11-07 16:17
I've created three changesets, addressing the issue in 2.7, 3.2, and 3.3, including tests. Please review and comment. If there are no objections, I'll push the changesets after 24 hours.
msg147253 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-07 19:21
Perhaps the reason should include the status code, too? It makes HTTP errors much more useful, as you'll immediately see what's going on from the status code.
msg147258 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-11-07 21:38
My initial instinct was to agree - the status code is useful. However, in looking at the FTP code, it sometimes just sets other objects (socket.error for example) as the 'reason'. The docs say 'reason' is a string or another exception.

I'm tempted to leave it as is. This fix is mainly to provide a reasonable value for .reason. The __str__ and .code are already exposed, so to create a reason with a code would create yet another string representation of the error which is different from every other representation already present.
msg147282 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-08 11:22
Ok. Sounds that you've already evaluated this alternative, so I'm fine
with it.
msg147313 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-11-08 17:39
Hi Jason & Petri,

urllib2.HTTPError never had a reason attribute. In the docs, it is mentioned that only URLError has the reason attribute. The HTTPError sublasses URLError and addinforurl class, but the further initialization happens only in the addinforurl class atrs.

If you got confused that the HTTPError as args and not reason, the 'args' is not coming from URLError.

HTTPError is raised for peculiar conditions such like authentication failures and it is 'used' as expected failure for certain authentication conditions. URLError is not so, it is seen more as an exception like socket errors.

The example your illustrated is an Authentication failure and as per docs, it is guaranteed to have code attribute to verify the kind of HTTP error are getting. msg is corresponding HTTP error code msg.

Take this example for URLError which will have reason attribute, it will work in both 2.7,3.2 and 3.3

import urllib.request, urllib.error, urllib.parse

except urllib.error.URLError as exc:

Because this is a socket error, the reason as "[Errno -2] Name or service not known" and HTTPError may not be a proper exception for this. 

This is more of an IOError which urllib calls a URLError.

I am not sure, how the need for 'reason' attribute for HTTPError exception was felt as the docs just say about 'code'.  (HTTPError  is informed as a subclass of URLError, but HTTPError does not call the URLError 's  __init__ and acts standalone. 

I am not sure, how to go about with this bug. If a new .reason attribute has to be added to HTTPError, then it is a feature request though, I wonder why we need when code and msg serve an adequate purpose.
msg147314 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-08 18:54
My impression was that because HTTPError is a subclass of URLError, it
should have the same attributes, and maybe some extra. A user reading
the docs doesn't know whether it calls the base class __init__ or not.
msg147318 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-11-08 21:39
That was my point. If HTTPError is a subclass of URLError, then an HTTPError _is an_ URLError, and thus should implement the same public interface.

The problem is better illustrated with this request:

except urllib2.URLError as exc:
  # We caught a URLError, what's the reason?

This code will fail with an attribute error, but only when the except clause catches an HTTPError (as it does in this case).

The documentation explicitly states that HTTPError is a subclass of URLError and it doesn't say anything about the addinfourl interface. The documentation strongly suggests (though implicitly) that HTTPError.reason will be present as it is with URLError.

If HTTPError does not implement the reason attribute, then there is no value in making it the subclass of URLError, and HTTPError should probably have the same parent class as URLError. However, this change is even more drastic and would almost certainly violate backward compatibility constraints.

The proposal I've made is generally backward compatible, and addresses the underlying bug (that URLError.__init__ is never called), and the symptom that a HTTPError does not implement the documented public interface.

Perhaps there's a better approach, but I believe based on the example code above, the implementation is broken.
msg148803 - (view) Author: Roundup Robot (python-dev) Date: 2011-12-03 14:46
New changeset ee94b89f65ab by Jason R. Coombs in branch '2.7':
Issue #13211: Add .reason attribute to HTTPError to implement parent class (URLError) interface.

New changeset abfe76a19f63 by Jason R. Coombs in branch '3.2':
Issue #13211: Add .reason attribute to HTTPError to implement parent class (URLError) interface.

New changeset deb60efd32eb by Jason R. Coombs in branch 'default':
Merged fix for #13211 from 3.2
msg148828 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-04 02:05
The 3.x buildbots are all broken:
msg148830 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-12-04 03:35
I suspect that this problem is caused by the fix for issue #12555.
msg148834 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-12-04 04:18
Antoine, I think Arfrever is on to something here. It does appear the new test added to test_urllib2 is failing, but for reasons I don't exactly understand. I'm initializing the HTTPError instance according to its signature in urllib.error, but it fails to accept keyword arguments.

I guess for the sake of this test, I can pass positional arguments, though this added limitation should probably be addressed in #12555.
msg148835 - (view) Author: Roundup Robot (python-dev) Date: 2011-12-04 04:20
New changeset a3ddee916808 by Jason R. Coombs in branch 'default':
Pass positional arguments - HTTPError is not accepting keyword arguments. Reference #13211 and #12555.
msg148851 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-12-04 14:19
After yet another commit, the build bots are green again:
msg177238 - (view) Author: Roundup Robot (python-dev) Date: 2012-12-09 21:53
New changeset e1ba514ddcd2 by Senthil Kumaran in branch '3.2':
Fix issue13211 - Document the reason attribute for urllib.error.HTTPError
Date User Action Args
2012-12-09 21:53:24python-devsetmessages: + msg177238
2012-12-08 05:54:40terry.reedysetversions: + Python 3.4
2011-12-04 14:19:21jason.coombssetmessages: + msg148851
2011-12-04 04:20:41python-devsetmessages: + msg148835
2011-12-04 04:18:06jason.coombssetmessages: + msg148834
2011-12-04 03:35:44Arfreversetnosy: + Arfrever
messages: + msg148830
2011-12-04 02:05:12pitrousetnosy: + pitrou
messages: + msg148828
2011-12-03 14:47:19jason.coombssetstatus: open -> closed
resolution: fixed
2011-12-03 14:46:12python-devsetnosy: + python-dev
messages: + msg148803
2011-11-08 21:49:31jason.coombssetcomponents: + Library (Lib)
2011-11-08 21:39:54jason.coombssetmessages: + msg147318
2011-11-08 18:54:38petri.lehtinensetmessages: + msg147314
2011-11-08 17:39:07orsenthilsetmessages: + msg147313
2011-11-08 11:22:27petri.lehtinensetmessages: + msg147282
2011-11-07 21:38:13jason.coombssetmessages: + msg147258
2011-11-07 19:21:12petri.lehtinensetmessages: + msg147253
2011-11-07 16:17:43jason.coombssetmessages: + msg147233
2011-11-07 16:15:58jason.coombssetfiles: + fffeff7721c0.diff
2011-11-07 16:15:25jason.coombssetkeywords: + patch, needs review
hgrepos: + hgrepo88
2011-10-27 16:53:43orsenthilsetnosy: + orsenthil
2011-10-26 09:04:39jason.coombssetmessages: + msg146426
2011-10-26 06:03:20petri.lehtinensetnosy: + petri.lehtinen
messages: + msg146416

keywords: + easy
stage: needs patch
2011-10-18 13:06:08jason.coombscreate