classification
Title: urllib.request example should use "with ... as:"
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Valery.Khamenya, berker.peksag, docs@python, eric.araujo, ezio.melotti, javawizard, martin.panter, ncoghlan, orsenthil, python-dev, terry.reedy
Priority: normal Keywords: patch

Created on 2011-09-10 10:30 by Valery.Khamenya, last changed 2015-04-12 10:54 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
urlopen-with.patch martin.panter, 2015-02-09 01:22 review
Messages (15)
msg143836 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-09-10 10:30
The following intuitive construction

with urllib2.build_opener().open() as:
    ...

leads to AttributeError: addinfourl instance has no attribute '__exit__'

http://docs.python.org/library/urllib2.html says almost nothing about concept of closing.

Could it be possible to add a canonical "with ... as:" example for urllib2.build_opener() ?

Thanks,
Valery
msg143841 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-09-10 15:58
JFYI, the attempt to use closing() manager from contextlib have revealed to the following issue:

https://bugs.pypy.org/issue867
msg144141 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-09-16 17:23
As I understand this, you are asking that 2.7 urllib2.build_opener().open(), which in 3.x is urllib.request.build_opener().open(), be upgraded to return an object that works as a context manager. Unless the docs say that this should already be the case, this is a feature request for 3.3.

I am unable to test whether this feature is already present (in 3.2.2). Your example line "with urllib2.build_opener().open() as:" has an obvious syntax error. When I correct that and adjust for 3.x

import urllib.request as ur
with ur.build_opener().open() as f:
    pass
#
TypeError: open() takes at least 2 arguments (1 given)

The doc for build_opener says it returns an OpenerDirector instance. help(ur.OpenerDirector.open) just says it needs a 'fullurl'. But when I add 'http:www.python.org' as an argument, I get
urllib.error.URLError: <urlopen error no host given>
I do not know what else is needed.

Please copy and paste the ACTUAL (minimal) code you ran to get the AttributeError.
msg144142 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-09-16 17:42
Just as a quick guideline. If you are talking about context manager
support for urlopen, it is available and present in 3.x but not on
2.7. And I fear, it is late to make it available on 2.7, because it is
a feature.

In any case, as Terry requested, a simple snippet would help us
understand the problem and original poster's expectation.
msg144146 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-09-16 18:15
Terry, Senthil, thanks, for replying to this ticket. OK, to the question:

1. @Terry, here is the full example as for CPython 2.7 I am talking about and the output:

#################

from urllib2 import Request, build_opener

request = Request('http://example.com')

with build_opener().open(request) as f:
    txt = f.read()
    print '%d characters fetched' % len(txt)
####################

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: addinfourl instance has no attribute '__exit__'


2. @Senthil, regarding the statement being a feature. I assume, that to open a connection, to read from a connection and to close it -- are the fundamental functions for what urllib2 was in particular created for. I was looking in docs for some hints of "canonical" way of doing this using urllib2.opener. After I have failed to find any guidance in docs, I've created this ticket. That is, I assume that no new feature is needed, but just a documented 5-lines example about a typical way of doing the above operations, especially *closing* the connection.

regards,
Valery
msg144165 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-09-16 21:54
On 3.2.2, your example (adapted) produces
>>> 
2945 characters fetched

So, as Senthil said, the requested feature already exists. But it cannot be added to the 2.7 series; the Python 2.7 *language* is feature frozen. 2.7.z bug fixes serve to make the implementation better match the intended language.

Prior to the addition of with..., the standard way to close things was with explicit .close() or implicit closing when the function returned or the program exited. In the 3.2 docs, "20.5.21. Examples", there are several examples with
  f = <some opener>
  <do something with f>
  (closing ignored)
Where possible, these could and should be changed in 3.2+ docs to
  with <some opener> as f:
    <do something with f>
to promote the new idiom.  This had been done in other chapters. Perhaps there is also a need for 'X supports the context manager protocol' to be added here or there. But showing that in the examples would make the point.

I have changed the title of this issue to match.
msg144167 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-09-17 00:15
Guys, in my item 2 the simplistic goal was stated clearly: open, read and close.

Do you confirm that this basic sequence is not supported by urllib2 under 2.7 ?

(I just requested for a tiny documentation update entry)

regards,
Valery
msg144168 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-09-17 00:49
Valery, yes. I shall update 2.7 documentation with this known limitation and 3.x documentation with the example usage scenarios.

Thanks!
msg144195 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-17 15:31
[Terry]
> But when I add 'http:www.python.org' as an argument, I get
> urllib.error.URLError: <urlopen error no host given>

Your URI lacks a host (netloc, in RFC parlance) component:
>>> urllib.parse.urlparse('http:python.org')
ParseResult(scheme='http', netloc='', path='python.org', params='', query='', fragment='')
msg152574 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-02-04 01:39
Senthil:
http://docs.python.org/dev/library/contextlib.html#contextlib.closing
currently has this example:

from urllib.request import urlopen
with closing(urlopen('http://www.python.org')) as page:

which is misleading in that the object returned from urlopen
<class 'http.client.HTTPResponse'>
has __enter__ and __exit__ methods and therefore is a c.m. in itself and does not need to be wrapped in closing(). I did not really understand from your comment whether there is any need to use closing() with anything returned in urllib. 

At the moment, shelves are not C.M.s, and would make better examples, but #13896 suggests 'fixing' that also.
msg152594 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-04 07:11
Either we find a commonly used stdlib class that is not a context manager but has a close method and is not going to become a context manager (I can’t see why such a thing would be), or we can add something like: “closing is useful with code that you can’t change to add context management, for example urllib.urlopen before Python 3.3”.
msg235580 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-02-09 00:37
Issue 22755 is about the example arms race for contextlib.closing().
msg235581 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-02-09 01:22
Here is a patch to change the urlopen() examples to use context managers where appropriate.

There were also a few examples of handling HTTPError which I didn’t touch, because the whole file object versus exception object thing is probably a separate can of worms.
msg240550 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-04-12 10:53
New changeset 6d5336a193cc by Berker Peksag in branch '3.4':
Issue #12955: Change the urlopen() examples to use context managers where appropriate.
https://hg.python.org/cpython/rev/6d5336a193cc

New changeset 08adaaf08697 by Berker Peksag in branch 'default':
Issue #12955: Change the urlopen() examples to use context managers where appropriate.
https://hg.python.org/cpython/rev/08adaaf08697
msg240551 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-04-12 10:54
Great patch. Thanks Martin.
History
Date User Action Args
2015-04-12 10:54:41berker.peksagsetstatus: open -> closed
resolution: fixed
messages: + msg240551

stage: patch review -> resolved
2015-04-12 10:53:00python-devsetnosy: + python-dev
messages: + msg240550
2015-02-14 18:17:14berker.peksagsetnosy: + berker.peksag

stage: needs patch -> patch review
2015-02-09 01:22:17martin.pantersetfiles: + urlopen-with.patch
keywords: + patch
messages: + msg235581
2015-02-09 00:37:07martin.pantersetnosy: + martin.panter
messages: + msg235580
2014-07-14 20:11:12terry.reedysetversions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3
2013-12-06 00:19:02javawizardsetnosy: + javawizard
2012-02-04 07:11:06eric.araujosetmessages: + msg152594
2012-02-04 01:39:17terry.reedysetnosy: + ncoghlan
messages: + msg152574
2011-09-17 15:31:04eric.araujosetnosy: + eric.araujo
messages: + msg144195
2011-09-17 00:49:40orsenthilsetassignee: docs@python -> orsenthil
messages: + msg144168
2011-09-17 00:15:49Valery.Khamenyasetmessages: + msg144167
2011-09-16 21:54:09terry.reedysetversions: + Python 3.2
title: urllib2.build_opener().open() is not friendly to "with ... as:" -> urllib.request example should use "with ... as:"
messages: + msg144165

components: + Documentation, - Library (Lib)
stage: test needed -> needs patch
2011-09-16 18:15:26Valery.Khamenyasetmessages: + msg144146
2011-09-16 17:42:34orsenthilsetmessages: + msg144142
2011-09-16 17:23:32terry.reedysettype: resource usage -> enhancement
components: - Documentation, IO
versions: - Python 2.6, Python 3.1, Python 2.7, Python 3.2, Python 3.4
nosy: + terry.reedy

messages: + msg144141
stage: test needed
2011-09-10 15:58:46Valery.Khamenyasetmessages: + msg143841
2011-09-10 10:52:53ezio.melottisetnosy: + orsenthil, ezio.melotti
2011-09-10 10:30:28Valery.Khamenyacreate