classification
Title: urlopen breaks when data parameter is used.
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: david193, docs@python, ezio.melotti, georg.brandl, orsenthil, python-dev
Priority: normal Keywords:

Created on 2011-02-20 22:07 by david193, last changed 2012-03-15 01:11 by python-dev. This issue is now closed.

Messages (5)
msg128922 - (view) Author: David Phillips (david193) Date: 2011-02-20 22:07
The following code works on python 3.1.3 but fails on Python 3.2rc2 (r32rc2:88269, Jan 30 2011, 14:30:28). (I run Mac OS X, version 10.6.6.)

-----------------------------

import urllib, urllib.request, urllib.error, urllib.parse

form = urllib.parse.urlencode({'field1':'Log in'})
try:
	response = urllib.request.urlopen('http://www.google.com/', form)
except urllib.error.HTTPError as exception:
	print (exception)

---------------------------------------

When loaded, the following error messages are generated:

---------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py", line 1057, in do_request_
    mv = memoryview(data)
TypeError: cannot make memory view because object does not have the buffer interface

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py", line 138, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py", line 364, in open
    req = meth(req)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/urllib/request.py", line 1062, in do_request_
    data))
ValueError: Content-Length should be specified                                 for iterable data of type <class 'str'> 'field1=Log+in'

------------------------------------
msg128930 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-02-21 01:51
On Sun, Feb 20, 2011 at 10:07:31PM +0000, David Phillips wrote:

> The following code works on python 3.1.3 but fails on Python 3.2rc2
> (r32rc2:88269, Jan 30 2011, 14:30:28). (I run Mac OS X, version
> 10.6.6.)

Is that a real world code? (As in used in production)?  Because,
things have been a bit tightened in 3.2 and it expects data to be
bytes and throws an exception if it is not bytes.

urlencode will output str and you have explicitly encode it to bytes
(Using the value Accept-Encoding response header) before you send the
data to urlopen. The updated docs reflect this change.
msg128935 - (view) Author: David Phillips (david193) Date: 2011-02-21 04:23
Converting the type of my variable "form" from string to bytes did, indeed, allow the code to run, but I have to wonder about changing the inputs to urlopen like this. 

The 3.1.3 docs call for the data parameter to be string, and I presume that has been the case all along. Changing the data parameter from string to bytes may be a relatively clean change to make, but this change is going to break a lot of existing code. Are you sure you want to do that?
msg128944 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-21 06:58
Quite a few docs still say "string" where in fact bytes are expected in Python 3.x; we're updating these as we go along.
msg155834 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-03-15 01:11
New changeset fddbe9a59d26 by Senthil Kumaran in branch '3.2':
Fix the wrong urllib exampls which use str for POST data. Closes Issue11261
http://hg.python.org/cpython/rev/fddbe9a59d26

New changeset 0345dc184e9a by Senthil Kumaran in branch 'default':
cpython:Fix the wrong urllib exampls which use str for POST data. Closes Issue11261
http://hg.python.org/cpython/rev/0345dc184e9a
History
Date User Action Args
2012-03-15 01:11:55python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg155834

resolution: fixed
stage: needs patch -> resolved
2011-11-15 20:03:00ezio.melottisetnosy: + ezio.melotti

stage: needs patch
2011-02-21 06:58:16georg.brandlsetnosy: georg.brandl, orsenthil, docs@python, david193
type: crash -> behavior
versions: + Python 3.3
2011-02-21 06:58:09georg.brandlsetnosy: + docs@python, georg.brandl
messages: + msg128944

assignee: docs@python
components: + Documentation, - Library (Lib)
2011-02-21 04:23:48david193setnosy: orsenthil, david193
messages: + msg128935
2011-02-21 01:51:04orsenthilsetnosy: orsenthil, david193
messages: + msg128930
2011-02-20 22:11:26ned.deilysetnosy: + orsenthil
2011-02-20 22:07:29david193create