I found this bug when I started to trying Python 3.2 release candidate 1.

When using urllib.request.urlopen to handle HTTP POST, I got the error message:

ValueError: Content-Length should be specified for iterable data of type <class 'str'> 'foo=bar'

I'll attach the patch and test case.

Senthil, could this be a regression of the recent urllib transfer-encoding changes?

The POST data should be bytes. So in the attached test case, instead of

request = urllib.request.Request('http://does.not.matter', 'foo=bar')

it should be:

request = urllib.request.Request('http://does.not.matter', b'foo=bar')

And the Content-Length will be calculated using this logic.

mv = memoryview(data)
Content-length = len(mv) * mv.itemsize

Should we emphasize further that data should be bytes? I think error
message can be improved here. This issue is not a blocker.

Since r70638, the http client encodes unicode to ISO-8859-1:
<< RFC 2616 says that iso-8859-1 is the default charset for HTTP entity
bodies, but we encoded strings using ascii. See
http://bugs.python.org/issue5314. Changed docs and code to use
iso-8859-1. >>

If the POST data should be bytes which I also think reasonable,

should urllib.parse.urlencode return bytes instead of str?

>>> urllib.parse.urlencode({'foo': 'bar'})
'foo=bar'
>>> urllib.parse.urlencode({b'foo': b'bar'})
'foo=bar'

That would seem correct to me.

It is also a question whether to disallow str explicitly, instead of letting it go through the Iterable check.

So, what's the decision to be taken? I'm willing to provide patches (if I need to), but I need to know *the reasonable behaviors*. :)

For this particular issue, I think, it is good idea to disallow str
explicitly, instead of letting it go through the Iterable and raise a
ValueError with a different message.

Then let us do that.

Senthil, what about urlencode of bytes values returning a str?

On Sat, Feb 5, 2011 at 12:33 AM, Georg Brandl <report@bugs.python.org> wrote:

Then let us do that.

Senthil, what about urlencode of bytes values returning a str?

Sorry, the late response. I needed some time to look at this one and I
could only do it today.
We have been having str return value from urlencode, due to quote,
quote_plus methods which can take bytes/str but return str. (urlencode
is nothing but quote_plus which acts on two-element sequences like
name,value pairs.

The resultant str is certainly useful when we want to construct a URL
query string. But, when we want to do a POST via urlopen, we need
bytes data. So, I think best to advice through the documentation that,
urlencoded value must be converted to bytes if it is to be used for
POST'ing using urlopen.

(Well, I also think that urllencode could be modified to the effect
that if the key,value pairs are bytes, return the bytes output.
Technically, this would involve overriding some behavior of
quote/quote_plus and the return value will suitable only for POST data
and not for url query construction. I would rather choose to discuss
this before settling on the behavior and I remember that quote's
behavior did take a significant amount discussion).

So, for this issue. A patch would a prevent str data from being posted
with ValueError explaining the reason that bytes and required and a
documentation update in the urlencode method which would inform the
users that the output be converted to bytes before it is POSTed,

Shall we go ahead with this, Georg? I shall post a patch immediately.

Here is a patch to resolve this issue. Let me know if it okay to commit.(Or feel free to commit too).

Georg Brandl: can this fix go into Python 3.2? It changes the API.

I like any patch rejecting unicode where unicode is irrevelant (where we don't know how to choose the right encoding).

About the patch: you should maybe add a test to ensure that str is rejected with a TypeError.

Victor - It does not change the API. Only that the ValueError message which had a confusing message that "for iterable data of type <class 'str'> " is made clear that POST should not be a str.
Yes, a TypeError instead of ValueError would be more appropriate here.

Victor: I don't see an API change here. A ValueError is raised in both cases, and error messages are not part of the API. (And BTW, you may call me Georg.)

Senthil: I'd like the documentation change to be a bit more explicit that a string is always returned. Since the string must be encoded, it would also be helpful to mention which encoding to use.

The new exception message looks slightly incorrect to me: the argument can also be an iterable of bytes.

Here is the patch with addressing the comments.

1. It should be TypeError instead of ValueError when str is rejected. We introduced ValueError for this release only, so there is no breakage here.
2. Informed the user that string should be encoded to bytes using user-specified encoding before it is sent as POST data. Also pointed to Examples in the urllib.request section. ( I could not hyperlink directly to #examples section. If that is possible, it would be better).

patch with Docs :ref: to proper section.

I still find "user-specified encoding" unclear. This can be addressed at a different time though.

Your exception message is missing a space between "bytes" and "or"; otherwise this is ok to commit.

Thanks. Committed in r88394.

Regarding the encoding information. An explanation of this sort might be helpful.

The string can be encoded using ISO-8859-1 which is the default encoding for POST data or the user can also encode using a custom encoding , in which case, Content-Type: header should specify the encoding value in addition to 'application/x-www-form-urlencoded' which is specified for POST data. For Example:, if one has to use utf-8

>> sdata = urllib.parse.urlencode({"ሀ":"ሢ"})
>> bytesdata = sdata.encode('utf-8')
>>reqheaders=dict([("Content-Type"," application/x-www-form-urlencoded ; charset=utf-8")])
>>urllib.request.Request('http://www.example.com',bytesdata,reqheaders)

Lowering priority and making a doc issue now that the code change has been made.

BTW, there was no entry in Misc/NEWS, so this was not in whatsnew/3.2.rst.

Here is the docs patch which will help us close the issue. Addressing Eric's last comment - I believe the what's new and News for this issue was added with the feature, this one was Exception msg change.

I think the information in the patch should go in the urlopen doc, not in urlencode. Adding a note to urlencode that says that the result must be encoded is fine, but all the details about the default encoding and the fact that an extra Content-Type header is necessary when the encoding is not iso-8859-1 belong to the urlopen doc IMHO. (Is the header actually necessary? I think I always sent utf-8 without header, and the example you added in r88394 also uses utf-8 with no extra headers.)
There are also a few minor issues with the patch -- I'll leave a note in rietveld for that.

New changeset 057cf78ed576 by Senthil Kumaran in branch '3.2':
Explain the use of charset parameter with Content-Type header. bpo-11082
http://hg.python.org/cpython/rev/057cf78ed576

New changeset 90e35b91756d by Senthil Kumaran in branch 'default':
Explain the use of charset parameter with Content-Type header: bpo-11082
http://hg.python.org/cpython/rev/90e35b91756d

I have rewritten some parts of the documentation, explaining the use of charset parameter with Content-Type header. Used the suggestions from the previous patch's review comments too. I see that other parts of the documentation can be improved further, I shall do that separately. I am closing this issue. Thanks!