while urllib following the redirection(302):
urllib.client.HTTPConnection.putrequest raise an error:
#

File "D:\Program Files\Python32\lib\http\client.py", line 1004, in _send_request
self.putrequest(method, url, **skips)
File "D:\Program Files\Python32\lib\http\client.py", line 868, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-111: ordinal not in range(128)
#----------------------------------------------------------
in the sourcode i found that:
at line 811
def putrequest(self, method, url, skip_host=0,skip_accept_encoding=0)...
the argument url may be a unicode,and it was unquoted..

i think we should replace:
request = '%s %s %s' (method,url,self._http_vsn_str)
with:
import urllib.parse
request = '%s %s %s' (method,urllib.parse.quote(url),self._http_vsn_str)

while urllib following the redirection(302):
http.client.HTTPConnection.putrequest raise an error:
#

...
File "D:\Program Files\Python32\lib\http\client.py", line 1004, in _send_request
self.putrequest(method, url, **skips)
File "D:\Program Files\Python32\lib\http\client.py", line 868, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 108-111: ordinal not in range(128)
#----------------------------------------------------------
in the sourcode i found that:
at line 811

def putrequest(self, method, url, skip_host=0,skip_accept_en...)
...

the argument url may be a unicode,and it was unquoted..
----------------------------note----------------------------------------
in my case:
...
purl="http://bbs.dospy.com/1111258attachdown.php?aid=14361277&bbsid=349"
req=urllib.request.Request(purl,headers=headers)
response=urllib.request.urlopen(req)
...

then,the http serve redirect me to a file download url...
and the url contains some Chinese word....
i have print out the argument url:

/f/1ba1f70606223af2aa5c3aeff6c6a46a/511f7b4c/day_111015/20111015_5949e996881b2e28403d26Ch6dOfj6LZ.rar/p/ÒâÁÖ03-08.part1.rar

Please give us

1. the exact Python version used. 3.2.3? or something earlier?
2. A minimal but complete example that we can run. What is 'headers'?
3. The complete traceback, not just the last two entries.
4. The result of running with the newer 3.3.0, if you possibly can. Perhaps the problem has already been fixed.

While line numbers have changed, even in 3.2.4 in repository, 3.2-3.4 all have

    request = '%s %s %s' % (method, url, self._http_vsn_str)
    # Non-ASCII characters should have been eliminated earlier
    self._output(request.encode('ascii'))

Since there is nothing earlier in the function that would eliminate non-ascii, there must be an assumption about what happens earlier in the call chain. That might have already been fixed, which is why we need an example to test.

This problem still exist in Python 3.3.2. The following code gives you an example:

import urllib.request
url = "http://www.libon.it/libon/search/isbn/3499155443"
req = urllib.request.Request(url)
response = urllib.request.urlopen(req, timeout=30)
the_page = response.read().decode('utf-8')
print(the_page)

Traceback (most recent call last):
  File "C:\X\webpy.py", line 4, in <module>
    response = urllib.request.urlopen(req, timeout=30)
  File "C:\Python33\lib\urllib\request.py", line 156, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Python33\lib\urllib\request.py", line 475, in open
    response = meth(req, response)
  File "C:\Python33\lib\urllib\request.py", line 587, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python33\lib\urllib\request.py", line 507, in error
    result = self._call_chain(*args)
  File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
    result = func(*args)
  File "C:\Python33\lib\urllib\request.py", line 692, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python33\lib\urllib\request.py", line 469, in open
    response = self._open(req, data)
  File "C:\Python33\lib\urllib\request.py", line 487, in _open
    '_open', req)
  File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
    result = func(*args)
  File "C:\Python33\lib\urllib\request.py", line 1268, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "C:\Python33\lib\urllib\request.py", line 1248, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "C:\Python33\lib\http\client.py", line 1061, in request
    self._send_request(method, url, body, headers)
  File "C:\Python33\lib\http\client.py", line 1089, in _send_request
    self.putrequest(method, url, **skips)
  File "C:\Python33\lib\http\client.py", line 953, in putrequest
    self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 78-79: ordinal not in range(128)

The script for demonstrating bug can be simplified to:

-----------------------------------------------------------------------

import urllib.request
url = "http://www.libon.it/ricerca/7817940/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-Käfer/order/date_desc"

req = urllib.request.Request(url)
response = urllib.request.urlopen(req, timeout=30)
the_page = response.read().decode('utf-8')
print(the_page)

Attached the simple patch to solve this problem.

The question is whether we should fix this problem in urllib or not because strictly speaking the url should be ascii characters only. But if the Firefox can open this url, why not urllib?

I will contemplate about this problem and if I (or other people) think that urllib should handle url containing non-ascii characters, then I will add additional unit test.

Until then, people can use third party package, which is
request package from http://docs.python-requests.org/en/latest/

----------------------------------------------------------------

r = requests.get("http://www.libon.it/ricerca/7817940/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-Käfer/order/date_desc")
print(r.text)

The problem may not be a bug but a deliberate design choice. urllib is rather low level and doesn't implement some browser magic. Browsers handle stuff like 'ä' -> '%C3%A4', ' ' -> '%20' or IDNA but urllib doesn't. I always saw it as may responsibility to quote and encode everything myself. Higher level APIs such as requests are free to implement browser magic.

Contrary to common believes an URL with an umlaut or space is *not* a valid URI. From http://docs.python.org/3/library/urllib.request.html#urllib.request.Request

url should be a string containing a valid URL.

I suggest that this ticket shall be closed as "won't fix".

I have no problem if this ticket is classified as "won't fix".

I am writing this for the confused souls who want to use urllib to access url containing non-ascii characters:

import urllib.request
from urllib.parse import quote
url = "http://www.libon.it/ricerca/7817940/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-Käfer/order/date_desc"

req = urllib.request.Request(url)
try:
    req.selector.encode('ascii')
except UnicodeEncodeError:
    req.selector = quote(req.selector)
response = urllib.request.urlopen(req, timeout=30)
the_page = response.read().decode('utf-8')
print(the_page)

The problem isn't the original requested url, as it is legit. The problem appears after the 302 redirect when a new (malformed) url is received from the server. There need to be some kind of check of the validity of that second url. And, preferably, an URLError returned if something is wrong.

Lars, I see.

For the uninitiated, the issue is the original url (containing only ascii character) redirects to the url containing non-ascii characters which upsets urllib.

To handle that situation, you can do something like this:
---------------------

import urllib.request
from urllib.parse import quote
url = "http://www.libon.it/libon/search/isbn/3499155443"
req = urllib.request.Request(url)
req.selector = urllib.parse.quote(req.selector)
response = urllib.request.urlopen(req, timeout=30)
the_page = response.read().decode('utf-8')
print(the_page)

I admit it that this code is clunky and not pythonic.

I also believe in python standard library, we should have a module to access url containing non-ascii character in an easy manner.

At the very least, maybe we can give proper error message. Something like this would be nice:

"The url is not valid and contains non-ascii character: http://www.libon.it/ricerca/7817940/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-Käfer/order/date_desc. This url is redirected from this url: http://www.libon.it/libon/search/isbn/3499155443"

Because users can be confused. They thought they already gave only-ascii-characters url (http://www.libon.it/libon/search/isbn/3499155443) to urllib, but why did they get encoding error?

What do you say, Christian?

Something else is going on here. A valid server never returns an URL with non-ASCII chars. Your test server does the right thing, too:

$ LC_ALL=C wget http://www.libon.it/libon/search/isbn/3499155443
--2013-07-19 11:01:54--  http://www.libon.it/libon/search/isbn/3499155443
Resolving www.libon.it (www.libon.it)... 83.103.59.131
Connecting to www.libon.it (www.libon.it)|83.103.59.131|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://www.libon.it/ricerca/7818684/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-K%C3%A4fer/order/date_desc [following]
Incomplete or invalid multibyte sequence encountered
--2013-07-19 11:01:54--  http://www.libon.it/ricerca/7818684/3499155443/dettaglio/3102314/Onkel-Oswald-und-der-Sudan-K%C3%A4fer/order/date_desc
Reusing existing connection to www.libon.it:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

I have digged through the code. Now I think that I know what's going on here. The header parsing code unquotes and converts the Location header. The code in the 302 handler doesn't compensate and therefore fails.

Here is a patch that corrects the code in the 302 function.

I think this patch needs a test. I left some comments on Reitveld as well. Perhaps there should also be a test to prove that redirects to URLs like /spaced%20path/ do not get mangled.

Have a look at the HTTPRedirectHandler.redirect_request() method. Perhaps the code translating spaces to %20 could be merged with the fix for this issue.

The patch bpo-17214 did fix this issue in my 3.4.2 install on Ubuntu LTS.

It triggered however another bug:

File "/usr/local/lib/python3.4/urllib/request.py", line 646, in http_error_302
path = urlparts.path if urlpaths.path else "/"
NameError: name 'urlpaths' is not defined

This is obviously a typo.

I'm not sure if that one has been reported yet (a short google search didn't find anything) and I don't know how to provoke it independently.

I should have looked more closely.

The typo is part of the patch. It should be corrected there.

This bug only applies to Python 3. In Python 2, the non-ASCII bytes are sent through to the redirect target verbatim. I think this would also be the ideal way to handle the problem in 3, but percent-encoding them as proposed also seems good enough, and does not require hacking the HTTPConnection.putrequest() internals.

My patch updates Christian’s patch:

• Tested, so hopefully no typos :)
• Add test cases based on bpo-22248, as well as a URL already including a percent sign
• Process entire URL, not just the path component. A non-ASCII byte could just as easily be in the query component, for example.
• Remove redundant encoding of space character from redirect_request() method.

I will look at committing this soon

New changeset cb09fdef19f5 by Martin Panter in branch '3.5':
Issue bpo-17214: Percent-encode non-ASCII bytes in redirect targets
https://hg.python.org/cpython/rev/cb09fdef19f5

New changeset 841a9a3f3cf6 by Martin Panter in branch 'default':
Issue bpo-14132, Issue bpo-17214: Merge two redirect handling fixes from 3.5
https://hg.python.org/cpython/rev/841a9a3f3cf6

I restored the “redundant” encoding of space, in case someone’s code was relying on this behaviour, and because redirect_request() is a publicly documented method.