On a URL request where the server returns a URL with spaces in, 
urllib doesn't correct it before requesting the new page.

I think this is technically a server error, however, it does work 
from web browsers (Mozilla, Safari) but not from Python urllib.

I would suggest that when urllib is following "moved temporarily" 
links (or similar) from a server it translates spaces to %20.

See example program file for more including detailed server/client 
transactions text.

Logged In: YES 
user_id=1000470

I've tested a change to "redirect_internal(self, url, fp, errcode, errmsg, 
headers, data)" in "urllib.py" that adds a single line newurl = 
newurl.replace(" ","%20") after the basejoin() function call that appears 
to fix the problem.

This information is placed in the public domain.

Logged In: YES 
user_id=371366

I agree that it is a server error to put something that doesn't 
meet the syntactic definition of a URI in the Location header 
of a response. I don't see any harm in correcting obvious 
errors, though, in the interest of usability.

As for your proposed fix, instead of just correcting spaces, I 
would do

    newurl = quote(newurl, safe="/:=&?#+!$,;'@()*[]")

quote() is a urllib function that does percent-encoding. It is 
way out of date and does poorly with Unicode strings, but if 
called with the above arguments, it should safely clean up 
most mistakes. The set of additional "safe" characters I am 
passing in is the complete set of "reserved" characters 
according to the latest draft of RFC 2396bis; these are 
characters that definitely do or might possibly have special 
meaning in a URL and thus should not be percent-encoded 
blindly.

I am currently working on a urllib.quote() replacement for 
4Suite's Ft.Lib.Uri library, and will then see about getting the 
improvements folded back into urllib.

Logged In: YES 
user_id=371366

In my previous example, I forgot to add a tilde to the "safe" 
characters. Tilde is an "unreserved" character, a class of 
characters allowed in a URI but for which percent-encoding is 
discouraged.

    newurl = quote(newurl, safe="/:=&?~#+!$,;'@()*[]")

Superseded by issue 1153027 ?

I agree - this appears to be the same as issue 1153027 ?

This bug refers to urllib.  Issue 1153027 refers to urllib2.  It's the
same problem in each case, though.

Any idea about how to test this one reliably?

Mike's list is missing one more character, "%" itself.  So, the
replacement should be:

quote(newurl, safe="%/:=&?~#+!$,;'@()*[]")

The replacement should be done in the .open() method, not in the code
that handles redirects.

A suitable test would be to derive a urllib.URLOpener subclass that has
an open_spam(url) method which records the URL.  Then call
.open("spam://") in the test and verify that the URL has been quoted.

Uh, that should have beeen something like
.open("spam://example.com/%2E/") of course.

The attached patch fixes this issue and adds the test too. Followed John
J Lee's suggestion for the patch.Can someone review this?

For the other referenced issue1153027, bug is not reproducible in the
trunk and I see that fix has been made in revision 43132. It follows the
approach of percent encoding space at redirect_request. I shall verify
if we need to change that to a different fix or present one would be
sufficient.

Fixed this in the revision 71780 for Python 2.7.

This seems a bit serious for inclusion in 2.7 IMHO.  urllib is used in all 
sorts of hackish ways in the wild and I really wonder if this is going to 
cause more problems for people than it's worth.  The 3.x series alone 
seems like the best place for this which is addressed by issue 1153027.

AdamN, did you specifically come across a scenario which broke due to
this change? I can understand your concern, in general. The
'non-breaking' existing tests is the one of confidence factor we have in
introducing the changes.

I can't think of too many specific scenarios.  It just seems like a non-
trivial behavior change (or rather, it is trivial but with possibly far 
reaching ramifications).

One issue I see is that the ticket morphed from just dealing with space 
characters to many special characters.  quote_plus() probably would have 
handled the space issue cleanly.

The other issue I'm concerned about is that there is no testing of non-
ascii URLs (or at least I don't see any).

Finally, a netloc like 'www.g oogle.com' would be converted invalidly to 
'www.g%20oogle.com' which would then be forwarded to an actual HTTP 
request.

I can't point to anything too specific - more of just a feeling that 
this ticket is already 5 1/2 years old and maybe things should be left 
as-is and enhancement made only to Python 3.x.  Of course, I'm really 
just a new Python convert so I don't have massive experience here by any 
means.

> For the other referenced issue1153027, bug is not reproducible in the
> trunk and I see that fix has been made in revision 43132.

This change was a fix for bpo-1353433:

commit ddb84d7c69addc5d5e2ab3e327260d97b52af3a7
Author: Georg Brandl <georg@python.org>
Date:   Sat Mar 18 11:35:18 2006 +0000

    Bug #1353433: be conciliant with spaces in redirect URLs