Cookies are not resend when a redirect is requested.

Blurb:
I've been trying to get a response off a server using
Python.
The response so far differs from the response using
Firefox.
In Python, I have set headers and cookies the way
Firefox does it.

I noticed that the server accepts the POST request, and
redirects the client to another address with the result
on it. This happens both with Python and Firefox correctly.

Cookie handling differs though:
The Python client, when redirected, using the standard
redirect handler, does not resend its cookies to the
redirected address.
Firefox does resend the cookies from the original request.

When I redefine the redirect handler and code it so
that it adds the cookies from the original request,
the response is the same as Firefox's response. This
confirms then that resending cookies is required to get
the server to respond correctly.

Is the default Python redirection cookie policy
different from Firefox's policy?
Could we improve the default redirection handler to
work like Firefox?
Is it a bug?

I noticed an old open bug report 511786, that looks
very much like this problem. It suggests it is fixed.

Cheers Hans Moleman. 

Logged In: YES 
user_id=11375

More detail is needed to figure out if there's a problem;
can you give a sample URL to exhibit the problem?  can you
provide your code?  From the description, it's unclear if
this might be a bug in the handling of redirects or in the
CookieProcessor class.

The bug in 511786 is still fixed; that bug includes sample
code, so I could check it.

Logged In: YES 
user_id=1610873

I am using this script to obtain monthly internet usage
statistics from my ISP.
My ISP provides a screen via HTTPS, to enter a usercode and
password, after which the usage statistics are displayed on
a different address. 
I cannot send this script with my usercode and password. My
ISP might not like me doing this either.

Therefore I'll try to find another server that behaves
similarly, and send you that.

Logged In: YES 
user_id=11375

Given the sensitive data in your script, it's certainly best
to not post it.  You'll have to dig into urllib2 yourself, I
think.

Start by looking at the code in redirect_request(), around
line 520 of urllib2.py, and adding some debug prints.  Print
out the contents of req.headers; is the cookie line in
there?  Change the __init__ of AbstractHTTPHandler to
default debuglevel to 1, not 0; this will print out all the
HTTP lines being sent and received.

Logged In: YES 
user_id=1610873

OK. I'll have a look at that. 
Thanks for the pointers.
 

Logged In: YES 
user_id=1610873

I believe that a bit of coding is missing.

When a cookie is added in 'add_cookie_header' in
cookielib.py, it is added under the
request.unredirected_hdrs. Line 1317.
When a request is resend after a redirect request in
'redirect_request' in urllib2.py, the request.headers are
used. Line 509.

Additional coding is required that moves cookies from
'unredirected_hdrs' to 'headers' if the domain of 
the original request matches the domain of the redirected
request.

I've used http://www.w3.org/Protocols/rfc2109/rfc2109 for that.
No idea if that rfc is still current though.

 

Sorry I turned up rather late here (is there a way to subscribe to
changes to all bugs whose comments or title contain a given string?)

If it works with Firefox and not with cookielib it's almost certainly a
bug.  However, it's not clear to me from the report what the real bug
might be, because cookies aren't really redirected -- they're just added
to the new request following the usual rules for adding Cookie: headers
to requests.  That's why the redirect handler deliberately doesn't add
the unredirected headers -- the Cookie header gets added later on, by
HTTPCookieProcessor.  So, a diagnosis of the problem would involve
finding out why cookielib thinks that your cookie should not be returned
to the server in the request in question (which just happens to be a
redirected request).  Running your script with cookielib's logging
enabled (see the documentation) will probably be all that's required to
find that out; if you can still reproduce this, please post the log here
(sanitized to remove usernames, passwords, etc.).  If not, a test case,
or failing that a sanitized dump of the HTTP traffic, would be very useful.

I'll close this in a couple of weeks unless anyone objects.

John: You can subscribe to http://mail.python.org/mailman/listinfo/new-bugs-announce and use email filters.

Senthil: I’m making you nosy since I think you’re the most qualified to do something here.

I found a repeatable, public test case: 

http://www.macupdate.com/download/26915/ScreenFlow-4.5.1.dmg

Using urllib2 with the following code leads to a redirect loop: 

#!/usr/bin/python

import urllib2

h = urllib2.HTTPHandler(debuglevel=1)
h2 = urllib2.HTTPSHandler(debuglevel=1)

opener = urllib2.build_opener(h)
opener2 = urllib2.build_opener(h2)
urllib2.install_opener(opener)
urllib2.install_opener(opener2)

request = urllib2.Request(url="http://www.macupdate.com/download/26915/ScreenFlow-4.5.1.dmg")
request.add_header( "User-Agent", "foo" )

url_handle = urllib2.urlopen(request)

Paul's script (with urllib.request instead of urllib2) works for me. 

Is this issue still relevant?