In urllib2 when using reusing a Request calling add_header doesn't work
when an unredirected_header has been added. 

A good example (and the one that caught me out) is content-type. When
making a POST request with no content-type set the current code sets the
content-type as an unredirected_header to
'application/x-www-form-urlencoded' (line 1034 of urllib2.py) but in a
subsequent request, setting the content type via add_header will see
this ignored as unredirected_headers are appended after the headers.

A possible solution is to check whether the header being added already
exists in the requests undredirected_headers and remove it if it does.

I can see that there will be a problem in this case.

However, it may be that the authors found it more intuitive to always
set the Content-Type to application/x-www-form-urlencoded when data is set.

Their implementation is inconsistent though:

#-------------------------------------------------------
request = urllib2.Request('http://www.somesite.com/')
request.add_data(data)
f = urllib2.urlopen(request)

# 'Content-Type: application/x-www-form-urlencoded' is sent in the header
#-------------------------------------------------------

whereas:

#-------------------------------------------------------
request = urllib2.Request('http://www.somesite.com/')
request.add_header('Content-Type', 'application/xml')
request.add_data(data)

# 'Content-Type: application/xml' is sent in the header
#-------------------------------------------------------

We need to decide what is the best way to handle this case.  Should the
normal headers be given preference, or should the unredirected headers
be given preference?

--- Martin

Sorry, the first example should read:

#-------------------------------------------------------
request = urllib2.Request('http://www.whompbox.com/headertest.php')
request.add_data(data)
f = urllib2.urlopen(request)

request.add_header('Content-Type', 'application/xml')
f = urllib2.urlopen(request)

# 'Content-Type: application/x-www-form-urlencoded' is sent in both
headers.  Second header (should?) be 'application/xml'
#-------------------------------------------------------

I decided that the user should have full control over the headers sent.
 Any call to add_header() or add_redirected_header() will now check if a
header with that same name has been set in the Request before (in both
the header and the unredirected header dictionaries), and if so
overwrite it.

The tests run fine with the patch installed, and also included is a
patch for the urllib2 documentation.