urlopen does not always return a single value for 'content-length'.  For example:


>>> import urllib2
>>> request = 'http://wwwsearch.sourceforge.net/mechanize/src/mechanize-0.1.11.zip'
>>> fp = urllib2.urlopen(request)
>>> fp.info().dict
{'content-length': '289519, 289519', 'x-varnish': '929586024', 'via': '1.1 varnish', 'age': '0', 'expires': 'Fri, 25 Mar 2011 14:36:43 GMT', 'server': 'Apache/2.2.3 (CentOS)', 'last-modified': 'Sat, 07 Feb 2009 19:15:15 GMT', 'connection': 'close', 'etag': '"46aef-46258f510b6c0"', 'date': 'Wed, 23 Mar 2011 14:36:43 GMT', 'content-type': 'application/zip'}

Python 2.7.1 (r271:86832, Nov 27 2010, 17:19:03) [MSC v.1500 64 bit (AMD64)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> request = 'http://wwwsearch.sourceforge.net/mechanize/src/mechanize-0.1.11.zip'
>>> fp = urllib2.urlopen(request)
>>> fp.info()['content-length']
'289519, 289519'
>>>

Not reproducible on Python 3.2+ (presumably 3.x, as well).

Interesting, the Content-Length header was sent twice:

HTTP/1.1 200 OK
Server: Apache/2.2.3 (CentOS)
Last-Modified: Sat, 07 Feb 2009 19:15:15 GMT
ETag: "46aef-46258f510b6c0"
Content-Length: 289519
Expires: Fri, 25 Mar 2011 17:32:49 GMT
Content-Type: application/zip
Content-Length: 289519
Date: Wed, 23 Mar 2011 17:32:49 GMT
X-Varnish: 930137777
Age: 0
Via: 1.1 varnish

This affects urllib, as well:

C:\Users\santa>python
Python 2.7.1 (r271:86832, Nov 27 2010, 17:19:03) [MSC v.1500 64 bit (AMD64)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> request = 'http://wwwsearch.sourceforge.net/mechanize/src/mechanize-0.1.11.z
ip'
>>> fp = urllib.urlopen(request)
>>> fp.headers['content-length']
'289519, 289519'

Yes, interesting that Content-Length is returned as a comma separated value of ints. 
Normally, this behavior is observed for other headers which can have multiple values and urllib appends the subsequent values of the header for e.g. Content-Type header.

curl seems to have got it right in this case, where it returns only one Content-Length header for the above URL. urllib needs to modify to deal with this scenario - If there are multiple content-lengths returned by the server, just use the last one if we are mimic curl behavior. (AFAIK, the standard is to return them comma separated, but in this case, it wont be useful).

One way I can think of is by resorting so some list of exceptions. Not quite elegant, but it works...

Yes, this is good enough for a non-standard behavior.

However, I would be curious to know what curl did. Did it do something
on based on the value or sequence in which it was obtained or just
over-wrote it.

Varnish on the sourceforge server has been upgraded and/or reconfigured (yesterday) to fix the issue that was happening with this file (and others).

Just an FYI that you will no longer be able to re-create the triggering error.

'content-length': '289519',
'via': '1.1 varnish'

It is better to close this issue as it was a Server Error. Standard just says that when there two headers with different values, combine them comma separated as urllib2 does.  Making special case exception for 'Content-Length' header when the server is at fault would be bad idea. We will not know which value to choose from if the values are different.

Closing this bug as Invalid.


>>> import urllib2
>>> req = urllib2.urlopen('http://wwwsearch.sourceforge.net/mechanize/src/mechanize-0.1.11.zip')
>>> req.info()['content-length']
'289519'