classification
Title: Problem with httplib and Content-Length: -1
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: airadier, christian.heimes, georg.brandl, sdownum
Priority: low Keywords: easy

Created on 2007-12-14 12:52 by airadier, last changed 2008-02-24 00:15 by georg.brandl. This issue is now closed.

Files
File name Uploaded Description Edit
1626_patch.diff sdownum, 2008-01-20 01:42 Proposed patch.
Messages (4)
msg58624 - (view) Author: Álvaro Iradier (airadier) Date: 2007-12-14 12:52
When opening an IP Webcam URL with urllib2, the response is a continuous
secuence of Jpeg files from the server, preceded by the following headers:

Server: DM-Web
Content-type:
multipart/x-mixed-replace;boundary=0plm(Pico-Web:Server-Push:Boundary-String)1qaz
Content-length: -1

As you can see, the Content-Type is multipart/x-mixed-replace, and the
Content-Length reported by the server is '-1', which is strange (I guess
it would be better not to report Content-Length)

The problem is trying to read anything from the file-like object
returned by urlopen will block forever. Problem seems to be here, in
httplib.py, class HTTPResponse, method 'begin':

...
        # will the connection close at the end of the response?
        self.will_close = self._check_close()

        # do we have a Content-Length?
        # NOTE: RFC 2616, S4.4, #3 says we ignore this if tr_enc is
"chunked"
        length = self.msg.getheader('content-length')
        if length and not self.chunked:
            try:
                self.length = int(length)
            except ValueError:
                self.length = None
        else:
            self.length = None
...

The length attribute is being set to '-1' which leads to blocking when
reading from the endless stream of data. (See the read method in class
_fileobject, socket.py).

I don't know if this is the right fix, but I would suggest changing:

        length = self.msg.getheader('content-length')
        if length and not self.chunked:

to:

        length = self.msg.getheader('content-length')
        if length >= 0 and not self.chunked:
msg58779 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-12-18 22:07
Your proposed fixed is not correct:

        length = self.msg.getheader("content-length")
        if length and not self.chunked:
            try:
                self.length = int(length)
            except ValueError:
                pass
        # patch
        if self.length < 0:
            self.length = None
msg60253 - (view) Author: Steven Downum (sdownum) Date: 2008-01-20 01:42
I have made a patch of the proposed fixed and ran the httplib test 
afterwards, to make sure it passed.
msg62851 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-02-24 00:15
Fixed in r61035.
History
Date User Action Args
2008-02-24 00:15:51georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg62851
nosy: + georg.brandl
2008-01-20 01:42:55sdownumsetfiles: + 1626_patch.diff
nosy: + sdownum
messages: + msg60253
2008-01-12 02:45:59christian.heimessetpriority: low
keywords: + easy
versions: + Python 2.6, - Python 2.5
2007-12-18 22:07:21christian.heimessetnosy: + christian.heimes
messages: + msg58779
2007-12-14 12:52:01airadiercreate