classification
Title: Httplib read routine is not tolerant to not well-formed chunked http responses.
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.1, Python 2.6, Python 2.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Andrei Korostelev (1)
Priority: Keywords patch

Created on 2009-09-28 16:40 by Andrei Korostelev, last changed 2009-09-28 16:50 by Andrei Korostelev.

Files
File name Uploaded Description Edit Remove
httplib.python-2.5.diff Andrei Korostelev, 2009-09-28 16:40 Patch for Python-2.5
httplib.python-2.6.2.diff Andrei Korostelev, 2009-09-28 16:45 Patch for Python-2.6.2
httplib.python-3.1.1.diff Andrei Korostelev, 2009-09-28 16:50 Patch for Python-3.1.1
Messages (3)
msg93215 - (view) Author: Andrei Korostelev (Andrei Korostelev) Date: 2009-09-28 16:40
HTTPResponse._read_chunked cannot handle "slightly" ill-formed HTTP
response not ended with 0 chunk-size. I did not make an analysis what
type of webservers generate such responses, but one of them is bing.com
(former msn.com).

Example correct chunked http response:

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

B
first chunk

A
last chunk

0

Example chunked http rsponse not ended with zero length:

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

B
first chunk

A
last chunk


Suggested solution: when an empty line is met where a hexadecimal
chunk-size is expected, treat it as the end of HTTP response. 

--- C:\Python25\Lib\httplib.py.orig	2008-02-12 20:48:24.000000000 +-0200
+++ C:\Python25\Lib\httplib.py.patched	2009-09-28 18:30:33.000000000 +-0200
@@ -542,12 +542,16 @@
         while True:
             if chunk_left is None:
                 line = self.fp.readline()
                 i = line.find(';')
                 if i >= 0:
                     line = line[:i] # strip chunk-extensions
+                # handle ill-formed response not ended with 0 chunk-size
+                line = line.strip()
+                if not line:
+                    break
                 chunk_left = int(line, 16)
                 if chunk_left == 0:
                     break
             if amt is None:
                 value += self._safe_read(chunk_left)
             elif amt < chunk_left:

Attached patches for Python-2.5, Python-2.6 and Python-3.1.
msg93216 - (view) Author: Andrei Korostelev (Andrei Korostelev) Date: 2009-09-28 16:45
Patch for Python-2.6
msg93218 - (view) Author: Andrei Korostelev (Andrei Korostelev) Date: 2009-09-28 16:50
Added patch for python-3.1.1
History
Date User Action Args
2009-09-28 16:50:53Andrei Korostelevsetfiles: + httplib.python-3.1.1.diff

messages: + msg93218
2009-09-28 16:45:45Andrei Korostelevsetfiles: + httplib.python-2.6.2.diff

messages: + msg93216
2009-09-28 16:40:29Andrei Korostelevcreate