classification
Title: httplib: header parsing is unlimited
Type: resource usage Stage: needs patch
Components: Library (Lib) Versions: Python 3.2, Python 3.1, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Arfrever, Lukasa, barry, benjamin.peterson, christian.heimes, georg.brandl, larry, nailor, pitrou, python-dev
Priority: release blocker Keywords: patch

Created on 2012-09-25 10:25 by christian.heimes, last changed 2014-03-12 10:25 by Lukasa.

Files
File name Uploaded Description Edit
issue16037_py27.patch nailor, 2013-02-23 19:45
issue16037_py32.patch nailor, 2013-02-23 19:52
issue16037_py26.patch nailor, 2013-09-04 10:19 review
issue16037_py27_v2.patch nailor, 2013-09-04 10:20 review
issue16037_py32_v2.patch nailor, 2013-09-04 10:20 review
issue16037_py32_v3.patch nailor, 2013-10-25 16:39 review
Messages (22)
msg171240 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-09-25 10:25
The httplib module / package can read arbitrary amounts of data from its socket when it's parsing the HTTP header. This may lead to issues when a user connects to a broken HTTP server or something that isn't a HTTP at all. The issue can be broken up into two parts: parsing the HTTP status line parsing and parsing the remaining HTTP headers.

Reading and parsing of the HTTP status line is already limited in Python 3.x. Python 2.7 and lower may read arbitrary amounts of bytes from the socket until it finds a newline char. The small patch below is a backport of the Python 3.x behavior to 2.7:

--- a/Lib/httplib.py
+++ b/Lib/httplib.py
@@ -362,7 +362,9 @@

     def _read_status(self):
         # Initialize with Simple-Response defaults
-        line = self.fp.readline()
+        line = self.fp.readline(_MAXLINE + 1)
+        if len(line) > _MAXLINE:
+            raise LineTooLong("header line")
         if self.debuglevel > 0:
             print "reply:", repr(line)
         if not line:


Both Python 2 and Python 3 accept an unlimited count of HTTP headers with a maximum length of 64k each. As headers are accumulated in an list it may consume lots of memory. I suggest that we limit the maximum amount of HTTP header lines to a sane value. How does 100 sound to you?
msg171250 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-25 11:29
New changeset 8a22a2804a66 by Christian Heimes in branch '2.7':
Issue #16037: Limit httplib's _read_status() function to work around broken
http://hg.python.org/cpython/rev/8a22a2804a66
msg171251 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-09-25 11:31
The readline() limitation in _read_status() was added at some point in the 3.2 line. Python 3.1 has an unlimited readline().
msg171258 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-25 12:30
100 headers sounds more than enough for everybody.
msg182194 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-02-15 23:58
CVE-2013-1752  Unbound readline() DoS vulnerabilities in Python stdlib
msg182803 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2013-02-23 19:45
Here's a patch that limits the headers to 100. If more than _MAXHEADERS headers are read, this raises exception TooMuchHeaders.

The patch is for 2.7, I'll cook one for 3.2 too.
msg182805 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2013-02-23 19:52
...and here's the patch for 3.2
msg185055 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-03-23 14:45
Not blocking 2.7.4 as discussed on mailing list.
msg187276 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2013-04-18 18:08
Patches LGTM but I suggest TooManyHeaders instead of TooMuchHeaders.  I've tried the 3.2 patch against the latest default repo on Windows Vista and it applies cleanly.  All tests passed so looks as if this could be committed.
msg196862 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-09-03 18:35
blocker for 2.6.9
msg196898 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2013-09-04 10:19
Reworded TooMuch to TooMany and made a patch for 2.6 too (2.7 didn't apply cleanly there)
msg198610 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-09-29 17:24
As we discussed in other issues regarding the similar problem, I don't really want to introduce a new exception in a point release of 2.6.  Is there any reason not to just raise HTTPException with the error message text?  Code that has to work across multiple 2.6.X versions won't be able to import the new exception, and thus cannot rely on it anyway.

If you agree, I'll make that change when I apply this patch.
msg198618 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2013-09-29 17:55
I'm fine with not introducing a new exception for 2.6 (or any other version for that matter), so go for it :)
msg198619 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-09-29 17:58
I'm just going to go ahead and commit this patch to 2.6 with the change I mentioned.  Does anything else need to be done for 2.6?
msg198620 - (view) Author: Roundup Robot (python-dev) Date: 2013-09-29 18:01
New changeset 582e5072ff89 by Barry Warsaw in branch '2.6':
- Issue #16037: HTTPMessage.readheaders() raises an HTTPException when more
http://hg.python.org/cpython/rev/582e5072ff89
msg198621 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-09-29 18:02
Thanks!
msg200349 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-19 01:22
Ping.  Please fix before "beta 1".
msg201162 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2013-10-24 18:47
Patch for py32 applies cleanly on 3.4 too, this should be good to go
msg201255 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2013-10-25 16:39
Third version of the 3.2 patch, this time with documentation of the exception TooManyHeaders
msg201424 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-27 06:39
New changeset e445d02e5306 by Georg Brandl in branch '3.3':
Issue #16037: HTTPMessage.readheaders() raises an HTTPException when more than
http://hg.python.org/cpython/rev/e445d02e5306
msg201429 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-10-27 06:45
Also merged to default.
msg213240 - (view) Author: Cory Benfield (Lukasa) * Date: 2014-03-12 10:25
I presume Barry's disinclination to merge this to 2.6 with a new exception applies equally to 2.7, which is why this hasn't been merged to 2.7 yet?

I'm happy to review an updated 2.7 patch that raises an HTTPException if that's what we need to keep this moving.
History
Date User Action Args
2014-03-12 10:25:54Lukasasetmessages: + msg213240
2014-03-12 10:23:06Lukasasetnosy: + Lukasa
2014-02-03 15:49:34BreamoreBoysetnosy: - BreamoreBoy
2013-10-27 06:45:59georg.brandlsetmessages: + msg201429
versions: - Python 3.3, Python 3.4
2013-10-27 06:39:04python-devsetmessages: + msg201424
2013-10-25 16:39:10nailorsetfiles: + issue16037_py32_v3.patch

messages: + msg201255
2013-10-24 18:47:36nailorsetmessages: + msg201162
2013-10-19 01:22:47larrysetmessages: + msg200349
2013-09-29 19:11:02Arfreversettitle: httplib: header parsing is not unlimited -> httplib: header parsing is unlimited
2013-09-29 18:02:43barrysetmessages: + msg198621
versions: - Python 2.6
2013-09-29 18:01:31python-devsetmessages: + msg198620
2013-09-29 17:58:39barrysetmessages: + msg198619
2013-09-29 17:55:20nailorsetmessages: + msg198618
2013-09-29 17:24:58barrysetmessages: + msg198610
2013-09-15 19:42:12Arfreversettitle: httplib: header parsing is not delimited -> httplib: header parsing is not unlimited
versions: + Python 3.1
2013-09-04 10:20:07nailorsetfiles: + issue16037_py32_v2.patch
2013-09-04 10:20:03nailorsetfiles: + issue16037_py27_v2.patch
2013-09-04 10:19:58nailorsetfiles: + issue16037_py26.patch

messages: + msg196898
2013-09-03 18:35:18barrysetpriority: critical -> release blocker

messages: + msg196862
2013-04-18 18:08:45BreamoreBoysetnosy: + BreamoreBoy
messages: + msg187276
2013-03-23 14:45:23benjamin.petersonsetpriority: release blocker -> critical

messages: + msg185055
2013-02-23 19:52:37nailorsetfiles: + issue16037_py32.patch

messages: + msg182805
2013-02-23 19:45:33nailorsetfiles: + issue16037_py27.patch

nosy: + nailor
messages: + msg182803

keywords: + patch
2013-02-22 23:33:45Arfreversetnosy: + Arfrever
2013-02-20 22:26:33barrysetnosy: + barry

versions: + Python 2.6
2013-02-15 23:58:46christian.heimessetmessages: + msg182194
2013-02-04 17:12:24christian.heimessetpriority: critical -> release blocker
nosy: + benjamin.peterson, georg.brandl, larry
2013-01-20 14:39:08christian.heimessetpriority: normal -> critical
assignee: christian.heimes
stage: needs patch
versions: + Python 3.4
2012-09-25 12:30:37pitrousetnosy: + pitrou
messages: + msg171258
2012-09-25 11:31:23christian.heimessetmessages: + msg171251
2012-09-25 11:29:54python-devsetnosy: + python-dev
messages: + msg171250
2012-09-25 10:25:22christian.heimescreate