Title: urllib http client possible infinite loop on a 100 Continue response
Type: security Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: christian.heimes, gen-xu, gregory.p.smith, leveryd, lukasz.langa, miss-islington, ned.deily
Priority: normal Keywords: patch

Created on 2021-05-03 17:13 by leveryd, last changed 2021-05-08 12:33 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25916 merged gen-xu, 2021-05-05 11:13
PR 25931 merged miss-islington, 2021-05-05 22:42
PR 25932 merged miss-islington, 2021-05-05 22:42
PR 25933 merged miss-islington, 2021-05-05 22:42
PR 25934 merged miss-islington, 2021-05-05 22:43
PR 25935 merged miss-islington, 2021-05-05 22:43
Messages (16)
msg392825 - (view) Author: guangli dong (leveryd) Date: 2021-05-03 17:13
if a client request a http/https/ftp service which is controlled by attacker, attacker can make this client hang forever, event client has set "timeout" argument.

maybe this client also will consume more and more memory. i does not test on this conclusion.
import urllib.request

req = urllib.request.Request('')
response = urllib.request.urlopen(req, timeout=1)
# coding:utf-8
from socket import *
from multiprocessing import *
from time import sleep

def dealWithClient(newSocket,destAddr):
    recvData = newSocket.recv(1024)
    newSocket.send(b"""HTTP/1.1 100 OK\n""")

    while True:
        # recvData = newSocket.recv(1024)

        if len(recvData)>0:
            # print('recv[%s]:%s'%(str(destAddr), recvData))

    # newSocket.close()

def main():

    serSocket = socket(AF_INET, SOCK_STREAM)
    serSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR  , 1)
    localAddr = ('', 8085)

        while True:
            newSocket,destAddr = serSocket.accept()

            client = Process(target=dealWithClient, args=(newSocket,destAddr))


if __name__ == '__main__':
msg393004 - (view) Author: Gen Xu (gen-xu) * Date: 2021-05-05 11:15
Added a possible PR. Review will be appreicated.
msg393005 - (view) Author: Gen Xu (gen-xu) * Date: 2021-05-05 11:30
Looks like it is caused by the httplib not limiting total header size after receiving 100. Added a counter for that to be same size as _MAXLINE=65536.
msg393037 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-05-05 20:12
The bug: Our http client can get stuck infinitely reading len(line) < 64k lines after receiving a '100 Continue' http response.  So yes, this could lead to our client being a bandwidth sink for anyone in control of a server.

Clear issue: That's a denial of network bandwidth and the denial of service in terms of CPU needed to process read and skip such lines.  The infinite lines are size bounded and are not buffered so there is no memory based DoS.

Maybe issue: If a the underlying socket has a timeout set on it, it can be used to prevent the timeout from triggering by sending a line more often than the timeout.  this is a denial of service by making a http client connection that an author may have assumed would timeout based on their socket.setdefaulttimeout() settings hang forever.

I expect there are plenty of other ways to accomplish the latter in our http client code though.  Ex: A regular response with a huge content length where one byte is transmitted occasionally could also effectively accomplish that.  The stdlib http stack doesn't have its own overall http transaction timeout as a feature.
msg393048 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-05-05 22:50
Thanks guangli dong (leveryd)!

This is in and the 3.10-3.6 PRs should automerge (thru 3.9) after the CI runs, or be merged by the release managers (3.6-3.8).
msg393050 - (view) Author: miss-islington (miss-islington) Date: 2021-05-05 23:06
New changeset ea9327036680acc92d9f89eaf6f6a54d2f8d78d9 by Miss Islington (bot) in branch '3.9':
bpo-44022: Fix http client infinite line reading (DoS) after a HTTP 100 Continue (GH-25916)
msg393051 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-05-05 23:14
New changeset 60ba0b68470a584103e28958d91e93a6db37ec92 by Miss Islington (bot) in branch '3.10':
bpo-44022: Fix http client infinite line reading (DoS) after a HTTP 100 Continue (GH-25916) (GH-25931)
msg393073 - (view) Author: guangli dong (leveryd) Date: 2021-05-06 08:28
can you assign "cve" for this security bug?

i will review the patch later.
msg393074 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-05-06 08:37
http.server is out of scope for CVEs. The module is not designed for security-sensitive usage and explicitly documented as insecure and not suitable for production use:

> Warning: http.server is not recommended for production. It only implements basic security checks.
msg393076 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2021-05-06 08:52
New changeset f396864ddfe914531b5856d7bf852808ebfc01ae by Miss Islington (bot) in branch '3.8':
bpo-44022: Fix http client infinite line reading (DoS) after a HTTP 100 Continue (GH-25916) (#25933)
msg393079 - (view) Author: guangli dong (leveryd) Date: 2021-05-06 10:09
@Christian Heimes 

this bug is about "urllib" client library, the key point is not "http.server" module.
msg393110 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-05-06 17:05
New changeset f68d2d69f1da56c2aea1293ecf93ab69a6010ad7 by Miss Islington (bot) in branch '3.6':
bpo-44022: Fix http client infinite line reading (DoS) after a HTTP 100 Continue (GH-25916) (GH-25935)
msg393113 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-05-06 17:10
New changeset 078b146f062d212919d0ba25e34e658a8234aa63 by Miss Islington (bot) in branch '3.7':
bpo-44022: Fix http client infinite line reading (DoS) after a HTTP 100 Continue (GH-25916) (GH-25934)
msg393137 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-05-06 19:27
If anyone wants a CVE for it, that's up to them.  This bug is in the CPython http.client module which is what urllib uses for http/https.  I'd rate it low severity.  A malicious server can hold a http connection from this library open as a network traffic sink.  There are other ways to do that.  ex: Just use omit a content-length header in a server response and start streaming an infinite response.

The difference in this case being that since the data is thrown away, it isn't going to result in memory exhaustion and kill the unfortunate process as trying to read an infinite response would.  That's the primary DoS potential from my point of view.
msg393194 - (view) Author: guangli dong (leveryd) Date: 2021-05-07 17:04
@Gregory P. Smith

yes, i agree that there are many other ways to make "urllib" or "httplib" such http client hang, because "timeout" is not global read timeout, this "timeout" has effects when every "read socket" operation.

why you think it will not result in memory exhaustion?

the "hlist" list will not be more and more larger? i use "top" command to observe, and find the "" process's memory is more and more larger slowly.

while True:
    line = self.fp.readline(_MAXLINE + 1)

the last, would you mind remove "100 Continue" in this bug title? i think it will maybe make others misunderstand that this bug only occur when response status code is "100".
msg393195 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-05-07 17:39 is a Python 2 concept.  Python 2 is end of life. no longer tracks issues with its code.  I don't doubt that Python 2.7 has bugs.  As a matter of policy, we don't care -  Python 3.6 as that is the oldest branch still open for security fixes.

The PRs associated with this issue fixed a codepath in Python 3 that only happened after a '100' response.  That codepath did not accumulate headers:

            if status != CONTINUE:
            # skip the header from the 100 response
            while True:
                skip = self.fp.readline(_MAXLINE + 1)
                if len(skip) > _MAXLINE:
                    raise LineTooLong("header line")
                skip = skip.strip()
                if not skip:

CONTINUE = 100; meaning that loop only runs after receiving what appears to be a 100 continue response.  And it does not accumulate data.

There is no `hlist` in the original pre-fix Python 3.6+ code.  Nor any header accumulation caused by this the talking to as described in this issues opening message.
Date User Action Args
2021-05-08 12:33:20lukasz.langasetmessages: - msg393236
2021-05-08 04:39:16leverydsetmessages: + msg393236
2021-05-07 17:39:21gregory.p.smithsetmessages: + msg393195
2021-05-07 17:04:05leverydsetmessages: + msg393194
2021-05-06 19:40:31ned.deilysetversions: + Python 3.8, Python 3.9, Python 3.10
2021-05-06 19:27:01gregory.p.smithsetmessages: + msg393137
2021-05-06 17:10:49ned.deilysetstage: commit review -> resolved
versions: + Python 3.6, Python 3.7, - Python 3.8, Python 3.9, Python 3.10, Python 3.11
2021-05-06 17:10:21ned.deilysetmessages: + msg393113
2021-05-06 17:05:53ned.deilysetnosy: + ned.deily
messages: + msg393110
2021-05-06 10:09:33leverydsetmessages: + msg393079
2021-05-06 08:52:42lukasz.langasetversions: + Python 3.8
2021-05-06 08:52:35lukasz.langasetnosy: + lukasz.langa
messages: + msg393076
2021-05-06 08:37:25christian.heimessetnosy: + christian.heimes
messages: + msg393074
2021-05-06 08:28:00leverydsetmessages: + msg393073
2021-05-05 23:14:36gregory.p.smithsetmessages: + msg393051
2021-05-05 23:06:00miss-islingtonsetmessages: + msg393050
2021-05-05 22:50:44gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg393048

stage: patch review -> commit review
2021-05-05 22:43:11miss-islingtonsetpull_requests: + pull_request24602
2021-05-05 22:43:05miss-islingtonsetpull_requests: + pull_request24601
2021-05-05 22:42:59miss-islingtonsetpull_requests: + pull_request24600
2021-05-05 22:42:54miss-islingtonsetpull_requests: + pull_request24599
2021-05-05 22:42:49miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24598
2021-05-05 20:12:08gregory.p.smithsetmessages: + msg393037
2021-05-05 20:03:09gregory.p.smithsetassignee: gregory.p.smith
title: "urllib" will result to deny of service -> urllib http client possible infinite loop on a 100 Continue response

nosy: + gregory.p.smith
versions: + Python 3.9, Python 3.10, Python 3.11
2021-05-05 11:30:46gen-xusetmessages: + msg393005
versions: - Python 3.7
2021-05-05 11:15:17gen-xusetmessages: + msg393004
2021-05-05 11:13:12gen-xusetkeywords: + patch
nosy: + gen-xu

pull_requests: + pull_request24585
stage: patch review
2021-05-03 17:13:03leverydcreate