classification
Title: [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler
Type: security Stage: resolved
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anselmo Melo, bc, koobs, larry, mgorny, miss-islington, ned.deily, tapakund, vstinner, ware
Priority: normal Keywords: patch

Created on 2020-01-30 15:11 by vstinner, last changed 2020-06-20 08:33 by larry. This issue is now closed.

Files
File name Uploaded Description Edit
bench_parser.py vstinner, 2020-03-30 17:21
bench_parser2.py bc, 2020-04-02 00:46 Benchmark
Pull Requests
URL Status Linked Edit
PR 18284 merged vstinner, 2020-01-30 15:20
PR 19291 closed miss-islington, 2020-04-02 00:52
PR 19292 closed miss-islington, 2020-04-02 00:52
PR 19296 merged miss-islington, 2020-04-02 01:39
PR 19297 merged miss-islington, 2020-04-02 01:39
PR 19299 closed tapakund, 2020-04-02 07:09
PR 19301 closed tapakund, 2020-04-02 09:13
PR 19302 closed tapakund, 2020-04-02 10:27
PR 19304 merged vstinner, 2020-04-02 11:59
PR 19305 merged vstinner, 2020-04-02 12:05
Messages (13)
msg361072 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 15:11
Copy of an email received on the Python Security Response team, 9 days ago. I consider that it's not worth it to have an embargo on this vulnerability, so I make it public.

Hi there,

I believe I've found a denial-of-service (DoS) bug in
urllib.request.AbstractBasicAuthHandler. To start, I'm operating on some
background information from this document: HTTP authentication
<https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication>. The bug
itself is a ReDoS <https://www.regular-expressions.info/redos.html> bug
causing catastrophic backtracking. To reproduce the issue we can use the
following code:

from urllib.request import AbstractBasicAuthHandler
auth_handler = AbstractBasicAuthHandler()
auth_handler.http_error_auth_reqed(
    'www-authenticate',
    'unused',
    'unused',
    {
        'www-authenticate': 'Basic ' + ',' * 64 + ' ' + 'foo' + ' ' +
'realm'
    }
)

The issue itself is in the following regular expression:

rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+'
                'realm=(["\']?)([^"\']*)\\2', re.I)

In particular, the (?:.*,)* portion. Since "." and "," overlap and there
are nested quantifiers we can cause catastrophic backtracking by repeating
a comma. Note that since AbstractBasicAuthHandler is vulnerable, then both
HTTPBasicAuthHandler and ProxyBasicAuthHandler are as well because they
call http_error_auth_reqed. Building from the HTTP authentication document
above, this means a server can send a specially crafted header along with
an HTTP 401 or HTTP 407 and cause a DoS on the client.

I won't speculate on the severity of the issue too much - you will surely
understand the impact better than I will. Although, the fact that this is
client-side as opposed to server-side appears to reduce the severity,
however the fact that it's a security-sensitive context (HTTP
authentication) may raise the severity.

One possible fix would be changing the rx expression to the following:

rx = re.compile('(?:[^,]*,)*[ \t]*([^ \t]+)[ \t]+'
                'realm=(["\']?)([^"\']*)\\2', re.I)

This removes the character overlap in the nested quantifier and thus
negates the catastrophic backtracking.

Let me know if you have any questions or what the next steps are from here.
Thanks for supporting Python security!

-- 
Matt Schwager
msg361073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 15:30
I added this vulnerability to the following page to track fixes in all Python supported branches:
https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html
msg361081 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 22:33
CVE-2020-8492 has been assigned to this vulnerability:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=2020-8492
msg361343 - (view) Author: Ben Caller (bc) * Date: 2020-02-04 10:53
Isn't this a duplicate of bpo-38826 ?
msg364996 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-25 16:19
> Isn't this a duplicate of bpo-38826 ?

Oh right. I marked it as a duplicate of this issue.
msg365335 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-30 17:21
bench_parser.py: Benchmark for AbstractBasicAuthHandler.http_error_auth_reqed().
msg365538 - (view) Author: Ben Caller (bc) * Date: 2020-04-02 00:24
Instead of

repeat_10_3 = 'Basic ' + ', ' * (10 ** 3) + simple

in the benchmark, try

repeat_10_3 = 'Basic ' + ', ' * (10 ** 3) + 'A'
msg365541 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-02 00:44
Ooooh, I see. I didn't measure the performance of the right header. I re-run a benchmark using the HTTP header (repeat=15):

    header = 'Basic ' + ', ' * 15 + 'A'

Now I see a major performance difference. Comparison between master ("ref") and PR 18284 ("fix"):

Mean +- std dev: [ref] 88.9 ms +- 2.4 ms -> [fix] 17.5 us +- 0.7 us: 5083.23x faster (-100%)

So the worst case is now way faster: more than 5000x faster!

It's even possible to go up to repeat=10**6 characters, it still takes less than 1 seconds: 412 ms +- 19 ms.

On the master branch, repeat=20 already takes around 3 seconds... The slowdown is exponential with repeat increase.
msg365542 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-02 00:52
New changeset 0b297d4ff1c0e4480ad33acae793fbaf4bf015b4 by Victor Stinner in branch 'master':
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284)
https://github.com/python/cpython/commit/0b297d4ff1c0e4480ad33acae793fbaf4bf015b4
msg365578 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-02 10:15
New changeset ea9e240aa02372440be8024acb110371f69c9d41 by Miss Islington (bot) in branch '3.8':
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284) (GH-19296)
https://github.com/python/cpython/commit/ea9e240aa02372440be8024acb110371f69c9d41
msg365579 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-02 10:16
New changeset b57a73694e26e8b2391731b5ee0b1be59437388e by Miss Islington (bot) in branch '3.7':
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284) (GH-19297)
https://github.com/python/cpython/commit/b57a73694e26e8b2391731b5ee0b1be59437388e
msg365663 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-03 01:16
New changeset 69cdeeb93e0830004a495ed854022425b93b3f3e by Victor Stinner in branch '3.6':
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284) (GH-19304)
https://github.com/python/cpython/commit/69cdeeb93e0830004a495ed854022425b93b3f3e
msg371921 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-06-20 06:27
New changeset 37fe316479e0b6906a74b0c0a5e495c55037fdfd by Victor Stinner in branch '3.5':
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284) (#19305)
https://github.com/python/cpython/commit/37fe316479e0b6906a74b0c0a5e495c55037fdfd
History
Date User Action Args
2020-06-20 08:33:12larrysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-06-20 08:30:06koobssetnosy: + koobs
2020-06-20 06:27:09larrysetnosy: + larry
messages: + msg371921
2020-04-03 01:16:03ned.deilysetnosy: + ned.deily
messages: + msg365663
2020-04-02 12:05:26vstinnersetpull_requests: + pull_request18667
2020-04-02 11:59:15vstinnersetpull_requests: + pull_request18666
2020-04-02 11:43:04vstinnersetversions: - Python 2.7
2020-04-02 10:27:09tapakundsetpull_requests: + pull_request18664
2020-04-02 10:16:20vstinnersetmessages: + msg365579
2020-04-02 10:15:59vstinnersetmessages: + msg365578
2020-04-02 09:13:52tapakundsetpull_requests: + pull_request18663
2020-04-02 07:09:03tapakundsetnosy: + tapakund
pull_requests: + pull_request18661
2020-04-02 01:39:24miss-islingtonsetpull_requests: + pull_request18656
2020-04-02 01:39:16miss-islingtonsetpull_requests: + pull_request18655
2020-04-02 00:52:45miss-islingtonsetpull_requests: + pull_request18651
2020-04-02 00:52:38miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request18650
2020-04-02 00:52:23vstinnersetmessages: + msg365542
2020-04-02 00:46:16bcsetfiles: - bench_parser2.py
2020-04-02 00:46:00bcsetfiles: + bench_parser2.py
2020-04-02 00:44:09vstinnersetmessages: + msg365541
2020-04-02 00:25:00bcsetfiles: + bench_parser2.py

messages: + msg365538
2020-03-30 17:21:05vstinnersetfiles: + bench_parser.py

messages: + msg365335
2020-03-25 16:19:57vstinnersetmessages: + msg364996
2020-03-25 16:19:39vstinnerlinkissue38826 superseder
2020-03-04 23:06:47waresetnosy: + ware
2020-03-02 09:23:41mgornysetnosy: + mgorny
2020-02-04 10:53:53bcsetnosy: + bc
messages: + msg361343
2020-01-31 17:24:57Anselmo Melosetnosy: + Anselmo Melo
2020-01-30 22:33:12vstinnersetmessages: + msg361081
title: [security] Denial of service in urllib.request.AbstractBasicAuthHandler -> [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler
2020-01-30 15:30:40vstinnersetmessages: + msg361073
2020-01-30 15:20:34vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request17659
2020-01-30 15:11:29vstinnercreate