classification
Title: CVE-2019-18348: CRLF injection via the host part of the url passed to urlopen()
Type: security Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: benjamin.peterson Nosy List: Anselmo Melo, b1tninja, benjamin.peterson, cstratak, epicfaace, gregory.p.smith, kim, mcepl, miss-islington, ned.deily, rschiron, tapakund, vstinner, ware, xtreak
Priority: release blocker Keywords: patch

Created on 2019-10-24 07:51 by rschiron, last changed 2020-04-02 08:26 by tapakund. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18995 merged epicfaace, 2020-03-14 14:54
PR 19000 merged miss-islington, 2020-03-14 18:56
PR 19001 merged miss-islington, 2020-03-14 18:56
PR 19002 merged miss-islington, 2020-03-14 18:56
PR 19052 merged mcepl, 2020-03-18 00:41
PR 19231 closed tapakund, 2020-03-30 15:49
PR 19300 open tapakund, 2020-04-02 08:26
Messages (12)
msg355294 - (view) Author: Riccardo Schirone (rschiron) Date: 2019-10-24 07:51
Copy-pasted from https://bugs.python.org/issue30458#msg347282

================
The commit b7378d77289c911ca6a0c0afaf513879002df7d5 is incomplete: it doesn't seem to check for control characters in the "host" part of the URL, only in the "path" part of the URL. Example:
---
try:
    from urllib import request as urllib_request
except ImportError:
    import urllib2 as urllib_request
import socket
def bug(*args):
    raise Exception(args)
# urlopen() must not call create_connection()
socket.create_connection = bug
urllib_request.urlopen('http://127.0.0.1\r\n\x20hihi\r\n :11211')
---

The URL comes from the first message of this issue:
https://bugs.python.org/issue30458#msg294360

Development branches 2.7 and master produce a similar output:
---
Traceback (most recent call last):
 ...
Exception: (('127.0.0.1\r\n hihi\r\n ', 11211), ..., None)
---

So urllib2/urllib.request actually does a real network connection (DNS query), whereas it should reject control characters in the "host" part of the URL.

***

A second problem comes into the game. Some C libraries like glibc strip the end of the hostname (strip at the first newline character) and so HTTP Header injection is still possible is this case:
https://bugzilla.redhat.com/show_bug.cgi?id=1673465

***

According to the RFC 3986, the "host" grammar doesn't allow any control character, it looks like:

   host          = IP-literal / IPv4address / reg-name

   ALPHA (letters)
   DIGIT (decimal digits)
   unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
      pct-encoded = "%" HEXDIG HEXDIG
      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="
   reg-name      = *( unreserved / pct-encoded / sub-delims )

   IP-literal    = "[" ( IPv6address / IPvFuture  ) "]"
   IPvFuture     = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
   IPv6address   =                            6( h16 ":" ) ls32
                 /                       "::" 5( h16 ":" ) ls32
                 / [               h16 ] "::" 4( h16 ":" ) ls32
                 / [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
                 / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
                 / [ *3( h16 ":" ) h16 ] "::"    h16 ":"   ls32
                 / [ *4( h16 ":" ) h16 ] "::"              ls32
                 / [ *5( h16 ":" ) h16 ] "::"              h16
                 / [ *6( h16 ":" ) h16 ] "::"
   h16           = 1*4HEXDIG
   ls32          = ( h16 ":" h16 ) / IPv4address
   IPv4address   = dec-octet "." dec-octet "." dec-octet "." dec-octet
================


CVE-2019-18348 was assigned to this flaw, which is similar to CVE-2019-9947 and CVE-2019-9740 but it is about the *host* part of a url.
msg357073 - (view) Author: Justin Capella (b1tninja) * Date: 2019-11-20 13:52
Can't see the specifics of that "restricted" redhat bug, but this was interesting bug and I wanted to ask if perhaps the domain in such cases should be IDN / punycoded ://xn--n28h.ws/ for example is ://💩.la
msg357442 - (view) Author: Riccardo Schirone (rschiron) Date: 2019-11-25 15:38
The glibc issue mentioned in the first comment is CVE-2016-10739 .
msg362353 - (view) Author: Matej Cepl (mcepl) * Date: 2020-02-20 21:41
Just to say this is reproducible only on rather old enterprise Linux distributions, where CVE-2016-10739 bug in glibc has not been fixed. I believe it means RHEL-6, SUSE SLE-10, 11, 12 (not sure whether it applies to some old Debian as well).
msg364190 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-14 18:56
New changeset 9165addc22d05e776a54319a8531ebd0b2fe01ef by Ashwin Ramaswami in branch 'master':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995)
https://github.com/python/cpython/commit/9165addc22d05e776a54319a8531ebd0b2fe01ef
msg364191 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-14 19:02
Thanks for the PR Ashwin!
msg364192 - (view) Author: miss-islington (miss-islington) Date: 2020-03-14 19:13
New changeset 34f85af3229f86c004a954c3f261ceea1f5e9f95 by Miss Islington (bot) in branch '3.7':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995)
https://github.com/python/cpython/commit/34f85af3229f86c004a954c3f261ceea1f5e9f95
msg364193 - (view) Author: miss-islington (miss-islington) Date: 2020-03-14 19:13
New changeset ff69c9d12c1b06af58e5eae5db4630cedd94740e by Miss Islington (bot) in branch '3.8':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995)
https://github.com/python/cpython/commit/ff69c9d12c1b06af58e5eae5db4630cedd94740e
msg364207 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-03-14 22:35
New changeset 83fc70159b24f5b11a5ef87c9b05c2cf4c7faeba by Miss Islington (bot) in branch '3.6':
bpo-38576: Disallow control characters in hostnames in http.client (GH-18995) (GH-19002)
https://github.com/python/cpython/commit/83fc70159b24f5b11a5ef87c9b05c2cf4c7faeba
msg364208 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-15 00:59
If anyone cares about 2.7, the *final* release is coming up in a few weeks.  They'll need to figure out what it looks like there and get a 2.7 PR reviewed by the release manager.
msg364499 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-03-18 04:09
marking as a 2.7 release blocker just to get benjamin's RM attention before the final 2.7.
msg364584 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-03-19 01:35
New changeset e176e0c105786e9f476758eb5438c57223b65e7f by Matěj Cepl in branch '2.7':
[2.7] closes bpo-38576: Disallow control characters in hostnames in http.client. (GH-19052)
https://github.com/python/cpython/commit/e176e0c105786e9f476758eb5438c57223b65e7f
History
Date User Action Args
2020-04-02 08:26:46tapakundsetpull_requests: + pull_request18662
2020-03-30 15:49:31tapakundsetnosy: + tapakund

pull_requests: + pull_request18591
2020-03-19 01:35:47benjamin.petersonsetstatus: open -> closed

messages: + msg364584
stage: patch review -> resolved
2020-03-18 04:09:51gregory.p.smithsetstatus: closed -> open
priority: high -> release blocker

assignee: gregory.p.smith -> benjamin.peterson
versions: - Python 3.5
nosy: + benjamin.peterson

messages: + msg364499
stage: resolved -> patch review
2020-03-18 00:41:45mceplsetpull_requests: + pull_request18403
2020-03-15 00:59:12gregory.p.smithsetstatus: open -> closed
versions: - Python 3.6, Python 3.7, Python 3.8, Python 3.9
messages: + msg364208

resolution: fixed
stage: patch review -> resolved
2020-03-14 22:35:55ned.deilysetnosy: + ned.deily
messages: + msg364207
2020-03-14 19:13:36miss-islingtonsetmessages: + msg364193
2020-03-14 19:13:02miss-islingtonsetmessages: + msg364192
2020-03-14 19:02:04gregory.p.smithsetassignee: gregory.p.smith
messages: + msg364191
2020-03-14 18:56:36miss-islingtonsetpull_requests: + pull_request18350
2020-03-14 18:56:28miss-islingtonsetpull_requests: + pull_request18349
2020-03-14 18:56:21miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request18348
2020-03-14 18:56:15gregory.p.smithsetmessages: + msg364190
2020-03-14 14:54:12epicfaacesetkeywords: + patch
nosy: + epicfaace

pull_requests: + pull_request18342
stage: needs patch -> patch review
2020-02-28 18:16:39waresetnosy: + ware
2020-02-20 21:41:12mceplsetmessages: + msg362353
2019-12-10 16:24:42mceplsetnosy: + mcepl
2019-12-09 03:08:06gregory.p.smithsetpriority: normal -> high
2019-11-25 15:38:02rschironsetmessages: + msg357442
2019-11-20 13:52:37b1tninjasetnosy: + b1tninja
messages: + msg357073
2019-11-20 12:04:45kimsetnosy: + kim
2019-11-19 14:21:31vstinnersetcomponents: + Library (Lib)
versions: + Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
2019-10-30 22:11:08Anselmo Melosetnosy: + Anselmo Melo
2019-10-24 16:55:24gregory.p.smithsetstage: needs patch
2019-10-24 16:55:12gregory.p.smithsetnosy: + gregory.p.smith
2019-10-24 13:27:28cstrataksetnosy: + cstratak
2019-10-24 10:47:17vstinnersettitle: CVE-2019-18348 CRLF injection via the host part of the url passed to urlopen() -> CVE-2019-18348: CRLF injection via the host part of the url passed to urlopen()
2019-10-24 07:55:28xtreaksetnosy: + vstinner, xtreak
2019-10-24 07:51:18rschironcreate