classification
Title: CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@
Type: security Stage: resolved
Components: Unicode Versions: Python 3.8, Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: benjamin.peterson, cstratak, ezio.melotti, hokousya, larry, lukasz.langa, miss-islington, ned.deily, orsenthil, rschiron, steve.dower, vstinner, xtreak
Priority: release blocker Keywords: 3.5regression, 3.6regression, 3.7regression, patch

Created on 2019-04-27 12:30 by hokousya, last changed 2019-09-07 06:33 by larry. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13017 merged steve.dower, 2019-04-29 22:36
PR 13023 merged miss-islington, 2019-04-30 12:03
PR 13024 merged miss-islington, 2019-04-30 12:03
PR 13025 merged steve.dower, 2019-04-30 12:11
PR 13042 merged steve.dower, 2019-05-01 16:03
PR 13812 merged steve.dower, 2019-06-04 15:31
PR 13813 merged miss-islington, 2019-06-04 15:55
PR 13814 merged miss-islington, 2019-06-04 15:56
PR 13815 merged steve.dower, 2019-06-04 16:04
PR 13937 merged vstinner, 2019-06-10 10:00
PR 14772 merged vstinner, 2019-07-14 09:54
Messages (23)
msg340983 - (view) Author: Chihiro Ito (hokousya) Date: 2019-04-27 12:30
urllib.parse.urlsplit raises an exception for an url including a non-ascii hostname in NFKD form and a port number.

example:
>>> urlsplit('http://\u30d5\u309a:80')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ito/.maltybrew/deen/lib/python3.7/urllib/parse.py", line 437, in urlsplit
    _checknetloc(netloc)
  File "/Users/ito/.maltybrew/deen/lib/python3.7/urllib/parse.py", line 407, in _checknetloc
    "characters under NFKC normalization")
ValueError: netloc 'プ:80' contains invalid characters under NFKC normalization
>>> urlsplit('http://\u30d5\u309a')
SplitResult(scheme='http', netloc='プ', path='', query='', fragment='')
>>> urlsplit(unicodedata.normalize('NFKC', 'http://\u30d5\u309a:80'))
SplitResult(scheme='http', netloc='プ:80', path='', query='', fragment='')

I believe this behavior was introduced at Python 3.7.3. Python 3.7.2 doesn't raise any exception for these lines.
msg341006 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-04-27 18:05
This could be due to issue36216.
msg341092 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-04-29 15:36
Yes, it's due to that. I guess we need to do netloc.rpartition(':') like we currently do for '@' in _checknetloc.

Promoting to release blocker and security issue to match the original issue. I can't get to this today, but I should be able to at the PyCon sprints next week if nobody else gets it sooner.
msg341125 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-04-29 22:39
I found the time to get the first patch. Hopefully backports to 3.6 and 3.7 are easy, but I think 2.7 will take manual steps.

Chihiro Ito - if you have other test scenarios, it would be great if you could try them out with the fix in PR 13017. It should be easy enough to copy into your installed Python.
msg341150 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-04-30 12:03
New changeset d537ab0ff9767ef024f26246899728f0116b1ec3 by Steve Dower in branch 'master':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
https://github.com/python/cpython/commit/d537ab0ff9767ef024f26246899728f0116b1ec3
msg341151 - (view) Author: miss-islington (miss-islington) Date: 2019-04-30 12:21
New changeset 4d723e76e1ad17e9e7d5e828e59bb47e76f2174b by Miss Islington (bot) in branch '3.7':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
https://github.com/python/cpython/commit/4d723e76e1ad17e9e7d5e828e59bb47e76f2174b
msg341171 - (view) Author: Chihiro Ito (hokousya) Date: 2019-05-01 00:16
I have confirmed that all of my app's test cases have passed.

What I've done:
1. Installed Python 3.7.3.
2. Replaced urllib/parse.py with the one from 781ffb1.
3. Ran my app's test cases.

Thank you for the quick fix!
msg341206 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-05-01 15:00
New changeset 98a4dcefbbc3bce5ab07e7c0830a183157250259 by Steve Dower in branch '2.7':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
https://github.com/python/cpython/commit/98a4dcefbbc3bce5ab07e7c0830a183157250259
msg341207 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-05-01 15:04
I'll leave the 3.6 backport in Ned's hands and close this issue.
msg341208 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-05-01 15:10
> I'll leave the 3.6 backport in Ned's hands and close this issue.

3.5 was added as an affected version and seems the original fix was merged to 3.5 too. 3.4 is EoL so is it worthy of backporting to 3.5? I guess the backport would not have merge conflicts and is straightforward.
msg341212 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-05-01 15:59
Yes, you're right. I'll do that port as well.
msg341282 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-02 16:02
New changeset e5f9f4adb95233c66578e6f7ea176687af2f78ca by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) (GH-13024)
https://github.com/python/cpython/commit/e5f9f4adb95233c66578e6f7ea176687af2f78ca
msg344595 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-06-04 15:56
New changeset 8d0ef0b5edeae52960c7ed05ae8a12388324f87e by Łukasz Langa (Steve Dower) in branch 'master':
bpo-36742: Corrects fix to handle decomposition in usernames (#13812)
https://github.com/python/cpython/commit/8d0ef0b5edeae52960c7ed05ae8a12388324f87e
msg344596 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2019-06-04 16:10
Thanks for this engagement and pull requests, Steve.
Thanks for reviews Karthikeyan.
msg344597 - (view) Author: miss-islington (miss-islington) Date: 2019-06-04 16:15
New changeset 250b62acc59921d399f0db47db3b462cd6037e09 by Miss Islington (bot) in branch '3.7':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812)
https://github.com/python/cpython/commit/250b62acc59921d399f0db47db3b462cd6037e09
msg344601 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-04 16:40
New changeset f61599b050c621386a3fc6bc480359e2d3bb93de by Steve Dower in branch '2.7':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812)
https://github.com/python/cpython/commit/f61599b050c621386a3fc6bc480359e2d3bb93de
msg344623 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-04 18:44
New changeset fd1771dbdd28709716bd531580c40ae5ed814468 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) (GH-13814)
https://github.com/python/cpython/commit/fd1771dbdd28709716bd531580c40ae5ed814468
msg344973 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-07 17:17
CVE-2019-10160 has been assigned by Red Hat to this flaw.
msg344981 - (view) Author: Riccardo Schirone (rschiron) Date: 2019-06-07 17:59
The fix for python-2.7 (https://github.com/python/cpython/pull/13815/files#diff-b577545d73dd0cdb2c337a4c5f89e1d7R183) causes errors when netloc contains characters that can't be encoded by 'ascii' codec.

You can see it by doing:
>>> netloc = u'example.com\uFF03@bing.com'
>>> raise ValueError(u"netloc '" + netloc + u"' contains invalid characters under NFKC normalization")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: <exception str() failed>
>>> str(netloc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\uff03' in position 11: ordinal not in range(128)

I suggest we use `repr(netloc)` instead of `netloc` in the ValueError message.
msg345116 - (view) Author: Riccardo Schirone (rschiron) Date: 2019-06-10 10:12
> CVE-2019-10160 has been assigned by Red Hat to this flaw.

For clarity, CVE-2019-10160 has been assigned to the bug introduced with the fix for the functional regression mentioned in this bug, and not to the bug itself explained in the first comment. See https://bugzilla.redhat.com/show_bug.cgi?id=1718388 for more details about it.
msg345218 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-11 10:45
New changeset 2b578479b96aa3deeeb8bac313a02b5cf3cb1aff by Victor Stinner in branch '2.7':
[2.7] bpo-36742: Fix urlparse.urlsplit() error message for Unicode URL (GH-13937)
https://github.com/python/cpython/commit/2b578479b96aa3deeeb8bac313a02b5cf3cb1aff
msg347880 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2019-07-14 08:16
New changeset 4655d576141ee56a69d2052431c636858fcb916a by larryhastings (Steve Dower) in branch '3.5':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) (#13042)
https://github.com/python/cpython/commit/4655d576141ee56a69d2052431c636858fcb916a
msg351285 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2019-09-07 06:33
New changeset 095373c32d16df575ba5fcb5f44bf44119b26193 by larryhastings (Victor Stinner) in branch '3.5':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) (GH-13814) (#14772)
https://github.com/python/cpython/commit/095373c32d16df575ba5fcb5f44bf44119b26193
History
Date User Action Args
2019-09-07 06:33:27larrysetmessages: + msg351285
2019-07-14 09:54:17vstinnersetpull_requests: + pull_request14565
2019-07-14 08:16:23larrysetmessages: + msg347880
2019-06-11 10:45:39vstinnersetmessages: + msg345218
2019-06-10 13:38:14cstrataksetnosy: + cstratak
2019-06-10 10:12:29rschironsetmessages: + msg345116
2019-06-10 10:00:52vstinnersetpull_requests: + pull_request13804
2019-06-07 17:59:17rschironsetnosy: + rschiron
messages: + msg344981
2019-06-07 17:17:04vstinnersetmessages: + msg344973
2019-06-07 17:16:53vstinnersettitle: urlsplit doesn't accept a NFKD hostname with a port number -> CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@
2019-06-04 18:44:00ned.deilysetmessages: + msg344623
2019-06-04 16:40:20steve.dowersetmessages: + msg344601
2019-06-04 16:15:27miss-islingtonsetmessages: + msg344597
2019-06-04 16:10:59orsenthilsetnosy: + orsenthil
messages: + msg344596
2019-06-04 16:04:20steve.dowersetpull_requests: + pull_request13701
2019-06-04 15:56:05miss-islingtonsetpull_requests: + pull_request13700
2019-06-04 15:56:00lukasz.langasetmessages: + msg344595
2019-06-04 15:55:52miss-islingtonsetpull_requests: + pull_request13699
2019-06-04 15:31:37steve.dowersetpull_requests: + pull_request13698
2019-05-02 16:02:39ned.deilysetmessages: + msg341282
2019-05-01 16:03:35steve.dowersetpull_requests: + pull_request12961
2019-05-01 15:59:45steve.dowersetmessages: + msg341212
2019-05-01 15:10:49xtreaksetmessages: + msg341208
2019-05-01 15:04:11steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg341207

stage: patch review -> resolved
2019-05-01 15:00:32steve.dowersetmessages: + msg341206
2019-05-01 00:16:57hokousyasetmessages: + msg341171
2019-04-30 12:21:05miss-islingtonsetnosy: + miss-islington
messages: + msg341151
2019-04-30 12:11:08steve.dowersetpull_requests: + pull_request12947
2019-04-30 12:03:22miss-islingtonsetpull_requests: + pull_request12946
2019-04-30 12:03:19steve.dowersetmessages: + msg341150
2019-04-30 12:03:15miss-islingtonsetpull_requests: + pull_request12945
2019-04-29 22:39:13steve.dowersetassignee: steve.dower
messages: + msg341125
2019-04-29 22:36:03steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request12940
2019-04-29 15:36:09steve.dowersetpriority: normal -> release blocker

type: behavior -> security
versions: + Python 2.7, Python 3.5, Python 3.6, Python 3.8
keywords: + 3.5regression, 3.6regression, 3.7regression
nosy: + ned.deily, larry, lukasz.langa, benjamin.peterson

messages: + msg341092
2019-04-27 18:05:17xtreaksetnosy: + xtreak, steve.dower
messages: + msg341006
2019-04-27 12:33:32hokousyasettype: behavior
2019-04-27 12:30:16hokousyacreate