CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@ #80923

hokousya · 2019-04-27T12:30:17Z

BPO	36742
Nosy	@orsenthil, @vstinner, @larryhastings, @benjaminp, @ned-deily, @ezio-melotti, @ambv, @zooba, @stratakis, @miss-islington, @tirkarthi, @ret2libc
PRs	bpo-36742: Fixes handling of pre-normalization characters in urlsplit() #13017 [3.7] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13023 [3.6] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13024 [2.7] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13025 [3.5] bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) #13042 bpo-36742: Corrects fix to handle decomposition in usernames #13812 [3.7] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) #13813 [3.6] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) #13814 [2.7] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) #13815 [2.7] bpo-36742: Fix urlparse.urlsplit() error message for Unicode URL #13937 [3.5] bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) (GH-13814) #14772

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-05-01.15:04:11.989>
created_at = <Date 2019-04-27.12:30:16.902>
labels = ['type-security', '3.7', '3.8', 'expert-unicode', 'release-blocker']
title = 'CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@'
updated_at = <Date 2019-09-07.06:33:27.509>
user = 'https://bugs.python.org/hokousya'

bugs.python.org fields:

activity = <Date 2019-09-07.06:33:27.509>
actor = 'larry'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-05-01.15:04:11.989>
closer = 'steve.dower'
components = ['Unicode']
creation = <Date 2019-04-27.12:30:16.902>
creator = 'hokousya'
dependencies = []
files = []
hgrepos = []
issue_num = 36742
keywords = ['patch', '3.5regression', '3.6regression', '3.7regression']
message_count = 23.0
messages = ['340983', '341006', '341092', '341125', '341150', '341151', '341171', '341206', '341207', '341208', '341212', '341282', '344595', '344596', '344597', '344601', '344623', '344973', '344981', '345116', '345218', '347880', '351285']
nosy_count = 13.0
nosy_names = ['orsenthil', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'lukasz.langa', 'steve.dower', 'cstratak', 'miss-islington', 'xtreak', 'hokousya', 'rschiron']
pr_nums = ['13017', '13023', '13024', '13025', '13042', '13812', '13813', '13814', '13815', '13937', '14772']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue36742'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

hokousya · 2019-04-27T12:30:17Z

urllib.parse.urlsplit raises an exception for an url including a non-ascii hostname in NFKD form and a port number.

example:
>>> urlsplit('http://\u30d5\u309a:80')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ito/.maltybrew/deen/lib/python3.7/urllib/parse.py", line 437, in urlsplit
    _checknetloc(netloc)
  File "/Users/ito/.maltybrew/deen/lib/python3.7/urllib/parse.py", line 407, in _checknetloc
    "characters under NFKC normalization")
ValueError: netloc 'プ:80' contains invalid characters under NFKC normalization
>>> urlsplit('http://\u30d5\u309a')
SplitResult(scheme='http', netloc='プ', path='', query='', fragment='')
>>> urlsplit(unicodedata.normalize('NFKC', 'http://\u30d5\u309a:80'))
SplitResult(scheme='http', netloc='プ:80', path='', query='', fragment='')

I believe this behavior was introduced at Python 3.7.3. Python 3.7.2 doesn't raise any exception for these lines.

tirkarthi · 2019-04-27T18:05:17Z

This could be due to bpo-36216.

zooba · 2019-04-29T15:36:10Z

Yes, it's due to that. I guess we need to do netloc.rpartition(':') like we currently do for '@' in _checknetloc.

Promoting to release blocker and security issue to match the original issue. I can't get to this today, but I should be able to at the PyCon sprints next week if nobody else gets it sooner.

zooba · 2019-04-29T22:39:14Z

I found the time to get the first patch. Hopefully backports to 3.6 and 3.7 are easy, but I think 2.7 will take manual steps.

Chihiro Ito - if you have other test scenarios, it would be great if you could try them out with the fix in PR 13017. It should be easy enough to copy into your installed Python.

zooba · 2019-04-30T12:03:19Z

New changeset d537ab0 by Steve Dower in branch 'master':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
d537ab0

miss-islington · 2019-04-30T12:21:05Z

New changeset 4d723e7 by Miss Islington (bot) in branch '3.7':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
4d723e7

hokousya · 2019-05-01T00:16:57Z

I have confirmed that all of my app's test cases have passed.

What I've done:

Installed Python 3.7.3.
Replaced urllib/parse.py with the one from 781ffb1.
Ran my app's test cases.

Thank you for the quick fix!

zooba · 2019-05-01T15:00:33Z

New changeset 98a4dce by Steve Dower in branch '2.7':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
98a4dce

zooba · 2019-05-01T15:04:12Z

I'll leave the 3.6 backport in Ned's hands and close this issue.

tirkarthi · 2019-05-01T15:10:50Z

I'll leave the 3.6 backport in Ned's hands and close this issue.

3.5 was added as an affected version and seems the original fix was merged to 3.5 too. 3.4 is EoL so is it worthy of backporting to 3.5? I guess the backport would not have merge conflicts and is straightforward.

zooba · 2019-05-01T15:59:45Z

Yes, you're right. I'll do that port as well.

ned-deily · 2019-05-02T16:02:40Z

New changeset e5f9f4a by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) (GH-13024)
e5f9f4a

ambv · 2019-06-04T15:56:01Z

New changeset 8d0ef0b by Łukasz Langa (Steve Dower) in branch 'master':
bpo-36742: Corrects fix to handle decomposition in usernames (bpo-13812)
8d0ef0b

orsenthil · 2019-06-04T16:10:59Z

Thanks for this engagement and pull requests, Steve.
Thanks for reviews Karthikeyan.

miss-islington · 2019-06-04T16:15:27Z

New changeset 250b62a by Miss Islington (bot) in branch '3.7':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812)
250b62a

zooba · 2019-06-04T16:40:21Z

New changeset f61599b by Steve Dower in branch '2.7':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812)
f61599b

ned-deily · 2019-06-04T18:44:00Z

New changeset fd1771d by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) (GH-13814)
fd1771d

vstinner · 2019-06-07T17:17:04Z

CVE-2019-10160 has been assigned by Red Hat to this flaw.

ret2libc · 2019-06-07T17:59:18Z

The fix for python-2.7 (https://github.com/python/cpython/pull/13815/files#diff-b577545d73dd0cdb2c337a4c5f89e1d7R183) causes errors when netloc contains characters that can't be encoded by 'ascii' codec.

You can see it by doing:
>>> netloc = u'example.com\uFF03@bing.com'
>>> raise ValueError(u"netloc '" + netloc + u"' contains invalid characters under NFKC normalization")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: <exception str() failed>
>>> str(netloc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\uff03' in position 11: ordinal not in range(128)

I suggest we use repr(netloc) instead of netloc in the ValueError message.

ret2libc · 2019-06-10T10:12:30Z

CVE-2019-10160 has been assigned by Red Hat to this flaw.

For clarity, CVE-2019-10160 has been assigned to the bug introduced with the fix for the functional regression mentioned in this bug, and not to the bug itself explained in the first comment. See https://bugzilla.redhat.com/show_bug.cgi?id=1718388 for more details about it.

vstinner · 2019-06-11T10:45:39Z

New changeset 2b57847 by Victor Stinner in branch '2.7':
[2.7] bpo-36742: Fix urlparse.urlsplit() error message for Unicode URL (GH-13937)
2b57847

larryhastings · 2019-07-14T08:16:23Z

New changeset 4655d57 by larryhastings (Steve Dower) in branch '3.5':
bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) (bpo-13042)
4655d57

larryhastings · 2019-09-07T06:33:27Z

New changeset 095373c by larryhastings (Victor Stinner) in branch '3.5':
bpo-36742: Corrects fix to handle decomposition in usernames (GH-13812) (GH-13814) (bpo-14772)
095373c

hokousya mannequin added 3.7 (EOL) end of life topic-unicode type-bug An unexpected behavior, bug, or error labels Apr 27, 2019

zooba added 3.8 only security fixes release-blocker type-security A security issue and removed type-bug An unexpected behavior, bug, or error labels Apr 29, 2019

zooba self-assigned this Apr 29, 2019

zooba closed this as completed May 1, 2019

vstinner changed the title ~~urlsplit doesn't accept a NFKD hostname with a port number~~ CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@ Jun 7, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@ #80923

CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@ #80923

hokousya mannequin commented Apr 27, 2019

hokousya mannequin commented Apr 27, 2019

tirkarthi commented Apr 27, 2019

zooba commented Apr 29, 2019

zooba commented Apr 29, 2019

zooba commented Apr 30, 2019

miss-islington commented Apr 30, 2019

hokousya mannequin commented May 1, 2019

zooba commented May 1, 2019

zooba commented May 1, 2019

tirkarthi commented May 1, 2019

zooba commented May 1, 2019

ned-deily commented May 2, 2019

ambv commented Jun 4, 2019

orsenthil commented Jun 4, 2019

miss-islington commented Jun 4, 2019

zooba commented Jun 4, 2019

ned-deily commented Jun 4, 2019

vstinner commented Jun 7, 2019

ret2libc mannequin commented Jun 7, 2019

ret2libc mannequin commented Jun 10, 2019

vstinner commented Jun 11, 2019

larryhastings commented Jul 14, 2019

larryhastings commented Sep 7, 2019

CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@ #80923

CVE-2019-10160: urlsplit NFKD normalization vulnerability in user:password@ #80923

Comments

hokousya mannequin commented Apr 27, 2019

hokousya mannequin commented Apr 27, 2019

tirkarthi commented Apr 27, 2019

zooba commented Apr 29, 2019

zooba commented Apr 29, 2019

zooba commented Apr 30, 2019

miss-islington commented Apr 30, 2019

hokousya mannequin commented May 1, 2019

zooba commented May 1, 2019

zooba commented May 1, 2019

tirkarthi commented May 1, 2019

zooba commented May 1, 2019

ned-deily commented May 2, 2019

ambv commented Jun 4, 2019

orsenthil commented Jun 4, 2019

miss-islington commented Jun 4, 2019

zooba commented Jun 4, 2019

ned-deily commented Jun 4, 2019

vstinner commented Jun 7, 2019

ret2libc mannequin commented Jun 7, 2019

ret2libc mannequin commented Jun 10, 2019

vstinner commented Jun 11, 2019

larryhastings commented Jul 14, 2019

larryhastings commented Sep 7, 2019