classification
Title: email.parser.Parser hang
Type: security Stage: patch review
Components: email Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Guido, Marcin Niemira, Nam.Nguyen, alex, barry, larry, maxking, miss-islington, ned.deily, r.david.murray, vstinner, xtreak
Priority: normal Keywords: patch, security_issue

Created on 2019-07-01 03:21 by Guido, last changed 2019-09-07 07:08 by larry.

Files
File name Uploaded Description Edit
reproducer.py vstinner, 2019-07-15 09:36
reproducer2.py vstinner, 2019-07-15 09:36
Pull Requests
URL Status Linked Edit
PR 14551 closed n0npax, 2019-07-02 10:40
PR 14794 merged maxking, 2019-07-16 15:08
PR 14816 merged miss-islington, 2019-07-17 16:44
PR 14817 merged miss-islington, 2019-07-17 16:44
PR 14818 merged miss-islington, 2019-07-17 16:45
PR 15430 merged GeeVye, 2019-08-24 04:29
PR 15432 merged GeeVye, 2019-08-24 04:33
PR 15446 merged maxking, 2019-08-24 04:55
Messages (17)
msg346953 - (view) Author: Guido Vranken (Guido) Date: 2019-07-01 03:21
The following will hang, and consume a large amount of memory:

from email.parser import BytesParser, Parser
from email.policy import default
payload = "".join(chr(c) for c in [0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x78, 0x3b, 0x61, 0x72, 0x1b, 0x2a, 0x3d, 0x22, 0x73, 0x4f, 0x27, 0x23, 0x61, 0xff, 0xff, 0x27, 0x5c, 0x22])
Parser(policy=default).parsestr(payload)
msg347014 - (view) Author: Marcin Niemira (Marcin Niemira) Date: 2019-07-01 13:07
Sounds like there is an infinite loop here:


```
Pdb) 
> /usr/lib/python3.7/email/_header_value_parser.py(2370)get_parameter()
-> v.append(token)
(Pdb) 
> /usr/lib/python3.7/email/_header_value_parser.py(2365)get_parameter()
-> while value:
```

the ```v.append(token)``` is just growing with ```ValueTerminal(''), ValueTerminal(''), ValueTerminal('')...```

I'd be happy to try to fix this.
msg347108 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-07-02 06:29
Since the parser could take user input this looks like a security issue to me along with high CPU usage. Feel free to remove the tag if it's not a security issue. Thanks.
msg347263 - (view) Author: Marcin Niemira (Marcin Niemira) Date: 2019-07-04 10:04
I'm terribly sorry, but I feel I won't be able to fix this issue. Sorry for fuss. Closing my PR, because it's broken.
msg347953 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-15 09:36
>>> bytes([0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x78, 0x3b, 0x61, 0x72, 0x1b, 0x2a, 0x3d, 0x22, 0x73, 0x4f, 0x27, 0x23, 0x61, 0xff, 0xff, 0x27, 0x5c, 0x22])
b'Content-Type:x;ar\x1b*="sO\'#a\xff\xff\'\\"'

The following loop of Lib/email/_header_value_parser.py does never stop:

def get_parameter(value):
    """ attribute [section] ["*"] [CFWS] "=" value

    The CFWS is implied by the RFC but not made explicit in the BNF.  This
    simplified form of the BNF from the RFC is made to conform with the RFC BNF
    through some extra checks.  We do it this way because it makes both error
    recovery and working with the resulting parse tree easier.
    """
    ...
    if remainder is not None:
        ...
        while value:
            ...

Attached reproducer.py is code from initial msg346953.

reproducer2.py simplify the input and calls directly get_parameter(). Simplified input string:

   r*="'a'\"
msg348060 - (view) Author: Abhilash Raj (maxking) * (Python committer) Date: 2019-07-17 14:00
I have proposed a PR for this: https://github.com/python/cpython/pull/14794

Reviews are welcome.
msg348064 - (view) Author: Guido Vranken (Guido) Date: 2019-07-17 15:04
I used fuzzing to find this bug. After applying your patch, the infinite loop is gone and it cannot find any other bugs of this nature.
msg348070 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-07-17 16:44
New changeset a4a994bd3e619cbaff97610a1cee8ffa87c672f5 by Barry Warsaw (Abhilash Raj) in branch 'master':
bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794)
https://github.com/python/cpython/commit/a4a994bd3e619cbaff97610a1cee8ffa87c672f5
msg348071 - (view) Author: miss-islington (miss-islington) Date: 2019-07-17 17:02
New changeset 391511ccaaf0050970dfbe95bf2df1bcf6c33440 by Miss Islington (bot) in branch '3.7':
bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794)
https://github.com/python/cpython/commit/391511ccaaf0050970dfbe95bf2df1bcf6c33440
msg348072 - (view) Author: miss-islington (miss-islington) Date: 2019-07-17 17:14
New changeset 6816ca30af7705db691343100e696ea3d8f447d5 by Miss Islington (bot) in branch '3.8':
bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794)
https://github.com/python/cpython/commit/6816ca30af7705db691343100e696ea3d8f447d5
msg348850 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-08-01 12:10
3.5 also seems to be affected. git cherry pick works and the patch fixes the problem so I assume the backport would be straightforward. Since 3.5 is open for security fixes with 3.5.8 as next release I am adding Larry.

$ git cherry-pick a4a994b
Performing inexact rename detection: 100% (566740/566740), done.
[detached HEAD 9877e9283c] bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794)
 Author: Abhilash Raj <maxking@users.noreply.github.com>
 Date: Wed Jul 17 09:44:27 2019 -0700
 3 files changed, 12 insertions(+)
 create mode 100644 bpo-37461.1Ahz7O.rst">Misc/NEWS.d/next/Security/2019-07-16-08-11-00.bpo-37461.1Ahz7O.rst
msg348867 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-01 16:36
New changeset 1789bbdd3e03023951a39933ef12dee0a03be616 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794) (GH-14817)
https://github.com/python/cpython/commit/1789bbdd3e03023951a39933ef12dee0a03be616
msg350344 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-24 04:30
New changeset 799e4e527019d9429fdef12d08a0c03b08a1fb59 by Ned Deily (GeeTransit) in branch '3.7':
[3.7] bpo-37461: Fix typo (inifite -> infinite) (GH-15430)
https://github.com/python/cpython/commit/799e4e527019d9429fdef12d08a0c03b08a1fb59
msg350345 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-24 04:33
New changeset f1f9c0c532089824791cfc18e6d6f29e1cd62596 by Ned Deily (GeeTransit) in branch '3.6':
[3.6] bpo-37461: Fix typo (inifite -> infinite) (#15432)
https://github.com/python/cpython/commit/f1f9c0c532089824791cfc18e6d6f29e1cd62596
msg350346 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-24 04:36
xtreak, would you be willing to make the PR for 3.5?  That will make it easier for Larry to decide whether to include it in a 3.5 security release.
msg350348 - (view) Author: Abhilash Raj (maxking) * (Python committer) Date: 2019-08-24 04:56
I manually created a backport PR for 3.5 and added Larry as a reviewer.

https://github.com/python/cpython/pull/15446
msg351289 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2019-09-07 07:08
New changeset c28e4a5160d3283b12514c7c28ed6e0a2a52271a by larryhastings (Abhilash Raj) in branch '3.5':
[3.5] bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794) (#15446)
https://github.com/python/cpython/commit/c28e4a5160d3283b12514c7c28ed6e0a2a52271a
History
Date User Action Args
2019-09-07 07:08:55larrysetmessages: + msg351289
2019-08-24 04:56:40maxkingsetmessages: + msg350348
2019-08-24 04:55:44maxkingsetpull_requests: + pull_request15138
2019-08-24 04:36:26ned.deilysetmessages: + msg350346
2019-08-24 04:33:39ned.deilysetmessages: + msg350345
2019-08-24 04:33:24GeeVyesetpull_requests: + pull_request15137
2019-08-24 04:30:28ned.deilysetmessages: + msg350344
2019-08-24 04:29:50GeeVyesetpull_requests: + pull_request15136
2019-08-01 16:36:50ned.deilysetnosy: + ned.deily
messages: + msg348867
2019-08-01 12:10:44xtreaksetnosy: + larry

messages: + msg348850
versions: + Python 3.5
2019-07-17 17:14:11miss-islingtonsetmessages: + msg348072
2019-07-17 17:02:11miss-islingtonsetnosy: + miss-islington
messages: + msg348071
2019-07-17 16:45:28miss-islingtonsetpull_requests: + pull_request14612
2019-07-17 16:44:55miss-islingtonsetpull_requests: + pull_request14611
2019-07-17 16:44:48miss-islingtonsetpull_requests: + pull_request14610
2019-07-17 16:44:41barrysetmessages: + msg348070
2019-07-17 15:04:05Guidosetmessages: + msg348064
2019-07-17 14:00:29maxkingsetmessages: + msg348060
2019-07-16 15:08:58maxkingsetpull_requests: + pull_request14588
2019-07-15 09:36:17vstinnersetfiles: + reproducer2.py
2019-07-15 09:36:10vstinnersetfiles: + reproducer.py
nosy: + vstinner
messages: + msg347953

2019-07-15 09:22:54vstinnersettype: crash -> security
2019-07-15 02:28:05Nam.Nguyensetnosy: + Nam.Nguyen
2019-07-14 20:22:02alexsetnosy: + alex
2019-07-04 10:04:43Marcin Niemirasetmessages: + msg347263
2019-07-02 10:40:08n0npaxsetkeywords: + patch
stage: patch review
pull_requests: + pull_request14369
2019-07-02 06:29:42xtreaksetversions: + Python 3.6, Python 3.7, Python 3.8
nosy: + xtreak

messages: + msg347108

keywords: + security_issue
2019-07-01 13:07:10Marcin Niemirasetnosy: + Marcin Niemira
messages: + msg347014
2019-07-01 03:32:43xtreaksetnosy: + maxking
2019-07-01 03:21:13Guidocreate