classification
Title: email.policy.SMTP.fold() issue for long filenames with spaces
Type: Stage: resolved
Components: email Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, jhillacre, r.david.murray
Priority: normal Keywords: patch

Created on 2017-06-27 19:58 by jhillacre, last changed 2017-12-04 00:50 by r.david.murray. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2622 closed jhillacre, 2017-07-07 18:14
PR 3488 merged r.david.murray, 2017-09-11 02:07
Messages (5)
msg297072 - (view) Author: Joel Hillacre (jhillacre) * Date: 2017-06-27 19:58
Found an issue in email.policy.SMTP.fold on "Content-Disposition" headers for long filenames with spaces. Managed to simply the issue to a test case as follows:

>>> from email.policy import SMTP; SMTP.fold('Content-Disposition', 'attachment; filename="{}"'.format('1234 67890' + '1234567890' * 6))

Below are the tracebacks various python versions produces for this code. I think that the difference between python 3.5 - 3.6 and 3.7 are related to the fix in bpo-30532.

Python 3.6.1 (default, Apr  7 2017, 09:32:32) & Python 3.5.3 (default, Jan 17 2017, 14:34:36)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python3.6/email/policy.py", line 183, in fold
    return self._fold(name, value, refold_binary=True)
  File "/usr/lib64/python3.6/email/policy.py", line 213, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib64/python3.6/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 1228, in _fold
    rest._fold(folded)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 347, in _fold
    if folded.append_if_fits(part):
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 149, in append_if_fits
    token._fold(self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 338, in _fold
    if folded.append_if_fits(part, tstr):
... last 4 lines repeated 482 more times ...
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 149, in append_if_fits
    token._fold(self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 325, in _fold
    tstr = str(part)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 196, in __str__
    return ''.join(str(x) for x in self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 196, in <genexpr>
    return ''.join(str(x) for x in self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 196, in __str__
    return ''.join(str(x) for x in self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 196, in <genexpr>
    return ''.join(str(x) for x in self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 196, in __str__
    return ''.join(str(x) for x in self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 196, in <genexpr>
    return ''.join(str(x) for x in self)
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 633, in __str__
    return quote_string(''.join(str(x) for x in self))
  File "/usr/lib64/python3.6/email/_header_value_parser.py", line 633, in <genexpr>
    return quote_string(''.join(str(x) for x in self))
RecursionError: maximum recursion depth exceeded while getting the str of an object

Python 3.7.0a0 (heads/master:e613e6add5, Jun 27 2017, 12:17:18) 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/joel/PycharmProjects/cpython/Lib/email/policy.py", line 183, in fold
    return self._fold(name, value, refold_binary=True)
  File "/home/joel/PycharmProjects/cpython/Lib/email/policy.py", line 213, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/home/joel/PycharmProjects/cpython/Lib/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 1226, in _fold
    rest._fold(folded)
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 345, in _fold
    if folded.append_if_fits(part):
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 145, in append_if_fits
    ws = token.pop_leading_fws()
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 272, in pop_leading_fws
    return self[0].pop_leading_fws()
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 272, in pop_leading_fws
    return self[0].pop_leading_fws()
  File "/home/joel/PycharmProjects/cpython/Lib/email/_header_value_parser.py", line 270, in pop_leading_fws
    if self[0].token_type == 'fws':
IndexError: list index out of range

Note that the following example without a space does not cause a traceback:

Python 3.7.0a0 (heads/master:e613e6add5, Jun 27 2017, 12:17:18) 
>>> from email.policy import SMTP; SMTP.fold('Content-Disposition', 'attachment; filename="{}"'.format('1234567890' + '1234567890' * 6))
'Content-Disposition: attachment;\r\n filename="1234567890123456789012345678901234567890123456789012345678901234567890"\r\n'
msg299437 - (view) Author: Joel Hillacre (jhillacre) * Date: 2017-07-28 20:53
Just wanted to ping this and see if there was someone available to review my PR associated with this issue.
msg301797 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-09-10 00:42
Thanks for the PR, but I've managd to finish my rewrite of the folder so that it not only doesn't traceback on this, but correctly folds it.  I'll probably post the PR tomorrow.
msg302664 - (view) Author: Joel Hillacre (jhillacre) * Date: 2017-09-20 22:31
I just took a quick peek at your PR to see if a similar test exists from this bug and it is looking good. Thanks for your work.
msg307531 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-12-04 00:50
The PR has been committed.
History
Date User Action Args
2017-12-04 00:50:36r.david.murraysetstatus: open -> closed
versions: - Python 3.5
messages: + msg307531

resolution: fixed
stage: patch review -> resolved
2017-09-20 22:31:42jhillacresetmessages: + msg302664
2017-09-11 02:07:55r.david.murraysetkeywords: + patch
stage: patch review
pull_requests: + pull_request3479
2017-09-10 00:42:29r.david.murraysetmessages: + msg301797
2017-07-28 20:53:14jhillacresetmessages: + msg299437
2017-07-07 18:14:05jhillacresetpull_requests: + pull_request2688
2017-06-27 19:59:17jhillacresettitle: email.policy.SMTP.fold() issue with long filenames with spaces -> email.policy.SMTP.fold() issue for long filenames with spaces
2017-06-27 19:58:41jhillacrecreate