classification
Title: non-ascii characters in headers causes TypeError on email.policy.Policy.fold
Type: behavior Stage: resolved
Components: email Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: altvod, barry, cheryl.sabella, matrixise, miss-islington, r.david.murray, rad164
Priority: normal Keywords: patch

Created on 2018-05-15 16:35 by rad164, last changed 2019-05-17 20:54 by cheryl.sabella. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6966 closed licht-t, 2018-05-18 12:03
PR 13391 merged maxking, 2019-05-17 18:48
PR 13393 merged miss-islington, 2019-05-17 19:29
Messages (6)
msg316680 - (view) Author: Rad164 (rad164) Date: 2018-05-15 16:35
Python 3.6.5 has trouble on folding email messages:

Traceback (most recent call last):
  File "emailtest.py", line 7, in <module>
    policy.fold("Subject", msg["Subject"])
  File "/usr/lib/python3.6/email/policy.py", line 183, in fold
    return self._fold(name, value, refold_binary=True)
  File "/usr/lib/python3.6/email/policy.py", line 205, in _fold
    return value.fold(policy=self)
  File "/usr/lib/python3.6/email/headerregistry.py", line 258, in fold
    return header.fold(policy=policy)
  File "/usr/lib/python3.6/email/_header_value_parser.py", line 144, in fold
    return _refold_parse_tree(self, policy=policy)
  File "/usr/lib/python3.6/email/_header_value_parser.py", line 2651, in _refold_parse_tree
    part.ew_combine_allowed, charset)
  File "/usr/lib/python3.6/email/_header_value_parser.py", line 2728, in _fold_as_ew
    first_part = to_encode[:text_space]
TypeError: slice indices must be integers or None or have an __index__ method


The message has non-ascii characters in header and set max_line_length=0, regardless length of the header. Here is the code to reproduce.


from email.message import EmailMessage
from email.policy import default

policy = default.clone(max_line_length=0)
msg = EmailMessage()
msg["Subject"] = "á"
policy.fold("Subject", msg["Subject"])


I first found this issue on Maildir.add, which saves the message to a file without word wrap.
msg316681 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-05-15 17:04
Oh, interesting.

I could argue that that's a missing feature in Python's slice handling.  The value of text_space at that point is '+inf', and I obviously incorrectly assumed that slicing would treat that as if it were [:].  The counter argument, of course, is that inf is a float not an integer.  I suppose we'll have to use sys.maxsize instead.
msg335436 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-02-13 12:58
See this issue as a duplicata of this one:

https://bugs.python.org/issue35985
msg342758 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2019-05-17 19:28
New changeset feac6cd7753425fba006e97e2d9b74a0c0c75894 by R. David Murray (Abhilash Raj) in branch 'master':
bpo-33524: Fix the folding of email header when max_line_length is 0 or None (#13391)
https://github.com/python/cpython/commit/feac6cd7753425fba006e97e2d9b74a0c0c75894
msg342763 - (view) Author: miss-islington (miss-islington) Date: 2019-05-17 20:47
New changeset 5386aaf07835889e90fb33e95b6d37197f8cfea0 by Miss Islington (bot) in branch '3.7':
bpo-33524: Fix the folding of email header when max_line_length is 0 or None (GH-13391)
https://github.com/python/cpython/commit/5386aaf07835889e90fb33e95b6d37197f8cfea0
msg342765 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2019-05-17 20:54
Thank you, @licht-t for the original patch and @maxking for the rebase.  Also, thank you, @r.david.murray for the review and merge.
History
Date User Action Args
2019-05-17 20:54:43cheryl.sabellasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-05-17 20:54:32cheryl.sabellasetnosy: + cheryl.sabella
messages: + msg342765
2019-05-17 20:47:16miss-islingtonsetnosy: + miss-islington
messages: + msg342763
2019-05-17 19:29:24miss-islingtonsetpull_requests: + pull_request13303
2019-05-17 19:28:47r.david.murraysetmessages: + msg342758
2019-05-17 18:48:06maxkingsetpull_requests: + pull_request13300
2019-02-13 13:06:27xtreaklinkissue35985 superseder
2019-02-13 12:58:43matrixisesetnosy: + matrixise
messages: + msg335436
2018-07-25 14:51:41r.david.murraysetnosy: + altvod

versions: + Python 3.7, Python 3.8, - Python 3.6
2018-07-25 14:51:04r.david.murraylinkissue34220 superseder
2018-05-18 12:03:03licht-tsetkeywords: + patch
stage: patch review
pull_requests: + pull_request6625
2018-05-15 17:04:08r.david.murraysetmessages: + msg316681
2018-05-15 16:35:30rad164create