This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Email parser creates a message object that can't be flattened as bytes.
Type: Stage: resolved
Components: email Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: barry, msapiro, r.david.murray
Priority: normal Keywords: patch

Created on 2020-01-18 19:48 by msapiro, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18056 closed python-dev, 2020-01-19 00:52
Messages (7)
msg360249 - (view) Author: Mark Sapiro (msapiro) * (Python triager) Date: 2020-01-18 19:48
This is similar to but is the opposite behavior. In that issue, the message couldn't be flattened as a string but could be flattened as bytes. Here, the message can be flattened as a string but can't be flattened as bytes.

The original message was created by an arguably defective email client that quoted a message containing a utf8 encoded RIGHT SINGLE QUOTATION MARK and utf-8 encoded separately the three bytes resulting in `â**` instead of `’`. That's not really relevant but is just to show how such a message can be generated.

The following interactive python session shows the issue.

>>> import email
>>> msg = email.message_from_string("""From Sat Jan 18 04:09:40 2020
... From:
... To:
... Subject: Century Dates for Insurance purposes
... Date: Fri, 17 Jan 2020 20:09:26 -0800
... Message-ID: <>
... MIME-Version: 1.0
... Content-Type: text/plain; charset="utf-8"
... Content-Transfer-Encoding: 8bit
...        Thursday-Monday will cover both days of staging and then storing goods
...        post-century. I think thatâ**s the way to go.
... """)
>>> msg.as_bytes()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/email/", line 178, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/local/lib/python3.7/email/", line 116, in flatten
  File "/usr/local/lib/python3.7/email/", line 181, in _write
  File "/usr/local/lib/python3.7/email/", line 214, in _dispatch
  File "/usr/local/lib/python3.7/email/", line 432, in _handle_text
  File "/usr/local/lib/python3.7/email/", line 249, in _handle_text
  File "/usr/local/lib/python3.7/email/", line 155, in _write_lines
  File "/usr/local/lib/python3.7/email/", line 406, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe2' in position 33: ordinal not in range(128)
msg360320 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2020-01-20 15:42
Since you parsed it as a string it is not really legitimate to serialize it as bytes.  (That will work if the input message only contains ascii, but not if it contains unicode).  You'll get the same error if you replace the garbage with the "’".  Using errors=replace is not crazy, but it hides the actual problem.  Let's see what other people think :)

In theory you could "fix" this by encoding the unicode using the charset specified by the container.  I have no idea how complicated it will be do that, and it would be a new feature: parsing strings is specified to only work with ASCII input, currently.

I put "fix" in quotes, because even if you make text parts like this example work, you still can't handle non-text 8bit mime parts.  Is it worth doing anyway?

Really, message_as_string and friends should just be avoided entirely, maybe even deprecated.
msg360331 - (view) Author: Mark Sapiro (msapiro) * (Python triager) Date: 2020-01-20 19:59
This came about because of an actual situation in a Mailman 3 installation. I can't say for sure what the actual original message looked like, but it was received by Mailman's LMTP server and parsed with email.message_from_bytes(), so it clearly wasn't exactly like the message excerpt I posted in the report above. However, All I had to go by was the message object from the shunted pickle file created as a result of the exception.

The message was processed by Mailman, but when Mailman's handler pipeline attempted to save it for the digest, it calls an instance of mailbox.MMDF to add the message to the mailbox accumulating messages for the digest, and that in turn calls the flatten method of an email.generator.BytesGenerator instance. and that's where the exception was thrown.

Perhaps the suggested patch in doesn't address every possible case, and it can result in a slightly garbled message due to replacing 'invalid' characters, but in my case at least, it is much preferable to the alternative.
msg361359 - (view) Author: Mark Sapiro (msapiro) * (Python triager) Date: 2020-02-04 17:06
Other Mailman3 installations are also encountering this issue. See
msg361360 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2020-02-04 17:58
If we can get an actual reproducer using message_as_bytes I'd feel more comfortable with the fix.  I worry that there is some other bug this is exposing that should be fixed instead.
msg361361 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2020-02-04 17:59
msg361469 - (view) Author: Mark Sapiro (msapiro) * (Python triager) Date: 2020-02-06 04:30
I've researched this further, and I know how this happens. The original message contains a text/html part (in my case, the only part) which contains a base64 or quoted-printable body which when decoded contains non-ascii. It is parsed correctly by email.message_from_bytes.

It is then processed by Mailman's content filtering which retrieves html payload via

    part.get_payload(decode=True).decode(ctype, errors='replace'))

where part is the text/html part and ctype is 'utf-8' in this case. It then uses elinks, lynx or some other configured command to convert the html payload to plain text and that plain text still contains non-ascii.

It then replaces the payload and sets the content type via

    del part['content-transfer-encoding']

And this results in a message which can't be flattened as_bytes.

The issue is set_payload() should encode the payload appropriately and in fact, it does if an appropriate charset is given, so this is our error in not providing a charset= argument to set_payload.

Closing this and the corresponding PR.
Date User Action Args
2022-04-11 14:59:25adminsetgithub: 83565
2020-02-06 04:30:03msapirosetstatus: open -> closed

messages: + msg361469
stage: patch review -> resolved
2020-02-04 17:59:26r.david.murraysetmessages: + msg361361
2020-02-04 17:58:59r.david.murraysetmessages: + msg361360
2020-02-04 17:06:33msapirosetmessages: + msg361359
2020-01-20 19:59:33msapirosetmessages: + msg360331
2020-01-20 15:42:27r.david.murraysetmessages: + msg360320
2020-01-19 06:05:48msapirosetversions: + Python 3.8, Python 3.9
2020-01-19 00:52:59python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request17449
2020-01-18 19:48:43msapirocreate