This is similar to https://bugs.python.org/issue32330 but is the opposite behavior. In that issue, the message couldn't be flattened as a string but could be flattened as bytes. Here, the message can be flattened as a string but can't be flattened as bytes.

The original message was created by an arguably defective email client that quoted a message containing a utf8 encoded RIGHT SINGLE QUOTATION MARK and utf-8 encoded separately the three bytes resulting in â** instead of ’. That's not really relevant but is just to show how such a message can be generated.

The following interactive python session shows the issue.

>>> import email
>>> msg = email.message_from_string("""From user@example.com Sat Jan 18 04:09:40 2020
... From: user@example.com
... To: recip@example.com
... Subject: Century Dates for Insurance purposes
... Date: Fri, 17 Jan 2020 20:09:26 -0800
... Message-ID: <75ccdd72-d71c-407c-96bd-0ca95abcfa03@email.android.com>
... MIME-Version: 1.0
... Content-Type: text/plain; charset="utf-8"
... Content-Transfer-Encoding: 8bit
... 
...        Thursday-Monday will cover both days of staging and then storing goods
...        post-century. I think thatâ**s the way to go.
... 
... """)
>>> msg.as_bytes()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/email/message.py", line 178, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/local/lib/python3.7/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/local/lib/python3.7/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/local/lib/python3.7/email/generator.py", line 214, in _dispatch
    meth(msg)
  File "/usr/local/lib/python3.7/email/generator.py", line 432, in _handle_text
    super(BytesGenerator,self)._handle_text(msg)
  File "/usr/local/lib/python3.7/email/generator.py", line 249, in _handle_text
    self._write_lines(payload)
  File "/usr/local/lib/python3.7/email/generator.py", line 155, in _write_lines
    self.write(line)
  File "/usr/local/lib/python3.7/email/generator.py", line 406, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe2' in position 33: ordinal not in range(128)
>>> 

Since you parsed it as a string it is not really legitimate to serialize it as bytes. (That will work if the input message only contains ascii, but not if it contains unicode). You'll get the same error if you replace the garbage with the "’". Using errors=replace is not crazy, but it hides the actual problem. Let's see what other people think :)

In theory you could "fix" this by encoding the unicode using the charset specified by the container. I have no idea how complicated it will be do that, and it would be a new feature: parsing strings is specified to only work with ASCII input, currently.

I put "fix" in quotes, because even if you make text parts like this example work, you still can't handle non-text 8bit mime parts. Is it worth doing anyway?

Really, message_as_string and friends should just be avoided entirely, maybe even deprecated.

This came about because of an actual situation in a Mailman 3 installation. I can't say for sure what the actual original message looked like, but it was received by Mailman's LMTP server and parsed with email.message_from_bytes(), so it clearly wasn't exactly like the message excerpt I posted in the report above. However, All I had to go by was the message object from the shunted pickle file created as a result of the exception.

The message was processed by Mailman, but when Mailman's handler pipeline attempted to save it for the digest, it calls an instance of mailbox.MMDF to add the message to the mailbox accumulating messages for the digest, and that in turn calls the flatten method of an email.generator.BytesGenerator instance. and that's where the exception was thrown.

Perhaps the suggested patch in #18056 doesn't address every possible case, and it can result in a slightly garbled message due to replacing 'invalid' characters, but in my case at least, it is much preferable to the alternative.

Other Mailman3 installations are also encountering this issue. See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/VQZORIDL5PNQ4W33KIMVTFTANSGZD46S/

If we can get an actual reproducer using message_as_bytes I'd feel more comfortable with the fix. I worry that there is some other bug this is exposing that should be fixed instead.

message_from_bytes

I've researched this further, and I know how this happens. The original message contains a text/html part (in my case, the only part) which contains a base64 or quoted-printable body which when decoded contains non-ascii. It is parsed correctly by email.message_from_bytes.

It is then processed by Mailman's content filtering which retrieves html payload via

part.get_payload(decode=True).decode(ctype, errors='replace'))

where part is the text/html part and ctype is 'utf-8' in this case. It then uses elinks, lynx or some other configured command to convert the html payload to plain text and that plain text still contains non-ascii.

It then replaces the payload and sets the content type via

del part['content-transfer-encoding']
part.set_payload(plain_text)
part.set_type('text/plain')

And this results in a message which can't be flattened as_bytes.

The issue is set_payload() should encode the payload appropriately and in fact, it does if an appropriate charset is given, so this is our error in not providing a charset= argument to set_payload.

Closing this and the corresponding PR.