classification
Title: EmailMessage as_bytes
Type: crash Stage:
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: marc.villain
Priority: normal Keywords:

Created on 2021-11-30 15:18 by marc.villain, last changed 2021-11-30 15:18 by marc.villain.

Messages (1)
msg407379 - (view) Author: Marc Villain (marc.villain) Date: 2021-11-30 15:18
I am parsing an email with a subject header where the encoding of a unicode character happens to be cut in half. When a second encoded unicode character is encountered, we get the following error:

> 'utf-8' codec can't encode characters in position 1-2: surrogates not allowed

This error can be reproduced using the following:
>>> from email.message import EmailMessage
>>> msg = EmailMessage()
>>> msg.add_header('subject', '=?UTF-8?Q?a=C3?= =?UTF-8?Q?=B1o_a=C3=B1o?=')
>>> print(str(msg))         # This will succeed
>>> print(msg.as_bytes())   # This will fail
>>> print(msg.as_string())  # This will fail

After a bit of investigations, it appears the library is at some poing trying to concatenate 'a\udcc3\udcb1o ' and 'cómo'. It then proceeds to try to call _ew.encode in email._header_value_parser._fold_as_ew on that. This obviously fails as '\udcc3\udcb1o' is not utf-8, whereas 'cómo' is.

More tests:
[OK] '=?UTF-8?Q?a=C3?= =?UTF-8?Q?=B1o_a=C3=B1o?='
     > b' subject: =?utf-8?q?a=C3=B1o_c=C3=B3mo?=\n\n'
[OK] '=?UTF-8?Q?a=C3?= =?UTF-8?Q?=B1o_cmo?='
     > b' subject: =?unknown-8bit?q?a=C3=B1o?= cmo\n\n'
[OK] '=?UTF-8?Q?a=C3?= =?UTF-8?Q?=B1o?= =?UTF-8?Q?a=C3?= =?UTF-8?Q?=B1o?='
     > b' subject: =?unknown-8bit?q?a=C3=B1oa=C3=B1o?=\n\n'
[KO] '=?UTF-8?Q?a=C3?= =?UTF-8?Q?=B1o_a=C3=B1o?='
     > 'utf-8' codec can't encode characters in position 1-2: surrogates not allowed

Not sure what is the best way to fix that.
History
Date User Action Args
2021-11-30 15:18:50marc.villaincreate