āžœ

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author andersk
Recipients andersk, barry, r.david.murray
Date 2021-02-25.18:18:07
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1614277088.24.0.58040288739.issue43323@roundup.psfhosted.org>
In-reply-to
Content
We ran into a UnicodeEncodeError exception using email.parser to parse this email <https://lists.cam.ac.uk/pipermail/cl-isabelle-users/2021-February/msg00135.html>, with full headers available in the raw archive <https://lists.cam.ac.uk/pipermail/cl-isabelle-users/2021-February.txt>.  The offending header is hilariously invalid:

Content-Type: text/plain; charset*=utf-8ā€''utf-8%E2%80%9D

but Iā€™m filing an issue since the parser is intended to be robust against invalid input.  Minimal reproduction:

>>> import email, email.policy
>>> email.message_from_bytes(b"Content-Type: text/plain; charset*=utf-8\xE2\x80\x9D''utf-8%E2%80%9D", policy=email.policy.default)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/email/__init__.py", line 46, in message_from_bytes
    return BytesParser(*args, **kws).parsebytes(s)
  File "/usr/local/lib/python3.10/email/parser.py", line 123, in parsebytes
    return self.parser.parsestr(text, headersonly)
  File "/usr/local/lib/python3.10/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/local/lib/python3.10/email/parser.py", line 57, in parse
    return feedparser.close()
  File "/usr/local/lib/python3.10/email/feedparser.py", line 187, in close
    self._call_parse()
  File "/usr/local/lib/python3.10/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/local/lib/python3.10/email/feedparser.py", line 256, in _parsegen
    if self._cur.get_content_type() == 'message/delivery-status':
  File "/usr/local/lib/python3.10/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/local/lib/python3.10/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/local/lib/python3.10/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/local/lib/python3.10/email/headerregistry.py", line 608, in __call__
    return self[name](name, value)
  File "/usr/local/lib/python3.10/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/local/lib/python3.10/email/headerregistry.py", line 453, in parse
    kwds['decoded'] = str(parse_tree)
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 126, in __str__
    return ''.join(str(x) for x in self)
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 126, in <genexpr>
    return ''.join(str(x) for x in self)
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 798, in __str__
    for name, value in self.params:
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 783, in params
    value = value.decode(charset, 'surrogateescape')
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 5-7: surrogates not allowed
History
Date User Action Args
2021-02-25 18:18:08andersksetrecipients: + andersk, barry, r.david.murray
2021-02-25 18:18:08andersksetmessageid: <1614277088.24.0.58040288739.issue43323@roundup.psfhosted.org>
2021-02-25 18:18:08andersklinkissue43323 messages
2021-02-25 18:18:07anderskcreate