This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author altvod
Recipients altvod, barry, r.david.murray
Date 2018-07-25.12:56:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1532523404.04.0.56676864532.issue34222@psf.upfronthosting.co.za>
In-reply-to
Content
(Discovered together with https://bugs.python.org/msg322348)

Email message serialization (in function _fold_as_ew) enters an infinite loop when folding non-ASCII headers whose words (after encoding) are longer than the given maxlen.

Besides being stuck in an infinite loop, it keeps appending to the `lines` list, so its memory usage keeps on growing also infinitely.
The code keeps appending encoded empty strings to the list like this:

lines: [
    'Subject: =?utf-8?q??=',
    ' =?utf-8?q??=',
    ' =?utf-8?q??=',
    ' =?utf-8?q??=',
    ' =?utf-8?q??=',
    ' =?utf-8?q??=',
    ' '
]
(and it keeps on growing)

Here is my code that can reproduce this issue (as a unittest):


import email.generator
import email.policy
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from unittest import TestCase


def create_message(subject, sender, recipients, body):
    msg = MIMEMultipart()
    msg.set_charset('utf-8')
    msg.policy = email.policy.SMTP
    msg.attach(MIMEText(body, 'html'))
    msg['Subject'] = subject
    msg['From'] = sender
    msg['To'] = ';'.join(recipients)
    return msg


class TestEmailMessage(TestCase):
    def _make_message(self, subject):
        return create_message(
            subject=subject, sender='me@site.com',
            recipients=['me@site.com'], body='Some text',
        )

    def test_ascii_message_with_len_limit(self):
        # very long subject consisting of a single word
        subject = 'Q' * 100
        msg = self._make_message(subject)
        self.assertTrue(msg.as_string(maxheaderlen=76))

    def test_non_ascii_message_with_len_limit(self):
        # very long subject consisting of a single word
        subject = 'Ц' * 100
        msg = self._make_message(subject)
        self.assertTrue(msg.as_string(maxheaderlen=76))


The ASCII test passes, but the non-ASCII one never finishes.

From what I can tell, the problem is in line 2728 of email/_header_value_parser.py:

            first_part = first_part[:-excess]

where `excess` is calculated from the encoded string
(which is several times longer than the original one),
but it truncates the original (non-encoded string).
The problem arises when `excess` is actually greater than `first_part`
So, it attempts to encode the exact same part of the header and fails in every iteration,
instead appending an empty string to the list and encoding it as ' =?utf-8?q??='

What this amounts to is that it's now practically impossible to send emails with non-ACSII subjects without either disregarding the RFC recommendations and requirements for line length or risking hangs and memory leaks.

Just like in https://bugs.python.org/msg322348, this behavior is new in Python 3.6. Also does not work in 3.7 and 3.8
History
Date User Action Args
2018-07-25 12:56:44altvodsetrecipients: + altvod, barry, r.david.murray
2018-07-25 12:56:44altvodsetmessageid: <1532523404.04.0.56676864532.issue34222@psf.upfronthosting.co.za>
2018-07-25 12:56:44altvodlinkissue34222 messages
2018-07-25 12:56:43altvodcreate