(Follow-up to bpo-14380)

The attached patch makes the email.mime.text.MIMEText constructor use the iso-8859-1 (aka latin-1) encoding for messages where all characters are in range(256). This also makes them use quoted-printable transfer encoding instead of base64.

So, the current algorithm of guessing encoding is as follows:

• all characters are in range(128) -> encoding is us-ascii
• all characters are in range(256) -> encoding is iso-8859-1 (aka latin-1)
• else -> encoding is utf-8

Updated the patch:

• Avoid using letter Ш in test, it's better to use chr(256) as the test case;
• Updated the comment in MIMEText constructor to reflect the new behaviour.

Thanks for the patch. I may not get to this until after the beta (or I might, you never know).

Could you submit a contributor agreement please? http://www.python.org/psf/contrib

Done, sent an e-mail to contributors@python.org.

Patch is interesting, using an encoder to detect validity. However, it suffers from some performance problems for long text that has large ASCII prefixes.

This seems to be an enhancement sort of request rather than a bug... so I wonder why Python 3.2 is listed?

And in Python 3.3 with PEP-393 strings the C API to strings provides a quick way to determine the maximum character in the string... although I see nothing in the PEP about how to access that information from Python. If it is available, it could provide a much quicker precheck rather than multiple attempts to encode strings with large ASCII prefixes only to discover that the next to last character is in (128,255) and the last character is > 255 (which would be about the worst case scenario for the algorithm in the patch).

This seems to be an enhancement sort of request rather than a bug... so I wonder why Python 3.2 is listed?

Fixed.

... although I see nothing in the PEP about how to access that information from Python.

You are right, it seems there is no Python API for that (yet?), so I don't see any better solutions for determining the maximum character for now. Also, note that this algorithm had already been used before my patch.

Well, the original change to using utf-8 by default was considered a bug fix. But I suppose you are right that this goes beyond that into enhancement territory. In which case we could wait for an enhancement to the C API to base it on, for which we'd need to open a new issue.

On the other hand, the email package already uses the "encode to see if we have ascii" trick elsewhere (though on smaller strings), and the ascii codec is the fastest codec, with latin-1 only slightly slower.

The critical difference here, though, is that we end up doing two encoding passes, once to test it and a second time to actually create the message body. The same is true of the ascii case. It should be possible to fix this, by using the encoded string in generating the _payload, short circuiting the set_payload mechanism. That's a somewhat ugly hack, necessitated because of the incomplete conversion of email to a unicode-centric design. I'm working on that :)

So, again, we may be waiting on other enhancements, in this case in the email package, to do this fix "right". But it would be worth figuring out *how* to do it, so that we know what kind of (internal?) API enhancements we want in order to serve this kind of use case.

The new email API (which was just made non-provisional) uses a "sniff" technique to decide what CTE to use for text bodies set via set_content. So I consider this done (finally). It does not change MIMEText, which is now the legacy API.