New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add utf8 alias for email charsets #48737
Comments
When using MIME email package you can specify "utf8" as the encoding. It It is confusing since Python itself happily understands both utf8 and utf-8. The patch adds "utf8" as an alias to "utf-8" encoding which means user Test case: msg = MIMEText(u'\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430')
msg.set_charset('utf8')
print msg.as_string() |
This seems entirely reasonable, helpful, and in accord with the mapping There are pobably other encoding names with the same issue of being This issue also affects 2.6.1 and 2.7trunk. I haven't checked 3.x. |
Has this patch been merged yet? |
I tested it on python 2.5, 2.6, 2.7 trunk and 3.2 varying msg.set_charset(x) with x = 'utf8' and 'utf-8' 07:35:40 l0nwlf-MBP:~ $ python2.5
Python 2.5.4 (r254:67916, Jul 7 2009, 23:51:24)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from email.MIMEText import MIMEText
>>> msg = MIMEText(u'\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430')
>>> msg.set_charset('utf8')
>>> print msg.as_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/message.py", line 131, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 84, in flatten
self._write(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 109, in _write
self._dispatch(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 135, in _dispatch
meth(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 178, in _handle_text
self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)
>>> msg.set_charset('utf-8')
>>> print msg.as_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/message.py", line 131, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 84, in flatten
self._write(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 109, in _write
self._dispatch(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 135, in _dispatch
meth(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/email/generator.py", line 178, in _handle_text
self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)
07:36:17 l0nwlf-MBP:~ $ python2.6
Python 2.6.5 (r265:79063, Apr 6 2010, 21:34:21)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from email.MIMEText import MIMEText
>>> msg = MIMEText(u'\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430')
>>> msg.set_charset('utf8')
>>> print msg.as_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/message.py", line 135, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 84, in flatten
self._write(msg)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 109, in _write
self._dispatch(msg)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 135, in _dispatch
meth(msg)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 178, in _handle_text
self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)
>>> msg.set_charset('utf-8')
>>> print msg.as_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/message.py", line 135, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 84, in flatten
self._write(msg)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 109, in _write
self._dispatch(msg)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 135, in _dispatch
meth(msg)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/email/generator.py", line 178, in _handle_text
self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)
07:36:37 l0nwlf-MBP:~ $ python2.7
Python 2.7a4+ (trunk:78750, Mar 7 2010, 08:09:00)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from email.MIMEText import MIMEText
>>> msg = MIMEText(u'\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430')
>>> msg.set_charset('utf8')
>>> print msg.as_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/email/message.py", line 135, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/usr/local/lib/python2.7/email/generator.py", line 83, in flatten
self._write(msg)
File "/usr/local/lib/python2.7/email/generator.py", line 108, in _write
self._dispatch(msg)
File "/usr/local/lib/python2.7/email/generator.py", line 134, in _dispatch
meth(msg)
File "/usr/local/lib/python2.7/email/generator.py", line 180, in _handle_text
self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)
>>> msg.set_charset('utf-8')
>>> print msg.as_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/email/message.py", line 135, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/usr/local/lib/python2.7/email/generator.py", line 83, in flatten
self._write(msg)
File "/usr/local/lib/python2.7/email/generator.py", line 108, in _write
self._dispatch(msg)
File "/usr/local/lib/python2.7/email/generator.py", line 134, in _dispatch
meth(msg)
File "/usr/local/lib/python2.7/email/generator.py", line 180, in _handle_text
self._fp.write(payload)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-7: ordinal not in range(128)
07:37:06 l0nwlf-MBP:~ $ python3.2
Python 3.2a0 (py3k:79532, Apr 1 2010, 01:48:52)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from email.mime.text import MIMEText
>>> msg = MIMEText('\u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430')
>>> msg.set_charset('utf8')
>>> print (msg.as_string())
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf8"
кирилица
>>> msg.set_charset('utf-8')
>>> print (msg.as_string())
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8" кирилица The aliasing of 'utf-8' into 'utf8' and may be 'utf_8' seems reasonable IMO however Test Case fails. |
MIMEText doesn't support unicode input. This was the reason OP Test case failed. For reference : http://bugs.python.org/issue1368247 |
For various reasons the email module has a table of character sets. What might be most effective would be for the email module to look a character set name up in the codecs module and find out the cannonical name of the character set, and then look that up in its table (ie: remove the aliases table from email completely, and instead depend on codecs to resolve the cannonical name). Unfortunately the codecs module does not recognize all of the aliases used by email, nor is there necessarily any guarantee that the two modules will agree on the proper cannonical name. The attached patch instead uses the codecs module as a fallback if the charset name does not appear in the email package's ALIASES or CHARSETS tables. It therefore makes both utf8 and utf_8 work, as well as all the other variants the codec module accepts. The unit test just tests 'utf8', since if that one works all the others should too. I'm tentatively reclassifying this as a bug rather than a feature request, since I think it is a reasonable expectation that email would support at least the same set of encoding names that the rest of Python does. |
Idea: Import the aliases mapping from codecs and extend it with email-specific aliases. Alternate idea: Add email’s names to codecs. Side note: “charset” stands for “character encoding”, not “character set”. See <http://www.w3.org/International/questions/qa-what-is-encoding#what\> |
R. David Murray wrote:
I think that the encodings package should be the only source of If there are aliases missing, then we can add them there. If the email packages needs different canonical names, it can apply |
Mark, any objection to my putting this patch in now, and then we'll fix the aliases implementation in 3.2? |
R. David Murray wrote:
No. Please open a new issue targeting Python 3.2 for this. Thanks,Marc-Andre Lemburg 2010-07-19: EuroPython 2010, Birmingham, UK 44 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 |
Patch committed to trunk in r81705. Leaving issue open pending porting to the other branches, but I've also opened bpo-8898 to further change things so that codecs becomes the sole authority for aliases in 3.2. |
David, can this issue be closed? |
Yes. Benjamin merged this to py3k in r82292. If someone wants to explain to me how to cherry pick the changeset into 3.1 I'd be happy to do it, otherwise I think I'm done with this one :) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: