classification
Title: email.encoders.encode_quopri doesn't work with python 3.2
Type: behavior Stage: resolved
Components: email Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: barry, mitya57, python-dev, r.david.murray
Priority: normal Keywords:

Created on 2012-03-18 09:36 by mitya57, last changed 2013-08-22 01:14 by python-dev. This issue is now closed.

Messages (13)
msg156238 - (view) Author: Dmitry Shachnev (mitya57) * Date: 2012-03-18 09:36
Currently my /usr/lib/python3.2/email/encoders.py has this code:

def _qencode(s):
    enc = _encodestring(s, quotetabs=True)
    # Must encode spaces, which quopri.encodestring() doesn't do
    return enc.replace(' ', '=20')

The problem is that _encodestring (which is just quopri.encodestring) always returns bytes, trying to run replace() on bytes raises "TypeError: expected an object with the buffer interface".

This leads to email.encoders.encode_quopri never working.

So, I think this should be changed to something like this:

    <...>
    return enc.decode().replace(' ', '=20')

Example log:

Python 3.2.3rc1 (default, Mar  9 2012, 23:02:43) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import email.encoders
>>> from email.mime.text import MIMEText
>>> msg = MIMEText(b'some text here')
>>> email.encoders.encode_quopri(msg)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/email/encoders.py", line 44, in encode_quopri
    encdata = _qencode(orig)
  File "/usr/lib/python3.2/email/encoders.py", line 23, in _qencode
    return enc.replace(' ', '=20')
TypeError: expected an object with the buffer interface

Reproduced on Ubuntu precise with Python 3.2.3rc1. Replacing encode_quopri with encode_base64 works fine.
msg156298 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-19 00:41
Interesting.  Apparently we have no tests for the encode_ functions, nor do we use them inside the email package itself (except for encode_7or8bit).

Do you have any interest in writing a patch with tests?
msg156493 - (view) Author: Dmitry Shachnev (mitya57) * Date: 2012-03-21 15:04
(Sorry for not replying earlier).

I think the main priority here is getting things working, not the tests (so I have little interest in that).

First of all, should quopri.encodestring() really return bytes? Everything it returns is ascii text, obviously.

Then, which types of argument should encode_* functions take (I think str should be supported, and it's not a case here as encode_quopri will only accept bytes)?
msg156494 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-21 15:47
Well, a patch won't get committed if it lacks tests, so commit would have to wait until I have time to write some, then.

The encode_ methods (from email.encoders) take *message* objects as their arguments.  MIMEText internally converts a byte string into the appropriate CTE *if* you give it a charset (if you don't it later produces an error, but that's a different bug).  So if you pass bytes you don't need to call an encode_ method separately.

In fact, there's really no reason to call an encode_ method at all, since if you pass a string to MIMEText when giving it a non-ascii unicode string, it will default to utf-8 and do the appropriate CTE encoding.

But given that they exist in the documented API and exist in Python2, they need to be fixed to work in an equivalent fashion in Python3.  I think the only case where they would do anything useful is if you don't like Python's default for the CTE encoding and want to change it.  (Note that you can accomplish that globally by updating the charset alias in the charset module.)

What is your use case, by the way?
msg156498 - (view) Author: Dmitry Shachnev (mitya57) * Date: 2012-03-21 17:04
> In fact, there's really no reason to call an encode_ method at all, since if you pass a string to MIMEText when giving it a non-ascii unicode string, it will default to utf-8 and do the appropriate CTE encoding.

No, it doesn't:
Python 3.2.3rc1 (default, Mar  9 2012, 23:02:43) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from email.mime.text import MIMEText
>>> print(MIMEText('йцукен'))
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

йцукен
>>>

As you can see, it leaves russian text in unmodified state and sets the charset to "us-ascii". Should it be considered as a bug?

> What is your use case, by the way?
I'm writing a "send via e-mail" plugin for my ReText editor (http://retext.sourceforge.net/).
msg156499 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-21 17:26
Oh, you are right.  I even noted that bug in my PyCon talk, but immediately forgot about it :(  I do intend to fix it.

You can get it to work by explicitly passing the charset:

  >>> x = MIMEText('йцукен', _charset='utf8')
  >>> str(x)
'Content-Type: text/plain; charset="utf8"\nMIME-Version: 1.0\nContent-Transfer-Encoding: base64\n\n0LnRhtGD0LrQtdC9\n'

When I asked the use case, I meant specifically for calling encode_quopri, etc.  Given the above, do you need it anymore?
msg156542 - (view) Author: Dmitry Shachnev (mitya57) * Date: 2012-03-22 05:22
> You can get it to work by explicitly passing the charset
Thanks, I didn't know about that.

> Given the above, do you need it anymore?
No.
msg156544 - (view) Author: Dmitry Shachnev (mitya57) * Date: 2012-03-22 06:45
> Then, which types of argument should encode_* functions take (I think str should be supported, and it's not a case here as encode_quopri will only accept bytes)?

I meant which types of *payload* should they accept. Here's an illustration of what I mean: http://paste.ubuntu.com/894731/.
msg156549 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-22 12:22
email in python3 doesn't necessarily work with binary payloads.  (Obviously here you've found the opposite problem, but it is in methods that aren't used by the package itself.)  There aren't any tests of binary payloads in the test suite.  Ultimately this needs to be fixed one way or another, but I'm not sure when it is going to happen.
msg191964 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-27 22:39
New changeset 1d5856849e64 by R David Murray in branch '3.3':
#14360: make encoders.encode_quopri work.
http://hg.python.org/cpython/rev/1d5856849e64

New changeset 9046ef201591 by R David Murray in branch 'default':
Merge #14360: make encoders.encode_quopri work.
http://hg.python.org/cpython/rev/9046ef201591
msg191965 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-27 22:41
This should now be fixed.  Calling MIMEApplication with a binary payload and passing it encode_quopri as the encoder will now actually work.
msg192011 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-28 19:10
New changeset 7c807bc15fa8 by R David Murray in branch '3.3':
#14360: Add news item.
http://hg.python.org/cpython/rev/7c807bc15fa8

New changeset 36cc8b0446b3 by R David Murray in branch 'default':
Merge #14360: Add news item.
http://hg.python.org/cpython/rev/36cc8b0446b3
msg195848 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-22 01:14
New changeset 64e004737837 by R David Murray in branch '3.3':
#18324: set_payload now correctly handles binary input.
http://hg.python.org/cpython/rev/64e004737837
History
Date User Action Args
2013-08-22 01:14:27python-devsetmessages: + msg195848
2013-06-28 19:10:04python-devsetmessages: + msg192011
2013-06-27 22:41:15r.david.murraysetstatus: open -> closed
2013-06-27 22:41:08r.david.murraysetresolution: fixed
stage: needs patch -> resolved
messages: + msg191965
versions: + Python 3.4, - Python 3.2
2013-06-27 22:39:03python-devsetnosy: + python-dev
messages: + msg191964
2012-05-28 20:22:40r.david.murraysetcomponents: + email, - Library (Lib)
2012-03-22 12:22:39r.david.murraysetmessages: + msg156549
2012-03-22 06:45:50mitya57setmessages: + msg156544
2012-03-22 05:22:52mitya57setmessages: + msg156542
2012-03-21 17:26:46r.david.murraysetmessages: + msg156499
2012-03-21 17:04:59mitya57setmessages: + msg156498
2012-03-21 15:47:30r.david.murraysetmessages: + msg156494
2012-03-21 15:04:14mitya57setmessages: + msg156493
2012-03-19 00:41:46r.david.murraysetversions: + Python 3.3
nosy: + r.david.murray

messages: + msg156298

assignee: r.david.murray
stage: needs patch
2012-03-18 09:58:16mitya57settype: crash -> behavior
2012-03-18 09:36:34mitya57create