classification
Title: imaplib.IMAP4.authenticate authobject does not work correctly in python3
Type: behavior Stage: resolved
Components: email Versions: Python 3.4, Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: barry, demian.brecht, etukia, joebauer, nagisa, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2012-01-03 01:11 by etukia, last changed 2013-02-19 17:22 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
issue13700.patch etukia, 2012-01-03 20:42 review
cram_md5.patch etukia, 2012-01-03 21:47
imaplib_authenticate.patch r.david.murray, 2012-04-03 18:26
imaplib_authenticate_v2.patch etukia, 2012-07-01 16:40 review
Messages (18)
msg150489 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 01:11
>>> import imaplib
>>> imap = imaplib.IMAP4_SSL("imap.example.com")

>>> authcb = lambda resp: "{0}\x00{0}\x00{1}".format("username","password")
>>> imap.authenticate("PLAIN", authcb)
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    imap.authenticate("PLAIN", authcb)
  File "/usr/lib/python3.1/imaplib.py", line 361, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mech)
  File "/usr/lib/python3.1/imaplib.py", line 1075, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.1/imaplib.py", line 889, in _command
    literal = literator(self.continuation_response)
  File "/usr/lib/python3.1/imaplib.py", line 1238, in process
    return self.encode(ret)
  File "/usr/lib/python3.1/imaplib.py", line 1257, in encode
    e = binascii.b2a_base64(t)
TypeError: must be bytes or buffer, not str

... and ...

>>> authcb = lambda resp: "{0}\x00{0}\x00{1}".format("username","password").encode()
>>> imap.authenticate("PLAIN", authcb)
Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    imap.authenticate("PLAIN", authcb)
  File "/usr/lib/python3.1/imaplib.py", line 361, in authenticate
    typ, dat = self._simple_command('AUTHENTICATE', mech)
  File "/usr/lib/python3.1/imaplib.py", line 1075, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python3.1/imaplib.py", line 889, in _command
    literal = literator(self.continuation_response)
  File "/usr/lib/python3.1/imaplib.py", line 1238, in process
    return self.encode(ret)
  File "/usr/lib/python3.1/imaplib.py", line 1259, in encode
    oup = oup + e[:-1]
TypeError: Can't convert 'bytes' object to str implicitly
msg150491 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-03 01:59
The first argument to authenticate must be bytes.  This is not well documented.  It might also be a bug, since I'm not sure anyone has done a thorough audit of what should be bytes and what should be string in imaplib.

3.1 no longer gets bug fixes, so I'm removing it from versions.  Likewise I remove 3.4 since that applies only to changes that will *not* be put in 3.3 for some reason.
msg150492 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 02:11
The same problems exists even if I use b"PLAIN" as the first argument in authenticate().
msg150493 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-03 02:24
Gah, I was looking at the wrong source code when I wrote that.  A string first argument is indeed valid.  I'm not sure where the problem is coming from since the internal CRAM_MD5 returns a string and that seems to work...
msg150494 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 02:59
File "/usr/lib/python3.1/imaplib.py", line 1257, in encode
    e = binascii.b2a_base64(t)

imaplib._Authenticator.encode() calls binascii.b2a_base64() function. In Python 2.6 that function returns a string, and in Python 3.1 it returns bytes. What is returned from b2a_base64() function is later in the encode() function concatenated with a string, with bytes that is not possible.

Should binascii.b2a_base64() return a string (2.6) or bytes (3.1)?
msg150496 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-03 03:18
Bytes definitely.  We hashed that out a while ago.

My point is that CRAM_MD5 login calls authenticate, and its authenticator returns a string, just like your example does.  So it ought to be going through the same code path.  I haven't followed the logic in detail, though, so there must be some difference...I'm pretty sure the MD5 login has a test now (but not 100% sure...)
msg150500 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 15:06
In Python 2.6 PLAIN authentication works, in Python 3.1 not.

Lib/test/test_imaplib.py does not test IMAP4.authenticate() or IMAP4.login_cram_md5() functions, only IMAP4.login().

I would still like to go back to imaplib._Authenticator.encode() function. The function is below.

# inp = authobject(response)
def encode(self, inp):
    oup = ''
    while inp:
        if len(inp) > 48:
            t = inp[:48]
            inp = inp[48:]
        else:
            t = inp
            inp = ''
        e = binascii.b2a_base64(t)
        if e:
            oup = oup + e[:-1]
    return oup

binascii.b2a_base64() takes bytes, so inp must therefore be bytes, and returns bytes (Python 3). Then str + bytes (out + e[:-1]) fails.

The fix would then be changing 
            oup = oup + e[:-1]
to
            oup = oup + e[:-1].decode()
msg150505 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 15:41
I tried to fix the problem and the correct fix is to change
    oup = ''
to
    oup = b''

in imaplib._Authenticator.encode() function, and not what I suggested in my previous post.

After changing that PLAIN authentication works.
msg150510 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-03 16:20
Would you be interested in providing a patch that includes tests?  I think Antoine set up a test framework for testing the login as part of issue 4471.
msg150530 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 20:42
Here's a patch with test.

I am not an IMAP guru, so please verify my patch.
msg150536 - (view) Author: Erno Tukia (etukia) Date: 2012-01-03 21:47
Here's another patch that should fix the CRAM-MD5 authentication. My previous patch is required with this one. The patch includes a test.
msg152608 - (view) Author: Johannes Bauer (joebauer) Date: 2012-02-04 11:23
Issue also affects Python3.1. Hunk succeeds against 3.1 imaplib.py and works for me.
msg157436 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-04-03 18:26
I made some time to work on this today.  Attached is a new patch.  I've incorporated the tests from the existing patches (though I'm doing the infrastructure a bit differently).  PLAIN seems to be a specific case of the general authenticate, so I just have a generalized test.

My fix is slightly different.  I'm changing the _Authenticate class to except either bytes or string as input.  String because that's more natural in Python3, bytes because there might be auth mechanisms that require bytes (since implementations can define 'X' auth mechanisms).

The authentication callback always gets passed the data as bytes, for the same reasons of generality.  So I'd be OK with the idea that the authentication handler always has to return bytes...which would require a different fix in login_cram_md5.

My login_cram_md5 test passes, so I *think* the fix I made there is OK.  However, I'm getting a traceback from SSL about the socket being shut down, which seems to arise from an imaplib.abort resulting from an unexpected 'b''' value.  I'm out of time for working in this right now, so I'm uploading the patch to see if anyone else has time to figure it out.  I'll come back to it as some point but I don't know when.
msg158262 - (view) Author: Simonas Kazlauskas (nagisa) Date: 2012-04-14 12:16
Exactly same thing happens with `XOAUTH` mechanism too, so this bug report should be made more general. (Py3.2.2)
msg164486 - (view) Author: Erno Tukia (etukia) Date: 2012-07-01 16:40
Here's the updated patch. Tests now works. PLAIN works for me, that's only I can test against live system.

test_login_cram_md5 test had extra \r\n in _send_tagged.

diff imaplib_authenticate.patch imaplib_authenticate_v2.patch
< + self._send_tagged(tag, 'OK', 'CRAM-MD5 successful\r\n')
---
> + self._send_tagged(tag, 'OK', 'CRAM-MD5 successful')
msg182391 - (view) Author: Demian Brecht (demian.brecht) * Date: 2013-02-19 15:24
Has there been any further work/review done on this issue? I just ran into the problem myself on 3.4(dev) and can verify that the patch (imaplib_authenticate_v.patch) fixes the issue.
msg182398 - (view) Author: Roundup Robot (python-dev) Date: 2013-02-19 17:21
New changeset 3d4302718e7c by R David Murray in branch '3.2':
#13700: Make imap.authenticate with authobject work.
http://hg.python.org/cpython/rev/3d4302718e7c

New changeset b21f955b8ba2 by R David Murray in branch '3.3':
Merge: #13700: Make imap.authenticate with authobject work.
http://hg.python.org/cpython/rev/b21f955b8ba2

New changeset d404d33a999c by R David Murray in branch 'default':
Merge: #13700: Make imap.authenticate with authobject work.
http://hg.python.org/cpython/rev/d404d33a999c
msg182399 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-19 17:22
Demian: thanks for the reminder, and the confirmation that it works on a real server.

Erno: thanks for the test fix.  That was a pretty stupid mistake on my part :)
History
Date User Action Args
2013-02-19 17:22:45r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg182399

stage: patch review -> resolved
2013-02-19 17:21:07python-devsetnosy: + python-dev
messages: + msg182398
2013-02-19 15:24:09demian.brechtsetnosy: + demian.brecht

messages: + msg182391
versions: + Python 3.4
2012-07-01 16:40:08etukiasetfiles: + imaplib_authenticate_v2.patch

messages: + msg164486
2012-05-28 20:20:57r.david.murraysetnosy: + barry
components: + email, - Library (Lib)
2012-04-14 12:24:30r.david.murraysettitle: imaplib.IMAP4.authenticate authobject fails with PLAIN mechanism -> imaplib.IMAP4.authenticate authobject does not work correctly in python3
2012-04-14 12:16:31nagisasetnosy: + nagisa
messages: + msg158262
2012-04-03 18:26:56r.david.murraysetfiles: + imaplib_authenticate.patch

assignee: docs@python -> r.david.murray
components: - Documentation
versions: - Python 3.1
nosy: - docs@python

messages: + msg157436
stage: patch review
2012-02-04 11:23:28joebauersetnosy: + joebauer

messages: + msg152608
versions: + Python 3.1
2012-01-03 21:47:15etukiasetfiles: + cram_md5.patch

messages: + msg150536
2012-01-03 20:42:39etukiasetfiles: + issue13700.patch
keywords: + patch
messages: + msg150530
2012-01-03 16:20:57r.david.murraysetmessages: + msg150510
2012-01-03 15:41:32etukiasetmessages: + msg150505
2012-01-03 15:06:15etukiasetmessages: + msg150500
2012-01-03 03:18:01r.david.murraysetmessages: + msg150496
2012-01-03 02:59:52etukiasetmessages: + msg150494
2012-01-03 02:24:23r.david.murraysetmessages: + msg150493
2012-01-03 02:11:10etukiasetmessages: + msg150492
2012-01-03 01:59:58r.david.murraysetversions: - Python 3.1, Python 3.4
nosy: + r.david.murray, docs@python

messages: + msg150491

assignee: docs@python
components: + Documentation
2012-01-03 01:11:28etukiacreate