This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: smtplib.py AUTH LOGIN code messed up sending login and password data since 3.5
Type: behavior Stage: resolved
Components: email Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Rosuav, barry, larry, merkel, miohtama, ned.deily, python-dev, r.david.murray
Priority: release blocker Keywords: 3.5regression, patch

Created on 2015-10-20 15:41 by merkel, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
smtplib.py merkel, 2015-10-20 21:16 Fixed version of smtplib.py
smtplib-patch.issue25446.patch merkel, 2015-10-20 22:06 patch including change proposal.
Messages (15)
msg253225 - (view) Author: Oliver Merkel (merkel) * Date: 2015-10-20 15:41
class SMTP:
    def auth_login(self, challenge=None):

The self.docmd should use cmd "AUTH" with parameter "LOGIN" + encoded login like

        (code, resp) = self.docmd("AUTH", "LOGIN " +
            encode_base64(self.user.encode('ascii'), eol=''))

with

    def auth(self, mechanism, authobject, *, initial_response_ok=True):

that should not send a "AUTH" in self.docmd in case the mechanism is 'LOGIN' and

        if initial_response is not None:

meaning

            if mechanism == 'LOGIN':
                (code, resp) = self.docmd(response)
            else:
                (code, resp) = self.docmd("AUTH", mechanism + " " + response)

---

Could someone kindly review, evtly come up with better suggestion?

In short:
$ diff /c/Python35/Lib/smtplib-old.py /c/Python35/Lib/smtplib.py
630c630,633
<             (code, resp) = self.docmd("AUTH", mechanism + " " + response)
---
>             if mechanism == 'LOGIN':
>                 (code, resp) = self.docmd(response)
>             else:
>                 (code, resp) = self.docmd("AUTH", mechanism + " " + response)
660c663
<         (code, resp) = self.docmd(
---
>         (code, resp) = self.docmd("AUTH", "LOGIN " +
msg253241 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-20 19:13
Does the fix at the end of issue 15014 address your concern?
msg253242 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-20 19:21
Sorry, I misremenbered and thought that fix didn't make it in to 3.5, but in fact you are talking about the behavior of that fix.
msg253243 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-20 19:23
Can you explain more about the failure you are seeing?
msg253256 - (view) Author: Oliver Merkel (merkel) * Date: 2015-10-20 21:16
Let us assume you want to establish a smtp session with AUTH LOGIN as the supported authentication type. Sample code to send mail: Typical preparation steps like

    with SMTP( mailserver, 587 ) as smtp:
      # smtp.set_debuglevel(1)
      smtp.ehlo()
      smtp.starttls()
      smtp.ehlo()
      smtp.login( account, password )

If you try to login (last line in sample code above) at the smtp server then the smtp server will expect a command to be send like

AUTH LOGIN <base64encodedaccountdata>

Now switching from sample code to our smtplib.py:

Since the "AUTH LOGIN" is missing in original Python35/Lib/smtplib.py in line...

660c663
<         (code, resp) = self.docmd(
---
>         (code, resp) = self.docmd("AUTH", "LOGIN " +

... the smtp server will answer that the library is sending an unknown command here. That is why I added... "AUTH", "LOGIN " + ...at this line.

Line 660 in class SMTP: def auth_login is called before it reaches line 630 in class SMTP: def auth

In case of authentication type AUTH LOGIN in line 630 you must not call with "AUTH", mechanism + " " +

So the following changes have to be applied at least for AUTH LOGIN mechanism

630c630,633
<             (code, resp) = self.docmd("AUTH", mechanism + " " + response)
---
>             if mechanism == 'LOGIN':
>                 (code, resp) = self.docmd(response)
>             else:
>                 (code, resp) = self.docmd("AUTH", mechanism + " " + response)

The first change affecting line 660 described above will will imply that the you remove the AUTH mechanism in line 630. For mechanism LOGIN the base64 encoded password will be needed to be sent in 660...

See possible fix in the diff above.

To ease understanding the fix I will apply a running version of my local Lib/smtplib.py (instead of just providing the diff lines). Feel free to directly use the file.
msg253259 - (view) Author: Oliver Merkel (merkel) * Date: 2015-10-20 21:46
Sample session log output showing the error with smtp.set_debuglevel(1):

send: 'ehlo <mysmtpserver>\r\n'
reply: b'250-<mysmtpserver> Hello [myIP4address]\r\n'
reply: b'250-SIZE 53248000\r\n'
reply: b'250-PIPELINING\r\n'
reply: b'250-DSN\r\n'
reply: b'250-ENHANCEDSTATUSCODES\r\n'
reply: b'250-STARTTLS\r\n'
reply: b'250-AUTH GSSAPI NTLM\r\n'
reply: b'250-8BITMIME\r\n'
reply: b'250-BINARYMIME\r\n'
reply: b'250 CHUNKING\r\n'
reply: retcode (250); Msg: b'<mysmtpserver> Hello [myIP4address]\
nSIZE 53248000\nPIPELINING\nDSN\nENHANCEDSTATUSCODES\nSTARTTLS\nAUTH GSSAPI NTLM
\n8BITMIME\nBINARYMIME\nCHUNKING'
send: 'STARTTLS\r\n'
reply: b'220 2.0.0 SMTP server ready\r\n'
reply: retcode (220); Msg: b'2.0.0 SMTP server ready'
send: 'ehlo [mymachinename]\r\n'
reply: b'250-<mysmtpserver> Hello [myIP4address]\r\n'
reply: b'250-SIZE 53248000\r\n'
reply: b'250-PIPELINING\r\n'
reply: b'250-DSN\r\n'
reply: b'250-ENHANCEDSTATUSCODES\r\n'
reply: b'250-AUTH GSSAPI NTLM LOGIN\r\n'
reply: b'250-8BITMIME\r\n'
reply: b'250-BINARYMIME\r\n'
reply: b'250 CHUNKING\r\n'
reply: retcode (250); Msg: b'<mysmtpserver> Hello [myIP4address]\
nSIZE 53248000\nPIPELINING\nDSN\nENHANCEDSTATUSCODES\nAUTH GSSAPI NTLM LOGIN\n8B
ITMIME\nBINARYMIME\nCHUNKING'
send: '<base64encodedaccountnamedata>==\r\n'
reply: b'500 5.3.3 Unrecognized command\r\n'
reply: retcode (500); Msg: b'5.3.3 Unrecognized command'
send: 'QUIT\r\n'
reply: b'221 2.0.0 Service closing transmission channel\r\n'
reply: retcode (221); Msg: b'2.0.0 Service closing transmission channel'
Traceback (most recent call last):
  File "sendtestmail.py", line 172, in <module>
    announcement.sendMail(password)
  File "sendtestmail.py", line 97, in sendMail
    smtp.login( self.getShortAddressList()[0], password )
  File "c:\Python35\lib\smtplib.py", line 730, in login
    raise last_exception
  File "c:\Python35\lib\smtplib.py", line 721, in login
    initial_response_ok=initial_response_ok)
  File "c:\Python35\lib\smtplib.py", line 627, in auth
    initial_response = (authobject() if initial_response_ok else None)
  File "c:\Python35\lib\smtplib.py", line 664, in auth_login
    raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (500, b'5.3.3 Unrecognized command')

due to missing AUTH LOGIN here as previously described...
msg253261 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-20 21:54
Ah, now I see what you are saying.  How in the world did we miss that?  Our unit tests must be broken too.
msg253262 - (view) Author: Oliver Merkel (merkel) * Date: 2015-10-20 22:06
Change proposal attached as a unified diff / patch file.
msg253287 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-21 13:24
Thanks, but special-casing login in the 'auth' method means that the auth method isn't working right, since special-casing defeats the whole purpose of the auth mechanism.

I think we need to change the logic in auth so that it is checking for a 334 even if it has been provided an initial response.  That is, outdent the block that starts with the '# Server replies...' comment.  Once that is done, auth_login becomes:

  def auth_login(self, challenge=None)
      if challenge is None:
          return encode_base64(self.user.encode('ascii'))
      else:
          return self.password

We may also need to add a try/except around the base64.decodebytes in auth.

And we need a unit test that demonstrates the current failure.

I'm also wondering now about the ascii encoding on the challenge and response.  Someone should check the RFC to see if those are limited to ascii or if they can contain other bytes.  If they are limited to ascii we should stick in a comment to that effect with a pointer to the relevant RFC section.
msg253867 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-11-01 18:50
I think it's about time to think about releasing 3.5.1.  But since this bug is marked as a "release blocker", 3.5.1 cannot be released until this is fixed.  Arguably I can't even really make a schedule for 3.5.1 until it's fixed, or at least I'm reasonably confident I know when it'll be fixed.

Any idea when this might be fixed?
msg253876 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-11-01 21:08
I will work on it this week, should have something committed before the end of next weekend.
msg254032 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-11-04 00:16
Okay, I'm scheduling 3.5.1rc1 on the assumption that you'll check in by next weekend.  If you're going to slip please let me know and I'll slip accordingly.
msg254327 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-08 06:06
New changeset d13263ecf0c6 by R David Murray in branch '3.5':
#25446: Fix regression in smtplib's AUTH LOGIN support.
https://hg.python.org/cpython/rev/d13263ecf0c6

New changeset 7368b86432c6 by R David Murray in branch 'default':
Merge: #25446: Fix regression in smtplib's AUTH LOGIN support.
https://hg.python.org/cpython/rev/7368b86432c6
msg254329 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-11-08 06:19
Fixed.  I figured it was better not to wait for a review in this case...the fix is straightforward, but writing the test took quite a bit of work.  The auth tests in smtplib are now much more robust, thanks in large part to Milan's code in issue 21935.
msg273966 - (view) Author: Miko Ohtamaa (miohtama) * Date: 2016-08-30 21:05
I think there is something more in this. 

I am running Python 3.5.0 (default, Apr 24 2016, 12:47:36).

Sparkpost servers require AUTH LOGIN approach as per their instructions https://support.sparkpost.com/customer/portal/articles/1988470-smtp-connection-problems

When trying to use these (free) servers smtplib authentication will result to smtplib.SMTPAuthenticationError: (500, b'5.5.2 unrecognized command') like in the issue description earlier. I am still investigating this.
History
Date User Action Args
2022-04-11 14:58:22adminsetgithub: 69632
2016-08-30 21:05:21miohtamasetnosy: + ned.deily, miohtama
messages: + msg273966
2015-11-08 06:19:42r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg254329

stage: needs patch -> resolved
2015-11-08 06:06:05python-devsetnosy: + python-dev
messages: + msg254327
2015-11-04 00:16:21larrysetmessages: + msg254032
2015-11-01 21:49:24Rosuavsetnosy: + Rosuav
2015-11-01 21:08:09r.david.murraysetmessages: + msg253876
2015-11-01 18:50:38larrysetnosy: + larry
messages: + msg253867
2015-10-21 13:24:39r.david.murraysetmessages: + msg253287
2015-10-20 22:06:51merkelsetfiles: + smtplib-patch.issue25446.patch
keywords: + patch
messages: + msg253262
2015-10-20 21:54:26r.david.murraysetpriority: normal -> release blocker
versions: + Python 3.6
messages: + msg253261

keywords: + 3.5regression
stage: needs patch
2015-10-20 21:46:14merkelsetmessages: + msg253259
2015-10-20 21:16:44merkelsetfiles: + smtplib.py

messages: + msg253256
2015-10-20 19:23:03r.david.murraysetmessages: + msg253243
2015-10-20 19:21:04r.david.murraysetmessages: + msg253242
2015-10-20 19:13:16r.david.murraysetmessages: + msg253241
2015-10-20 15:41:32merkelcreate