classification
Title: Possible bug in smtplib when initial_response_ok=False
Type: behavior Stage:
Components: email, Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Dario D'Amico, Mario Colombo, barry, r.david.murray, redstone-cold
Priority: normal Keywords:

Created on 2016-08-21 00:29 by Dario D'Amico, last changed 2018-09-13 05:54 by redstone-cold.

Messages (4)
msg273255 - (view) Author: Dario D'Amico (Dario D'Amico) Date: 2016-08-21 00:34
oo
msg273256 - (view) Author: Dario D'Amico (Dario D'Amico) Date: 2016-08-21 00:35
I have reasons to believe that smtlib.py does not support AUTH LOGIN well.

My guts feeling are that the auth_login method should be changed into:

    def auth_login(self, challenge=None):
        print("auth_login", challenge)
        """ Authobject to use with LOGIN authentication. Requires self.user and
        self.password to be set."""
        if challenge is None:
            return self.user
        elif challenge == b'Username:':
            return self.user
        elif challenge == b'Password:':
            return self.password

While the if at line 634, in the auth method, should actually be a while,
so that this:

        # If server responds with a challenge, send the response.
        if code == 334:
            challenge = base64.decodebytes(resp)
            response = encode_base64(
                authobject(challenge).encode('ascii'), eol='')
            (code, resp) = self.docmd(response)

is turned into this:

        # If server responds with a challenge, send the response.
        # Note that there may be multiple, sequential challenges.
        while code == 334:
            challenge = base64.decodebytes(resp)
            response = encode_base64(
                authobject(challenge).encode('ascii'), eol='')
            (code, resp) = self.docmd(response)

First, some background on AUTH LOGIN; based on my understanding of
http://www.fehcom.de/qmail/smtpauth.html there are two possible ways
to authenticate a client using AUTH LOGIN:

Method A
    C: AUTH LOGIN
    S: 334 VXNlcm5hbWU6
    C: <ENCODED_USERNAME>
    S: 334 UGFzc3dvcmQ6
    C: <ENCODED_PASSWORD>

Method B
    C: AUTH LOGIN <ENCODED_USERNAME>
    S: 334 UGFzc3dvcmQ6
    C: <ENCODED_PASSWORD>

The second method saves two round trips because the client sends
the username together with the AUTH LOGIN command. Note that the
strings VXNlcm5hbWU6 and UGFzc3dvcmQ6 are fixed and they are,
respectively, the Base64 encodings of 'Username:' and 'Password:'.

In the following I will detail my experience with smtplib.

Everything begun from this code fragment:

    smtpObj = smtplib.SMTP("smtp.example.com", "25")
    smtpObj.set_debuglevel(2)
    smtpObj.login("noreply@example.com", "chocolaterain")
    smtpObj.sendmail(sender, receivers, message)

The debug log produced by smtplib looked like this:

01:53:32.420185 send: 'ehlo localhost.localdomain\r\n'
01:53:32.624123 reply: b'250-smtp.example.com\r\n'
01:53:32.862965 reply: b'250-AUTH LOGIN\r\n'
01:53:32.863490 reply: b'250 8BITMIME\r\n'
01:53:32.863844 reply: retcode (250); Msg: b'smtp.example.com\nAUTH LOGIN\n8BITMIME'
01:53:32.868414 send: 'AUTH LOGIN <<<ENCODED_USERNAME>>>\r\n'
01:53:33.069884 reply: b'501 syntax error\r\n'
01:53:33.070479 reply: retcode (501); Msg: b'syntax error'
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dario/Programming/DigitalOcean/s.py", line 48, in <module>
    smtpObj.login("noreply@example.com", "chocolaterain")
  File "/usr/lib/python3.5/smtplib.py", line 729, in login
    raise last_exception
  File "/usr/lib/python3.5/smtplib.py", line 720, in login
    initial_response_ok=initial_response_ok)
  File "/usr/lib/python3.5/smtplib.py", line 641, in auth
    raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (501, b'syntax error')

This is most likely not an issue with smtplib, but simply an
indication that smtp.example.com does not support receiving the
username together with AUTH LOGIN (method B), and it replies with 501 syntax error.

I figured out that I could force the alternate data flow (method A), in which
the username is issued in a separate command, by setting
initial_response_ok=False when logging in:

    smtpObj = smtplib.SMTP("smtp.example.com", "25")
    smtpObj.set_debuglevel(2)
    smtpObj.login("noreply@example.com", "chocolaterain", initial_response_ok=False)
    smtpObj.sendmail(sender, receivers, message)

This resulted in a slightly more interesting behaviour:
    
01:53:54.445118 send: 'ehlo localhost.localdomain\r\n'
01:53:54.648136 reply: b'250-smtp.example.com\r\n'
01:53:54.884669 reply: b'250-AUTH LOGIN\r\n'
01:53:54.885197 reply: b'250 8BITMIME\r\n'
01:53:54.885555 reply: retcode (250); Msg: b'smtp.example.com\nAUTH LOGIN\n8BITMIME'
01:53:54.890051 send: 'AUTH LOGIN\r\n'
01:53:55.089540 reply: b'334 VXNlcm5hbWU6\r\n'
01:53:55.090119 reply: retcode (334); Msg: b'VXNlcm5hbWU6'
01:53:55.090955 send: '<<<ENCODED_PASSWORD>>>=\r\n'
01:53:55.296243 reply: b'334 UGFzc3dvcmQ6\r\n'
01:53:55.296717 reply: retcode (334); Msg: b'UGFzc3dvcmQ6'
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dario/Programming/DigitalOcean/s.py", line 57, in <module>
    smtpObj.login("noreply@example.com", "16226464", initial_response_ok=False)
  File "/usr/lib/python3.5/smtplib.py", line 729, in login
    raise last_exception
  File "/usr/lib/python3.5/smtplib.py", line 720, in login
    initial_response_ok=initial_response_ok)
  File "/usr/lib/python3.5/smtplib.py", line 641, in auth
    raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (334, b'UGFzc3dvcmQ6')

The above log shows two issues:

- When server says '334 VXNlcm5hbWU6' is actually asking for the
  username, and instead the client replies with the password.
- The above would be enough to make the authentication fail, but
  there is more; when the server asks for the password,
  `334 UGFzc3dvcmQ6`, the client remains silent.
  
Due to the way this method is written, the final (code, resp),
which is (334, b'UGFzc3dvcmQ6') is raised as an error; but in
reality is just another challenge from the server.

The fix is in two part:

 - As of now, method auth_login is called twice only
   if initial_response_ok is set. The first time without challenge
   from line 627:
   
       initial_response = (authobject() if initial_response_ok else None)
       
   The second time with a challenge from line 636-637:
   
       response = encode_base64(
           authobject(challenge).encode('ascii'), eol='')
           
   auth_login will return the username the first time and the
   password the second time. Everything works. If the server
   supports the client sending the initial response and the
   client does so, everything works properly.
   
   But if initial_response_ok is False, the server will first
   ask for the username and, as shown before, the client will
   reply with the password. This is wrong and can be fixed by
   explicitly checking what the challenge is:
   
       def auth_login(self, challenge=None):
           print("auth_login", challenge)
           """ Authobject to use with LOGIN authentication. Requires self.user and
           self.password to be set."""
           if challenge is None:
               return self.user
           elif challenge == b'Username:':
               return self.user
           elif challenge == b'Password:':
               return self.password

 - Second thing to alter is the fact that if the server keeps
   issuing challenge, the client must respond. This happens
   because if initial_response_ok is False, the server will
   issue '334 VXNlcm5hbWU6' and '334 UGFzc3dvcmQ6' separately,
   and the client must solve both challenges. This is an easy fix
   as it simly amount to turning an if into a while:
               
   From:
   
        # If server responds with a challenge, send the response.
        if code == 334:
            challenge = base64.decodebytes(resp)
            response = encode_base64(
                authobject(challenge).encode('ascii'), eol='')
            (code, resp) = self.docmd(response)

   to:

        # If server responds with a challenge, send the response.
        # Note that there may be multiple, sequential challenges.
        while code == 334:
            challenge = base64.decodebytes(resp)
            response = encode_base64(
                authobject(challenge).encode('ascii'), eol='')
            (code, resp) = self.docmd(response)

With the two changes discussed above login works also for servers
that do not support initial_response_ok=True.
            
01:54:42.256276 send: 'ehlo localhost.localdomain\r\n'
01:54:42.458961 reply: b'250-smtp.example.com\r\n'
01:54:42.697463 reply: b'250-AUTH LOGIN\r\n'
01:54:42.697769 reply: b'250 8BITMIME\r\n'
01:54:42.697962 reply: retcode (250); Msg: b'smtp.example.com\nAUTH LOGIN\n8BITMIME'
01:54:42.700297 send: 'AUTH LOGIN\r\n'
01:54:42.902224 reply: b'334 VXNlcm5hbWU6\r\n'
01:54:42.902728 reply: retcode (334); Msg: b'VXNlcm5hbWU6'
01:54:42.903890 send: '<<<ENCODED_USERNAME>>>\r\n'
01:54:43.109534 reply: b'334 UGFzc3dvcmQ6\r\n'
01:54:43.109899 reply: retcode (334); Msg: b'UGFzc3dvcmQ6'
01:54:43.110521 send: '<<<ENCODED_PASSWORD>>>\r\n'
01:54:43.315055 reply: b'235 authentication successful\r\n'
01:54:43.315417 reply: retcode (235); Msg: b'authentication successful'
msg273405 - (view) Author: Mario Colombo (Mario Colombo) Date: 2016-08-22 21:31
Yes, this (or something similar) totally bit me, when for another unrelated reason 'AUTH PLAIN' authentication failed:

https://gist.github.com/macolo/bf2811c14d985d013dda0741bfd339e0

Python then tries auth_login, but doesn't send 'AUTH LOGIN' to the mail server. The second auth method also fails.
msg325234 - (view) Author: iMath (redstone-cold) Date: 2018-09-13 05:54
I encountered the same issue ,  Dario D'Amico's changing works ! please fix the problem !
History
Date User Action Args
2018-09-13 05:54:03redstone-coldsetnosy: + redstone-cold
messages: + msg325234
2018-07-11 07:44:41serhiy.storchakasettype: crash -> behavior
2016-08-22 21:31:18Mario Colombosetnosy: + Mario Colombo
messages: + msg273405
2016-08-21 05:39:43r.david.murraysetnosy: + barry, r.david.murray
components: + email
2016-08-21 00:35:21Dario D'Amicosetmessages: + msg273256
2016-08-21 00:34:58Dario D'Amicosetmessages: + msg273255
2016-08-21 00:30:09Dario D'Amicosetfiles: - SmtplibBugReport.txt
2016-08-21 00:29:36Dario D'Amicosetfiles: + SmtplibBugReport.txt
2016-08-21 00:29:10Dario D'Amicocreate