classification
Title: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: matrixise, ned.deily, r.david.murray, thunderk
Priority: normal Keywords: patch

Created on 2018-01-30 15:10 by thunderk, last changed 2018-01-31 23:33 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5451 merged matrixise, 2018-01-30 16:48
PR 5455 merged miss-islington, 2018-01-31 00:03
Messages (17)
msg311275 - (view) Author: Michaël Lemaire (thunderk) Date: 2018-01-30 15:10
Hi,

I noticed that SMTP.send_message, when getting the sender and recipients from the Message object, strips the name from recipients (to keep only the address), but not from the sender.

        if from_addr is None:
            # Prefer the sender field per RFC 2822:3.6.2.
            from_addr = (msg[header_prefix + 'Sender']
                           if (header_prefix + 'Sender') in msg
                           else msg[header_prefix + 'From'])
        if to_addrs is None:
            addr_fields = [f for f in (msg[header_prefix + 'To'],
                                       msg[header_prefix + 'Bcc'],
                                       msg[header_prefix + 'Cc'])
                           if f is not None]
            to_addrs = [a[1] for a in email.utils.getaddresses(addr_fields)]

There is an ugly side-effect to that (starting with Python 3.5) : if the sender name contains a non-ascii character, send_message will then require the SMTPUTF8 option from the SMTP server, and raise a SMTPNotSupportedError if unavailable. This is not wanted because the sender name is not actually sent to the SMTP server in the "MAIL FROM:" command (it is only sent in the MIME payload), so the SMTPUTF8 option should not be required based on it (it should only depend on the addresses).
msg311276 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2018-01-30 15:31
Do you have an example with a test?

Thank you
msg311278 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-30 15:46
Yes, that looks like a bug.  Should be a one line bug fix, plus tests and news item if someone wants to make a PR...
msg311279 - (view) Author: Michaël Lemaire (thunderk) Date: 2018-01-30 15:47
Sure :

#coding: utf-8
import email.utils
from email.message import EmailMessage
from smtplib import SMTP
m = EmailMessage()
m['From'] = email.utils.formataddr(("Michaël", "michael@example.com"))
m['To'] = email.utils.formataddr(("René", "bounce@rodacom.fr"))
with SMTP('localhost') as smtp:
    smtp.set_debuglevel(2)
    smtp.send_message(m)
#END

On a server without SMTPUTF8, this outputs:

---
16:39:26.351302 send: 'ehlo localhost\r\n'
16:39:26.351391 reply: b'250-localhost\r\n'
16:39:26.351414 reply: b'250-PIPELINING\r\n'
16:39:26.351427 reply: b'250-SIZE 10240000\r\n'
16:39:26.351437 reply: b'250-VRFY\r\n'
16:39:26.351448 reply: b'250-ETRN\r\n'
16:39:26.351458 reply: b'250-ENHANCEDSTATUSCODES\r\n'
16:39:26.351468 reply: b'250-8BITMIME\r\n'
16:39:26.351477 reply: b'250 DSN\r\n'
16:39:26.351490 reply: retcode (250); Msg: b'localhost\nPIPELINING\nSIZE 10240000\nVRFY\nETRN\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
16:39:26.351832 send: 'QUIT\r\n'
16:39:26.351901 reply: b'221 2.0.0 Bye\r\n'
16:39:26.351923 reply: retcode (221); Msg: b'2.0.0 Bye'
Traceback (most recent call last):
  File "/usr/lib/python3.5/smtplib.py", line 943, in send_message
    ''.join([from_addr, *to_addrs]).encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\xeb' in position 5: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "testsmtp.py", line 12, in <module>
    smtp.send_message(m)
  File "/usr/lib/python3.5/smtplib.py", line 947, in send_message
    "One or more source or delivery addresses require"
smtplib.SMTPNotSupportedError: One or more source or delivery addresses require internationalized email support, but the server does not advertise the required SMTPUTF8 capability
---


With removing the accented character in the sender name :
m['From'] = email.utils.formataddr(("Michael", "michael@example.com"))

It works :

---
16:42:49.882358 send: 'ehlo localhost\r\n'
16:42:49.882437 reply: b'250-localhost\r\n'
16:42:49.882460 reply: b'250-PIPELINING\r\n'
16:42:49.882474 reply: b'250-SIZE 10240000\r\n'
16:42:49.882486 reply: b'250-VRFY\r\n'
16:42:49.882498 reply: b'250-ETRN\r\n'
16:42:49.882509 reply: b'250-ENHANCEDSTATUSCODES\r\n'
16:42:49.882520 reply: b'250-8BITMIME\r\n'
16:42:49.882530 reply: b'250 DSN\r\n'
16:42:49.882543 reply: retcode (250); Msg: b'localhost\nPIPELINING\nSIZE 10240000\nVRFY\nETRN\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
16:42:49.883439 send: 'mail FROM:<michael@example.com> size=86\r\n'
16:42:49.886567 reply: b'250 2.1.0 Ok\r\n'
16:42:49.886603 reply: retcode (250); Msg: b'2.1.0 Ok'
16:42:49.886666 send: 'rcpt TO:<bounce@rodacom.fr>\r\n'
16:42:49.889642 reply: b'250 2.1.5 Ok\r\n'
16:42:49.889676 reply: retcode (250); Msg: b'2.1.5 Ok'
16:42:49.889694 send: 'data\r\n'
16:42:49.889769 reply: b'354 End data with <CR><LF>.<CR><LF>\r\n'
16:42:49.889827 reply: retcode (354); Msg: b'End data with <CR><LF>.<CR><LF>'
16:42:49.889843 data: (354, b'End data with <CR><LF>.<CR><LF>')
16:42:49.889939 send: b'From: Michael <michael@example.com>\r\nTo: =?utf-8?q?Ren=C3=A9?= <bounce@rodacom.fr>\r\n\r\n.\r\n'
16:42:49.892726 reply: b'250 2.0.0 Ok: queued as D92C540105\r\n'
16:42:49.892752 reply: retcode (250); Msg: b'2.0.0 Ok: queued as D92C540105'
16:42:49.892764 data: (250, b'2.0.0 Ok: queued as D92C540105')
16:42:49.892786 send: 'QUIT\r\n'
16:42:49.892862 reply: b'221 2.0.0 Bye\r\n'
16:42:49.892896 reply: retcode (221); Msg: b'2.0.0 Bye'
---

As you can see, the sender and recipients names are not sent in SMTP command, except in the MIME body, so they should not be used to determine whether SMTPUTF8 is required or not.
msg311280 - (view) Author: Michaël Lemaire (thunderk) Date: 2018-01-30 16:24
Oups @r.david.murray, just saw I posted over your message, and removed you from the nozy list, sorry.
msg311282 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2018-01-30 16:40
@r.david.murray I started to work on a fix.
msg311284 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2018-01-30 16:50
@r.david.murray I am interested with your opinion about a test for this PR, if you have an idea, because all the tests pass on my laptop
msg311286 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-30 17:39
Yes, you'll have to write a test.  Add a test after test_send_unicode_without_SMTPUTF8, but put the unicode in the name field of the 'from' address and test that it is accepted.  You should be able to figure out how to check for success from the other tests in that class (I hope ;)
msg311296 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2018-01-30 20:07
@r.david.murray I just updated the PR with a test, but I am not sure about it, could you give me your opinion because the current test suite works fine without my new test.
msg311297 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-30 20:12
The current test suite had better work fine without your new test, otherwise your fix broke something :)  I will take a look.
msg311298 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2018-01-30 20:18
yep, you can take a look but all the tests are green on Travis, maybe you will understand why I don't see the interest of my new test. you are the master for the email part ;-)
msg311301 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-30 20:39
We need a test that will fail without your fix.
msg311302 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2018-01-30 21:31
The new test crashes without my fix but once we use the fix, all the
tests pass. I have modified the SimSMTPServer.process_manage method
msg311306 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-31 00:02
New changeset 8d83e4ba7823827bcbc119db887004d5c3a63dc6 by R. David Murray (Stéphane Wirtel) in branch 'master':
bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs (#5451)
https://github.com/python/cpython/commit/8d83e4ba7823827bcbc119db887004d5c3a63dc6
msg311370 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-31 20:54
New changeset 85a92d00bd278022cc00fda6b124c06f614c5ebb by R. David Murray (Miss Islington (bot)) in branch '3.6':
bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs (GH-5451) (#5455)
https://github.com/python/cpython/commit/85a92d00bd278022cc00fda6b124c06f614c5ebb
msg311374 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-01-31 21:05
Thanks, Stéphane.
msg311385 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-31 23:33
New changeset 209108bd6997258948d13f48f2b5a2f1220c1a35 by Ned Deily (Stéphane Wirtel) in branch '3.7':
bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs (#5451)
https://github.com/python/cpython/commit/209108bd6997258948d13f48f2b5a2f1220c1a35
History
Date User Action Args
2018-01-31 23:33:09ned.deilysetnosy: + ned.deily
messages: + msg311385
2018-01-31 21:05:31r.david.murraysetstatus: open -> closed
versions: - Python 3.4, Python 3.5
messages: + msg311374

resolution: fixed
stage: patch review -> resolved
2018-01-31 20:54:12r.david.murraysetmessages: + msg311370
2018-01-31 00:03:04miss-islingtonsetpull_requests: + pull_request5283
2018-01-31 00:02:53r.david.murraysetmessages: + msg311306
2018-01-30 21:31:11matrixisesetmessages: + msg311302
2018-01-30 20:39:21r.david.murraysetmessages: + msg311301
2018-01-30 20:18:51matrixisesetmessages: + msg311298
2018-01-30 20:12:05r.david.murraysetmessages: + msg311297
2018-01-30 20:07:24matrixisesetmessages: + msg311296
2018-01-30 17:39:37r.david.murraysetmessages: + msg311286
2018-01-30 16:50:27matrixisesetmessages: + msg311284
2018-01-30 16:48:59matrixisesetkeywords: + patch
stage: patch review
pull_requests: + pull_request5282
2018-01-30 16:40:53matrixisesetmessages: + msg311282
2018-01-30 16:24:11thunderksetnosy: + r.david.murray
messages: + msg311280
2018-01-30 15:47:51thunderksetnosy: - r.david.murray
messages: + msg311279
2018-01-30 15:46:45r.david.murraysetnosy: + r.david.murray
messages: + msg311278
2018-01-30 15:31:25matrixisesetnosy: + matrixise
messages: + msg311276
2018-01-30 15:10:32thunderkcreate