I'm using CPython 2.7 with the smtplib and email modules to send emails with SMTP.
Today, one of our clients complained that the email sent is not RFC 5322 compliant because the required Date header is missing. The RFC states in section 3.6.:

"The only required header fields are the origination date field and
 the originator address field(s).  All other header fields are
  syntactically optional."

Our program has been sending millions of email message this way and this is the first complaint.

I guess that the library doesn't add the header field automatically.

That is correct.  Most SMTP gateways add the header on submission if it is missing.  At least a few other MUA programs do not automatically add the Date header, they let the SMTP server do it.  I have one person who sends me email that saw this same problem (I had my server set to reject messages without the date header), and they weren't using smtplib to send.  (I think the sender used Exchange, though I don't remember for sure.)

I can give a little more information.
First, I created a very simple stand-alone test script (for Python >= 2.6):

#!/bin/env python
# -*- coding: utf-8 -*-

import smtplib

# Adjust these!

HOST = "smtp.nowhere.local"
PORT = 25

from_name = u"Test Python smtplib"
from_addr = u"valid.address@nowhere.local"

to_name = u"Python Test Recipient"
to_addr = u"some.address@yourcompany.com"

subject = "Test Nr. 1"
smtp = smtplib.SMTP(HOST, PORT, timeout=10)
smtp.set_debuglevel(1)
smtp.sendmail(from_addr, [to_addr], subject)
smtp.quit()


This script shows the same behavior, which shows that the problem is not related to my program, but it is indeed a problem with smtplib.py.

Unfortunately, I could only test this with Python 2.6, because I'm not allowed to install Python 2.7 on that machine; OTOH I have to run it on that machine to have access to the mail server.

The observations are:

* The message is sent (I sent it to one of my email addresses).

* Spam Assassin flags the message with additional header lines: X-Amavis-Alert: BAD HEADER SECTION, Missing required header field: "Date"
X-Spam-Status: No, score=-2.36 ...
* In some cases the message is silently dropped, because the missing date header caues a negative score; which in turn may be bad enough to classify it as spam, depending on the content.
* The SMTP server used here is Postfix.
* When I receive the message, a Date header is present (obviously it has been inserted during the message's journey through the internet).

Personal opinion: 
I think classifying a message as spam just because a practically useless header field is missing is bad behavior of Spam Assaassin.
Nevertheless, the standard library should try to conform to RFC 5322.
It is good practise to be forgiving while reading but pedantic while writing.

As I implied but did not say explicitly, this is the expected behavior of smtplib.  You are responsible for adding any headers to the message that you want smtplib to send.  In particular, the 'sendmail' method takes a string to send, and smtplib does not modify it except for cr/lf transformation.  smtplib does not itself understand RFC5322 message syntax.

We can, however, add a Date header in the new send_message method of the python3 email library, because it accepts a Message object, and smtplib can use the knowledge the Message object encapsulates to check for the Date header and add one if it is missing.  That's why I've left this issue open.  I have now adjusted versions accordingly (ie: this is not a bug in python2.7, it is an enhancement request for Python3).  Sorry I wasn't clear about this earlier.

Hmm.  Actually, we can argue that it is an RFC compliance issue, as you have suggested, and change it in 3.6 as well, since it isn't likely to break anyone's working code.  So I'll put 3.6 in the versions unless someone objects to that logic.  But even after this is changed in python3, the smtplib sendmail method will not add a Date header, only the send_message method.

Hi,  Not sure this is where the comment goes...

I work with the smtplib and email libraries.  I understand Henning von Bargen when he say that we should have a way to support RFC 5322 without asking the user to understand how to support it.  The issue is that the SMTP protocol is NOT the protocol that format the e-mail.  SMTP is the protocol that identify the "from", the "to", start encryption and finally transfert the message.  The actual e-mail content is all passed inside the SMTP "DATA" Command.  I strongly believe that an email should not be modified by a SMTP library.

the discussion should be focused on trying to make it available to the user WITHOUT changing the current behavior of email.message class.  


In other words, I disagree to change the SMTPlib module and suggest that it's how you construct your email in the first place that should consider it;
class MessageRfc5322(email.message.Message):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self.get('Date', None) is None:
            self.add_header('Date', email.utils.formatdate())

msg = email.message_from_string(string_message, MessageRfc5322)



But, that's my opinion as someone who uses the smtplib and email library but also need to support rfc822 clients...

I tend to agree with Eric Lafontaine, looking at the quote Henning von Bargen posted the originator address field is also required, but yet we don't explicitly check its presence in the code, but rely on the SMTP server to error out.

The sendmail function will never modify the RFC822+ content.  send_message, however, already does several manipulations of the message headers to make sending email simpler.  Practicality (make it easy to send messages without knowing the details of the SMTP/RFC5322 rules) beats purity (an SMTP library should not modify the content of the DATA) in this case, especially since smtplib *does* provide the purity version if you prefer to work that way.

That is, 'sendmail' is pure SMTP, while send_message is a practical enhancement that provides additional services to the caller, and can (and does and probably should) do checks for RFC required headers.

If someone wants to do a purity refactor (to which I would not object, and would in fact encourage), the RFC5322-aware code could be factored out into one or more functions or Message object methods in the email library that smtplib would call from the send_message method.

My visualization is that the email library should allow you to construct a valid email in whatever order you want (adding the Date header late in the process, for example), but should support validating the email before it is sent.  One way to do this would be to have the SMTP policy do the unambiguous fixups such as date headers, and raising errors for the rest (probably only if the strict flag is on, at this point) when the message is serialized.

Note that we previously fixed send_message to add a Resent-Date header if there are Resent- headers and no Resent-Date, so the precedent is already set (that is, smptlib send_message is already "not pure" :)

I've chatted a bit with David about this feature. Here are some thoughts:
- check what SMTP standard says about some validation rules
- add validate method, probably into email package

Hi all,

Thanks for the enlightment.  I never figured that there was a send_message function XD.  Never needed it and it's true that the example in the email library use sendmail and not send_message.
https://docs.python.org/2/library/smtplib.html#smtplib.SMTP.sendmail
https://docs.python.org/3.5/library/smtplib.html#smtplib.SMTP.send_message

This function is fairly recent (python 3.2) from what I see.

Reading the documentation of the python 3.5 send_message function :
"[...] If from_addr is None or to_addrs is None, send_message fills those arguments with addresses extracted from the headers of msg as specified in RFC 5322: from_addr is set to the Sender field if it is present, and otherwise to the From field. to_addrs combines the values (if any) of the To, Cc, and Bcc fields from msg. "

As we're already using this function for convenience of the RFC 5322, then I agree to add it.  We should also modify the doc & comment inside the code to make it clear that date is added if absent and following RFC 5322. (I've looked at the source and the send_message only mention RFC2822 in the comments, no RFC 5322).

Finally, why would we want to add a validate fonction to the email library?  What would it do ? validate that we respect a certain RFC?  Who other than SMTPlib would use it?  I would like to understand the reasonning behind it.  

Again, all this are opinions to let the discussion continue :).

For now, what I see we need to do (this bullet point list is intended to be expanded with what you think we need to do):
- implement a patch for the code to add a missing "Date" field if it doesn't exist
- Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate
- Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).
- Fix it on all Python3 versions?  It should have been supported since 3.2 right?

As it's my first time trying to contribute... I still don't know how to do so...

Regards,
Eric Lafontaine
eric.lafontaine1@gmail.com <= if you can help me outside of this discussion to contribute, it would be my pleasure.

Sure the comments can be updated.  Some of them elsewhere have been already.

The reason for the email library to have a validation function is that it has an 'SMTP' policy that is designed to produce valid SMTP messages when the message is serialized.  (It also has an HTTP policy that is designed to produce valid HTTP header blocks...though I'm sure there are bugs there as well).  These policies are relatively new, even newer than the send_message method of smtplib.

If this gets done it time it could be fixed in 3.5...Larry is planning the final non-security-fix 3.5 release some time after 3.6.0 final goes out the door, as is our tradition.  After that it could only go into what will very shortly be the maintenance release (3.6) and the next feature release (3.7).

To contribute, create a diff against the tip of the default branch and post it here.  (We will be switching to github "soon", but posting a patch here will always work).  See docs.python.org/devguide for more details on contributing.

OK, I understand the arguments. 
If I understand correctly, this will be fixed in one way or another in Python 3.6 or 3.7.
For Python 2.7, this will not be fixed (so I have to work around this somehow, should be quite easy).
I think that at least the documentation and the examples for smtplib should be updated.

The doc should make it clear that the message is passed "as-is" and it's the caller's reponsibility to create a valid message string.

The examples at https://docs.python.org/2/library/email-examples.html should note which header fields are required by RFC 5322 and provide an example.

Hi all, I was looking at the code of the function and also noticed multiple "itching".

1- Why raise an error when 2 resent-date exist?  it may exist and completely legitimate as per RFC5322.  The code should always take the first one it sees as per the RFC.
2- Also, there was an "ehlo_or_helo_if_needed", but it's done in sendmail function so makes it always irrelevant.
3- The signature of the function is different from the one of sendmail function.  My understanding is that send_message is a replacement of sendmail with more feature (praticality) and should be simple to change by just swaping the 2 of them. 
4- This "send_message" shouldn't be dependant on the email.message.Message class.  It should allow a dictionary to be passed as well.  So I removed the "get_all" to be replaced with a "get" method (which align with what dictionaries do. 


I have addressed 1,2 and 4 in my patch (modified the test case as well) and my efforts to make this better and clearer, but number 3 seems to be something that may be a big change for clients... Should we handle it at all?

Please review and give me feedbacks please, I'm open to negative ones as well ;).  I need to know if I'm doing something wrong and I don't know all the principle (as you may have noticed).  

I've ran the "python -m test" & "./python -m smtplib" command, but is there something else I should've done?

List of things to do :
- Implement a patch for the code to add a missing "Date" field if it doesn't exist .   (in review)
- Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate
- Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).  ( in review)
- Modify the example of Python 2 to present a RFC 5322 examples

Thanks Henning von Bargen, David, maciej.  I appreciate your support.  

Regards,
Eric Lafontaine

The signature of send_message cannot be changed for backward compatibility reasons.  It's signature was not intended to be an extension of the sendmail signature.

send_message cannot accept a dictionary, as then it would have to know how to format that dictionary into an RFC5322 compliant text string.  Nor can a dictionary represent a collection of email headers accurate.  Representing the headers and doing the serialization is the email package's job.

It is true that the ehlo_or_helo_if_needed is technically redundant, however, what it means is that once you've called any smtplib command you know that has been done, even if the subcommand raises an error.  This can make debuging easier (consistency of state), at a trivial cost.

The "more than one Resent block" error is explained in the comments.  If you want to write the heuristics to remove it, go ahead :)

Looking at that code, I see that I misremembered what it was doing.  I thought it was adding Resent-Date if it was missing, but it is instead determining what form of address headers it needs to add if Reset-Date is present.  That doesn't change the analysis in this issue, though.

Hi all,

this is not a short answer, like I hoped it would be ;).  But skip to the last section if you don't want to read it all.
________________________________
the heuristic problem :

For the heuristic of the resent headers, it's clearly say in the RFC5322 that all resent block should be PREPENDED to messages.  So only the first ones you see should be handled.  email.message keeps the order of the headers while reading and "get" always takes the first one if it exist :).    

from the RFC: 
   Each new set of resent fields is prepended to the
   message; that is, the most recent set of resent fields appears
   earlier in the message.  No other fields in the message are changed
   when resent fields are added.

For the behavior of the send_message, we used Resent-Date as it should always be present for EACH set of resent : 
   When resent fields are used, the "Resent-From:" and "Resent-Date:"
   fields MUST be sent.  The "Resent-Message-ID:" field SHOULD be sent.
   "Resent-Sender:" SHOULD NOT be used if "Resent-Sender:" would be
   identical to "Resent-From:".
   
It's fantastic that things work well like this for coders :).

The issues is with the msg object being passed...  It does retain the order, but doesn't prepend new-headers... it only append new headers (putting them at the bottom of the e-mail).

If someone wants to prepend headers (i.e. the "Resend-"), they will have to take the msg obj and do the addition manually; 
    msg._headers.insert(0,msg.policy.header_store_parse(name, val)).
    
They already have to do it though as the as_string function will print them at the bottom anyway.  Changing the "__setitem__" of message change the behavior too much...  even though it would be the right thing to do... email.message read e-mail from top to bottom and "set" each line.  In other words, for order dependant headers like the Resent, we're screwed.

i.e. from the test case about the multiple resent, here is what is sent down the "data" smtp command :
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: foo@bar.com
To: John
CC: Sally, Fred
Bcc: John Root <root@localhost>, "Dinsdale" <warped@silly.walks.com>
Resent-Date: Thu, 1 Jan 1970 17:42:00 +0000
Resent-From: holy@grail.net
Resent-To: Martha <my_mom@great.cooker.com>, Jeff
Resent-Bcc: doe@losthope.net
Resent-Date: Thu, 2 Jan 1970 17:42:00 +0000
Resent-To: holy@grail.net
Resent-From: Martha <my_mom@great.cooker.com>, Jeff
Date: Sun, 18 Dec 2016 20:32:11 -0000

A test message


___________________________________

Finally, for the "ehlo_or_helo_if_needed", I thought about it and here's what I think and what I believe we should be doing (I'm not attached to the idea though);
    The send_message is a higher level of thinking and should be able to be modified if someone wants to act differently by sub-classing the smtplib.SMTP class.  in other words, if that someone wants to modify the SMTP behavior, he could modify the sendmail function, which would also impact the send_message function.
    
    This leads the developpers to have to use ehlo and prepare the connection for send_message, but doesn't enforce it :).  Which would also already be the case of doing it with sendmail and that instead of having to over-ride 2 methods, only one is over-written.  
    
    If this is not desired, than I suggest "protecting our reference" by using thunder sendmail instead.
    
I prefer to remove the ehlo from the send_message and allow "flexibility" on the send_message itself.  
___________________________________

The issue I have right now, is what David Mentionned about the heuristic... I should raise an error when 2 Resent-Date are found, but that will prevent me from using it for production ( as I have more than one resent field present sometime...).  This would also prevent people from using it and I find it sad.  For now, i'm guessing that adding a keyword parameter "guess" would be the right things to do, but will let that be in an other ticket.  David, I find that somehow the RFC5322 isn't clear enough on the "Resent-" header order, so I've written to the IETF to have some explanation on the rules in edge cases.  The heuristic is implemented (takes the first one it finds), but just not activated yet for multiple Resent-Date.

I'll re-iterate my need for someone to review the code though :).

Eric Lafontaine

forgot to re-post the to-do, and to correct the documentation :
List of things to do :
- Implement a patch for the code to add a missing "Date" field if it doesn't exist .   (in review)
- Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate 
- Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).  ( in review)
- Modify the example of Python 2 to present a RFC 5322 examples 

Feedback welcome as always!

I think Eric's To-Do list sums it up pretty well.

I just wanted to say how much I appreciate the Python community. It seems that every little detail is very well-thought-out.

There is at least one open ticket (maybe two, I forget) about providing a way to insert headers at arbitrary locations in the header list.  Our trouble has been deciding on an API.  There is a mailing list (email-sig) where such things can be discussed (it has been idle for a while now...).

I think you are right about the 'heuristic'.  Thinking about it now I believe the problem was partly the lack of an API for inserting headers at the top: if there is only one resent block we are OK appending the address headers, but if there is more than one block then that would be wrong.  The other aspect, to which the comment is referring, is that if we are "augmenting" headers, how do we decide if we are augmenting an existing block, or creating a new one?  Looking at it now with fresh eyes I think we could just decide on the rules and document them.  And we probably want an option that says "this is a resend, add all the Resent headers as a new block".  Or maybe that is enough by itself.  No heuristics, refuse to guess :)  Either the user adds all the headers themselves, or none of them, and we construct them in send_message.  (I haven't reviewed the RFC lately, so I'm not sure that covers all the use cases.)

I don't have a strong opinion on the ehlo_helo issue, but I'm not clear on why its presence causes any difficulty for subclassing.

Hi all,

@David, the "Resent-" block should be added by the user and he needs to know what he's doing.  Hell, I work with emails all the time and never knew about this.  The Worse part of it is that I have to communicate with the IETF just to know how we can do the rules... that doesn't mean another programmer didn't interpret it wrong...

@Henning, this is my first contribution :P  I've just been brain-washed by Raymond Hettinger and his beautifuls speechs.  I suggest (no shame plug) you look at the first 5-6 videos of the following link if you got free time.  https://www.youtube.com/playlist?list=PLRVdut2KPAguz3xcd22i_o_onnmDKj3MA

I've received his "call to arms" ; https://www.youtube.com/watch?v=-TdrFjDJn5E&list=PLRVdut2KPAguz3xcd22i_o_onnmDKj3MA&index=16 

David, I said the API was awkward because of what is mention in this latest link.  I really believe we ought to change it ASAP if it's possible at all and make it more like sendmail... that's only an opinion though.  https://youtu.be/-TdrFjDJn5E?list=PLRVdut2KPAguz3xcd22i_o_onnmDKj3MA&t=4231

Finally, Could I know how to modify the Doc of the Python2 ? 
Regards,
Eric Lafontaine


- Implement a patch for the code to add a missing "Date" field if it doesn't exist .   (in review)
- Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate (in review)
- Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).  (in review)
- Modify the example of Python 2 to present a RFC 5322 examples

The documentation is all in the Doc subdirectory of the checkout.

So, this should teach me to rely on vague memory instead of reading the code.  I've probably still missed something because I don't have enough time to read it properly.

Yes, you are correct, the Resent headers need to be added by the user.  So adding a Resent-Date header is the only question, and we can assume that it should go before the first Resent-X header we find.

Here is my reconstruction of why I made that restriction and comment:  There is no specified order to the Resent- block headers.  So how do we know where the first block ends and the next begins?  It will almost always be at the Resent-Date header...but we can't assume that that will *always* be true.

For sendmail/send_message, I'm not going to look at a youtube video, so you'll have to make your argument in text :)  But you may not want to bother, it is indeed too late: we can't change the signature for backward compatibility reasons.  In any case IMO the signature is optimal as it is: the only required argument is msg, and that, therefore, has to come first.  Otherwise the arguments are already in the same order as they are for sendmail.

I spent a lot of time reading RFCs while working on these libraries, but it has been a while since my head was buried in that space :)

Hi David, Henning,

Sorry for the delay.  Here is the patch for python 2... but I find that adding an "example" just for RFC 5322 is kind of over-kill...

So I've made both :) please choose which one you prefer and let me know.  I personnally prefer to use the non-overkill (which is only a comment and a line of code).

David, I know you might not like it, but could you be my mentor?  I feel you have an understanding of messaging applications :).  I know you don't have much time, but I feel you're a big participant and you could delegate some tasks on to me.  

Regards,
Eric Lafontaine

- Implement a patch for the code to add a missing "Date" field if it doesn't exist .   (in review)
- Modify the documentation at the SMTPLib for the send_message to mention that it add missing date using the email.utils.formatdate 
- Modify the comment of the send_message code to mention RFC 5322 in there (ideally with the section of the RFC).  ( in review)
- Modify the example of Python 2 to present a RFC 5322 examples ( in review)

With this, I've done all I could... What else can I start to do to get reviewed?

overkill file

I'm happy to comment on issues as far as mentoring goes, and yes email is my primary area of responsibility for CPython.  However, I don't have much free time, so getting to reviews is proving to be problematic.  There's a chance a might have some time this weekend, but no promises :)

Hi all,

The IETF didn't answer yet :(.  

I'll await your news regarding this patch ("issue_28879_V4.patch").

I would like to have feedback if I need to change something.

Thanks a lot in advance,
Eric Lafontaine

Hi all,

I've received an answer from the IETF Pete Resnick.  The answer does say however that there is no guaranteed way of getting the right headers if the message doesn't respect the standard; 

"[...] 
In this case, if the trace fields were not present, you would not be able to clearly distinguish. Certainly you know which blocks the Resent-Date and Resent-From belong to, and in this case you know which blocks the Resent-Message-ID belong to (since there are two of them and only two Resent-* blocks), and you know which block the Resent-Bcc belongs to (because it comes between the first Resent-Date and Resent-Message-ID), but you can't tell which block the Resent-To: belongs to. And if the either Resent-Message-ID was missing, you would be unable to tell where the Resent-Bcc or Resent-Message-ID belongs. This is simply a weakness in the standard.

However, the trace fields should exist, and that should divide the Resent-* blocks. As it says in 3.6:

header fields SHOULD NOT be reordered when a message is transported
or transformed. More importantly, the trace header fields and resent
header fields MUST NOT be reordered, and SHOULD be kept in blocks
prepended to the message.

I hope that helps.


pr
-- 
Pete Resnick http://www.qualcomm.com/~presnick/
Qualcomm Technologies, Inc. - +1 (858)651-4478
"
______________

In other words, if we were to take an e-mail that would have more than one Resent- headers, there should be traces in between the resent block.  If there are no traces for a Resent- block, and we can detect there are 2 blocks in what is supposed to be a block , we should raise an error.  

However, if we were to do the implementation and make it to detect "blocks" of resents (with the position of the resent for example).  We could use the first block as the one we should be using and see if there are 2 Resent-Dates or 2 Resent-From in that block only.

What do you think David?  I would like to know :).

P.s. I can forward the e-mail to does who want to have it :).

Regards,
Eric Lafontaine

I think this constitutes the heuristic I was talking about in that comment, that will get it right 99+% of the time.  Strict mode should raise an error, but strict is not the default in the email package.

I probably won't have time to do any review for a while yet, I'm afraid.

Hi,

I've implemented the heuristic, but it's messy with the issue of this ticket. 

I'm going to do some clean-up and separate the issue from the heuristic and post them separated.

Date issue ;
Test Case
Documentation
Implementation

Heuristic of Resent ;
Test Case
Documentation
Implementation


Regards,
Eric Lafontaine

Hi all,

Here you go :).

Regards,
Eric Lafontaine

Resent-heuristic

Hi,

Could someone put this ticket on "waiting for review"?

Regards,
Eric Lafontaine

Hi All,

Should I try to make this a github PR instead to accelerate the review?

Regards,
Eric Lafontaine

Yes.  There's a chance someone else will review it if you do that.  I'm still not likely to have time for a while, myself, but ping me again in a month if I haven't gotten to it.

Any updates on this? Looks like the proposed change has not been merged into mainstream yet?

I’m having problems with Google rejecting emails:

    (555, b'5.5.2 Syntax error, goodbye. r10-v6sm7321838qtj.41 - gsmtp', '…')

and using IETF’s message linter (https://tools.ietf.org/tools/msglint/) I get the following:

    ERROR: missing mandatory header 'date' lines 1-7
    ERROR: missing mandatory header 'return-path' lines 1-7

amongst a few others.