classification
Title: email.Generator does not separate headers with "\r\n"
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: Malcolm.Box, barry, l0nwlf, manlioperillo, r.david.murray, t-v
Priority: normal Keywords: easy, patch

Created on 2005-11-05 16:50 by manlioperillo, last changed 2010-10-23 22:22 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
email_linesep.patch r.david.murray, 2010-10-21 11:59
Messages (16)
msg54654 - (view) Author: Manlio Perillo (manlioperillo) Date: 2005-11-05 16:50
Regards.

The email.Generator module does not separates headers
with "\r\n".


Manlio Perillo
msg54655 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2006-01-17 05:35
Logged In: YES 
user_id=12800

Correct; this is by design.  If you're worried about
protocols such as RFC 2821 requiring \r\n line endings,
don't. The smtplib module automatically ensures proper line
endings for the on-the-wire communication.
msg54656 - (view) Author: Manlio Perillo (manlioperillo) Date: 2006-01-17 09:20
Logged In: YES 
user_id=1054957

Ok, thanks.
But what if I don't use the smtplib module?

I discovered the bug because I have written a small NNTP
server with twisted, using email module for parsing...
msg54657 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2006-01-17 12:54
Logged In: YES 
user_id=12800

The module that speaks the wire protocol should do the
conversion.  IMO, there's no other way to guarantee that
you're RFC compliant.  You could be getting your data from
the email package, but you could be getting it from anywhere
else, and /that/ source may not be RFC line ended either. 
Since you can't change every possible source of data for
NNTP or SMTP, your network interface must guarantee conformance.
msg54658 - (view) Author: Manlio Perillo (manlioperillo) Date: 2006-01-17 16:26
Logged In: YES 
user_id=1054957

I do not agree here (but I'm not an expert).

First - the documentation says:
"""The email package attempts to be as RFC-compliant as
possible, supporting in addition to RFC 2822, such
MIME-related RFCs as RFC 2045, RFC 2046, RFC 2047, and RFC 2231.
"""

But, as I can see, the generated email does not conform to
RFC 2822.

Second - I use email package as a "filter".
read raw email text, do some processing, generate raw email
text.

Really, I don't understand why generated headers don't are
separed by '\r\n' and one must rely on an external tool for
the right conversion.

Thanks.
msg54659 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2006-01-17 23:47
Logged In: YES 
user_id=12800

I hear what you're saying, but so far, it has been more
convenient for developers when the generator outputs native
line endings.  I can see a case for a flag or other switch
on the Generator instance to force RFC 2822 line endings.

I would suggest joining the email-sig and posting a request
there so the issue can be discussed as an RFE.
msg54660 - (view) Author: Manlio Perillo (manlioperillo) Date: 2006-01-20 10:05
Logged In: YES 
user_id=1054957

But the generator does not output in native line endings!

On Windows:
>>> from email.Message import Message
>>> msg = Message()
>>> msg["From"] = "me"
>>> msg["To"] = "you"
>>> print repr(msg.as_string())
'From: me\nTo: you\n\n'
msg54661 - (view) Author: Thomas Viehmann (t-v) Date: 2007-01-08 21:34
Hi,

could you please reconsider closing this bug and consider fixing it or at least providing an option for standard behaviour?
Leaving aside the question of performance impact of postprocessing in longer mails (for those, email may not a be good optionin the first place), the module as is renders the email.Generator mostly useless for multipart messages with binary data that needs to be standards compliant, e.g. Multipart-Messages containing images, possibly signed or uploading (with httplib) multipart/form-data.

Thank you for your consideration.

Kind regards

Thomas
msg54662 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2007-01-08 22:10
I am reopening this as a feature request.  I still think it's better for protocols that require these line endings to ensure that their data is standards compliant, but I can see that there may be other use cases where you'd want to generate protocol required line endings.  I'm not totally convinced, but it's worth opening the issue for now and discussing this on the email-sig.
msg97592 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-01-11 17:18
Consider also issue 975330.
msg108373 - (view) Author: Malcolm Box (Malcolm.Box) Date: 2010-06-22 13:29
Echoing the comment of Thomas Viehmann, the current behaviour makes it impossible to use this library to generate a correct multipart/mixed message with 7_or_8_bit encoding on Unix.

The MIME standard specifies that headers are to be CRLF terminated - this is independent of any lower-level transport encoding.  

Binary bodies are simply octet sequences - so may contain \n or \r characters.

The correct MIME encoding would have the headers terminated with CRLF, and the bodies as raw byte sequences.  However on Unix, since the headers are generated with \n not CRLF, you can't even post-process this (e.g. message.as_string().replace('\n', '\r\n') as that will screw up the binary body data.

The current behaviour is basically wrong and should be fixed.
msg119183 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-20 01:33
Malcolm: a Content-Transfer-Encoding of 8bit may only contain \r and \n characters as part of the line ending sequence.  8bit is *not* binary; to use a CTE of binary the SMTP server must support BINARYMIME, which I don't think is all that common yet.  At any rate, my up-to-date postfix server doesn't support it.

And in any case, the email package doesn't support the binary CTE, so it's kind of irrelevant anyway :)

All that said, however, it certainly seems useful to be able to generate crlf terminated messages as an option.  And it turns out that that capability is needed in order to be able to generate binary messages in Python3 while still remaining backward compatible with the behavior of Python2 when parsing binary messages (see issue 10134).  (Actually I think the scenario I'm finding problematic in Python3 is also problematic in Python2, it's just that there are tricks you can use to work around it in Python2 that aren't allowed in Python3 because of the byte/string separation.)

So, attached find a patch implementing a linesep option in Generator.flatten and Header.encode.

Note that this patch also fleshes out the currently abbreviated documentation for BytesGenerator.
msg119200 - (view) Author: Malcolm Box (Malcolm.Box) Date: 2010-10-20 10:58
David: Great to see a patch for this.

You're right of course, 8bit isn't binary - I meant binary.  The main place this shows up is when you're using MIME not in email (e.g. on the web), where binary transport is entirely possible.

This fix should mean that the MIME libraries are much more usable in non-email environments.  Thanks.
msg119277 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-21 11:54
Updated patch that adds a missing test for BytesGenerator.flatten and fixes the bugs in it.  Also added versionchanged tags to the docs for the linesep argument.

I think this is ready to go in.
msg119278 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-21 11:59
Removed some debugging cruft from the latest patch.
msg119478 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-23 22:22
Committed in r85811.
History
Date User Action Args
2010-10-23 22:22:46r.david.murraysetstatus: open -> closed
resolution: accepted
messages: + msg119478

stage: commit review -> resolved
2010-10-21 11:59:59r.david.murraysetfiles: + email_linesep.patch

messages: + msg119278
2010-10-21 11:59:03r.david.murraysetfiles: - email_linesep.patch
2010-10-21 11:54:43r.david.murraysetfiles: - email_linesep.patch
2010-10-21 11:54:32r.david.murraysetfiles: + email_linesep.patch

messages: + msg119277
stage: patch review -> commit review
2010-10-20 10:58:28Malcolm.Boxsetmessages: + msg119200
2010-10-20 01:33:15r.david.murraysetfiles: + email_linesep.patch
versions: - Python 2.7
title: email.Generators does not separates headers with "\r\n" -> email.Generator does not separate headers with "\r\n"
messages: + msg119183

keywords: + patch
stage: test needed -> patch review
2010-10-20 01:27:30r.david.murraylinkissue10134 dependencies
2010-06-22 14:53:31l0nwlfsetnosy: + l0nwlf
2010-06-22 13:29:10Malcolm.Boxsetnosy: + Malcolm.Box
messages: + msg108373
2010-05-05 13:37:58barrysetassignee: barry -> r.david.murray
2010-01-12 02:07:17r.david.murraylinkissue1571841 superseder
2010-01-11 17:18:54r.david.murraysetnosy: + r.david.murray

messages: + msg97592
versions: + Python 3.2, - Python 3.1
2009-03-20 23:12:11ajaksu2setkeywords: + easy
stage: test needed
versions: + Python 3.1, Python 2.7
2005-11-05 16:50:14manlioperillocreate