classification
Title: email header indentation destroyed
Type: behavior Stage: test needed
Components: email Versions: Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: barry, frispete, r.david.murray
Priority: normal Keywords:

Created on 2016-06-07 19:14 by frispete, last changed 2016-06-16 15:33 by frispete.

Files
File name Uploaded Description Edit
utf8-header-failure.mail frispete, 2016-06-07 19:14 mail as read from email.message_from_binary_file
mf.9__mi0bf.out frispete, 2016-06-07 19:15 resulting mail from BytesGenerator.flatten
email_flatten.py frispete, 2016-06-16 15:33
Messages (3)
msg267732 - (view) Author: Hans-Peter Jansen (frispete) * Date: 2016-06-07 19:14
In the course of replacing an old Python 2.7 email filter tool with a rewritten Python3 version, I stumbled across a ugly case, where such an header:

X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTDJQUjAyTUI1MTQ7MjM6bEtRRlNaUHQvVTk5WCttdktlOUVrUGQvVFBH?=
 =?utf-8?B?cDFJemVUeXFzOGNzYnZOYWlwMDZpR0YzbXZyY09WaTBKM2pkeUl4S1VDMkxw?=
 =?utf-8?B?eVRkNWthRW9waUhJTzczTWd5WDZOQ3hMNU1haGFvQTVzVTdRZmxJUnZlblpW?=
 ...

is regenerated as:

X-Microsoft-Exchange-Diagnostics:
 1;BL2PR02MB514;23:lKQFSZPt/U99X+mvKe9EkPd/TPG
p1IzeTyqs8csbvNaip06iGF3mvrcOVi0J3jdyIxKUC2Lp
yTd5kaEopiHIO73MgyX6NCxL5MahaoA5sU7QflIRvenZV

which is plain wrong of course.

I'm using email.message_from_binary_file for parsing and BytesGenerator.flatten for regeneration. Since those are LKML public mails, I'm attaching both versions.

I'm using 3.4.4, but also the email module from current hg for testing.
msg267916 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-08 22:24
Hmm.  There appear to be at least two bugs here, using the SMTP policy.  I thought I had test cases like this, but clearly I don't :(
msg268675 - (view) Author: Hans-Peter Jansen (frispete) * Date: 2016-06-16 15:33
Sorry guys for not providing this earlier.

It turned out, that the sub optimal behaviour is related to a unfortunate policy choice: email.policy.SMTP.
History
Date User Action Args
2016-06-16 15:33:50frispetesetfiles: + email_flatten.py

messages: + msg268675
2016-06-10 23:34:11terry.reedysettitle: header indentation destroyed -> email header indentation destroyed
stage: test needed
type: behavior
versions: - Python 3.4
2016-06-08 22:24:26r.david.murraysetmessages: + msg267916
2016-06-07 19:15:48frispetesetfiles: + mf.9__mi0bf.out
2016-06-07 19:14:46frispetecreate