Title: email.Generator fails to flatten message parsed by email.Parser
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7, Python 2.6, Python 2.5
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: charlesc, r.david.murray
Priority: normal Keywords: patch

Created on 2010-02-20 00:25 by charlesc, last changed 2010-02-21 04:50 by r.david.murray. This issue is now closed.

File name Uploaded Description Edit charlesc, 2010-02-20 00:25 Script demonstrating error
testfile.txt charlesc, 2010-02-20 00:25 Example message which triggers the problem
email_message_rfc822.patch r.david.murray, 2010-02-20 16:01
Messages (7)
msg99606 - (view) Author: Charles Cazabon (charlesc) Date: 2010-02-20 00:25
email.Generator fails to flatten a message parsed by email.Parser; it throws an exception with an odd (but apparently legal) message.  First, the exception:

  File "/usr/local/lib/python2.6/email/", line 84, in flatten
  File "/usr/local/lib/python2.6/email/", line 109, in _write
  File "/usr/local/lib/python2.6/email/", line 135, in _dispatch
  File "/usr/local/lib/python2.6/email/", line 266, in _handle_message
    g.flatten(msg.get_payload(0), unixfrom=False)
  File "/usr/local/lib/python2.6/email/", line 189, in get_payload
    raise TypeError('Expected list, got %s' % type(self._payload))
TypeError: Expected list, got <type 'str'>

The oddity of the message its handling is that it's a single-part MIME message, where the content-type is declared as message/rfc822.  Most MUAs, when forwarding message, create a multipart MIME message with a single attachment part, and have the attachment be message/rfc822.  But Groupwise, when configured to forward messages to another address (without specifying an additional text to insert with the forwarded message) creates these slightly odd messages.

I've not seen them before, but a couple of getmail users have run into this issue.  I've confirmed the problem is the same in Python 2.3, 2.4, 2.5, and 2.6, and I presume it's the same in 2.7 but haven't tested yet.

I'll attach a minimal test script and a datafile which is a minimal message showing the problem.
msg99607 - (view) Author: Charles Cazabon (charlesc) Date: 2010-02-20 04:27
I've confirmed the problem still exists in 2.7a3.
msg99609 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-20 04:32
The problem only arises with HeaderParser.  The full parser turns the body into a list containing a Message object, because that's how the email package models the message structure.  HeaderParser treats the body as a single string.  All that is fine, but generator's flatten method has special handlers for certain mime types, and message/rfc822 is one of them.  It is looking for what it "knows" the message/rfc822 body looks like (a list containing a Message), but what it finds is a string.

The fix is probably to special case get_payload returning a string inside the _handle_message.  I've attached a patch that adds your test file as a unit test and provides that fix.
msg99620 - (view) Author: Charles Cazabon (charlesc) Date: 2010-02-20 15:39
Thanks, David - sounds good.  But I don't see the patch attached here.

msg99622 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-20 16:01
Hmm.  You are right.  Not sure if that was user error or browser error.  Let's try again.
msg99626 - (view) Author: Charles Cazabon (charlesc) Date: 2010-02-20 19:10
Yep, works perfect.  Thanks, David, for the extremely quick turnaround.
msg99643 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-21 04:50
Applied to trunk in r78274, 2.6 in r78275, py3k in r78276, and 3.1 in r78277.
Date User Action Args
2010-02-21 04:50:11r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg99643

stage: patch review -> resolved
2010-02-20 19:10:07charlescsetmessages: + msg99626
2010-02-20 16:01:41r.david.murraysetfiles: + email_message_rfc822.patch
keywords: + patch
messages: + msg99622
2010-02-20 15:39:41charlescsetmessages: + msg99620
2010-02-20 04:32:22r.david.murraysetpriority: normal

nosy: + r.david.murray
messages: + msg99609

assignee: r.david.murray
stage: patch review
2010-02-20 04:27:24charlescsetmessages: + msg99607
versions: + Python 2.7
2010-02-20 00:25:38charlescsetfiles: + testfile.txt
2010-02-20 00:25:10charlesccreate