classification
Title: email.header error during .flatten()
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: python-dev, r.david.murray, sdaoden
Priority: normal Keywords:

Created on 2011-03-04 20:49 by sdaoden, last changed 2011-03-16 22:30 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
issue_11401_mail.txt sdaoden, 2011-03-05 13:01
issue_11401_runner.py sdaoden, 2011-03-05 13:01
Messages (8)
msg130072 - (view) Author: Steffen Daode Nurpmeso (sdaoden) Date: 2011-03-04 20:49
Hello, David, the error from Issue 11116 occurred again,
and it still can be healed with the patch file20675.
(I'm opening a new issue because 11116 turned over to
the mailbox fix.
My repo and installation is at:

21:34 ~/arena/code.extern.repos/py3k.hg $ hg identify
ddd8bf84e27d+ (py3k) tip
21:34 ~/arena/code.extern.repos/py3k.hg $ hg slog -r tip
changeset:   10028:ddd8bf84e27d
branch:      py3k
tag:         tip
user:        eli.bendersky
date:        Fri Mar 04 11:38:14 2011 +0100
summary:     [svn r88742]

Here is the dump:

| Opening mbox mailbox /private/var/folders/Da/DaZX3-k5G8a57zw6MSmjJ++++TM/-Tmp-/mail/test
David: <class 'mailbox.mbox'>  <->  <class 'mailbox.MaildirMessage'>
PANIC: [box] test: message-add failed.
  Traceback (most recent call last):
    File "/Users/steffen/usr/bin/s-postman.py", line 1093, in save_ticket
      mb.add(ticket.message())
	...
        try:
            print('David:', type(mb), ' <-> ', type(ticket.message()))
            mb.add(ticket.message())
            mb.flush()
        except Exception as e:
            efun('[box] ', self.id, ': message-add failed.\n', extb=E())
        finally:
            mb.unlock()
	...
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 595, in add
      self._toc[self._next_key] = self._append_message(message)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 733, in _append_message
      offsets = self._install_message(message)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 805, in _install_message
      self._dump_message(message, self._file, self._mangle_from_)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 215, in _dump_message
      gen.flatten(message)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line 88, in flatten
      self._write(msg)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line 141, in _write
      self._write_headers(msg)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line 373, in _write_headers
      self.write(header.encode(linesep=self._NL)+self._NL)
    File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/header.py", line 317, in encode
      formatter.feed(lines[0], charset)
  Exception: IndexError: list index out of range

In the meanwhile the thing has dispatched tens of thousands
of mails, and no error did ever occur (reading from MBox,
storing as Maildir).  This happened when saving the first
message read from a Maildir mailbox to a MBox one.
I think i'll gonna do some round-trip tests tomorrow and
monday evening+.  Anyway, patching in file20675 heals the
problem and the thing finishs gracefully.
msg130118 - (view) Author: Steffen Daode Nurpmeso (sdaoden) Date: 2011-03-05 13:01
> Changes by R. David Murray <rdmurray@bitdance.com>:
> assignee:  -> r.david.murray

David, i'm sending you data to reproduce the error on your local 
machine!

Please create a test directory and place the files therein 
(an output.mbox will be created).  Just do

  mkdir -p issue_11401/input.mdir/{cur,new,tmp}
  mv issue_11401_mail.txt "issue_11401/input.mdir/cur/test:2,S"
  mv issue_11401_runner.py issue_11401/
  cd issue_11401
  python3 issue_11401_runner.py

(I'm looking forward for a glass of red wine, just in case we'll 
ever meet.-)
msg130259 - (view) Author: Steffen Daode Nurpmeso (sdaoden) Date: 2011-03-07 14:36
David, it seems my patch http://bugs.python.org/file20675/email_header.patch 
is a real requirement for EMAIL 5.1 code.

This is because BytesGenerator._write_headers() creates a Header 
instance in 'else:' (the other branches are not entered at all for 
the test mail), passing 'v' as the 's=' arg of Header ctor. 
It follows that Header._chunks[] is never the empty list. 
'v' however may be the empty string for header fields like 
'X-Status:', 'X-Keywords:' and maybe more, resulting in 
string.splitlines() trying to break up the empty string.

I don't understand why i could pass thousands of mails, including 
the test mail, dozens of times through my thing without showing 
this up?  Does the mailbox.mbox message subclass adjust such things? 
Bye.
msg130264 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-03-07 17:09
I plan to take a look at this and other email bugs during the Pycon sprints, if not before.
msg131058 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-03-15 22:44
It turns out this is a bug in 3.1, not something introduced by email5.1 in 3.2.  The minimum reproducer is stringifying any message containing a header with no body.
msg131103 - (view) Author: Steffen Daode Nurpmeso (sdaoden) Date: 2011-03-16 11:47
On Tue, Mar 15, 2011 at 10:44:43PM +0000, R. David Murray wrote:
> header with no body.

(Those are the *real* dangerous ones!)
msg131187 - (view) Author: Roundup Robot (python-dev) Date: 2011-03-16 22:28
New changeset d9c3cfd36b58 by R David Murray in branch '3.1':
#11401: handle headers with no value.
http://hg.python.org/cpython/rev/d9c3cfd36b58

New changeset 2de45ced8291 by R David Murray in branch '3.2':
Merge #11401 fix from 3.1.
http://hg.python.org/cpython/rev/2de45ced8291

New changeset e318d84efe6e by R David Murray in branch 'default':
Merge #11401 fix from 3.2.
http://hg.python.org/cpython/rev/e318d84efe6e
msg131188 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-03-16 22:30
OK, now this bit works like it did in Python2.
History
Date User Action Args
2011-03-16 22:30:15r.david.murraysetstatus: open -> closed

messages: + msg131188
resolution: fixed
stage: needs patch -> resolved
2011-03-16 22:28:22python-devsetnosy: + python-dev
messages: + msg131187
2011-03-16 11:47:55sdaodensetmessages: + msg131103
2011-03-15 22:44:42r.david.murraysetversions: + Python 3.1
messages: + msg131058
stage: needs patch
2011-03-07 17:09:56r.david.murraysetmessages: + msg130264
2011-03-07 14:36:22sdaodensetmessages: + msg130259
2011-03-05 13:05:28sdaodensetversions: + Python 3.2
2011-03-05 13:01:08sdaodensetfiles: + issue_11401_mail.txt, issue_11401_runner.py

messages: + msg130118
2011-03-04 23:16:36r.david.murraysetassignee: r.david.murray
2011-03-04 20:49:44sdaodencreate