Title: maxlinelen exceeded by email module's body_encode() function
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 3.3
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: r.david.murray Nosy List: barry, michael.henry, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2011-03-19 12:13 by michael.henry, last changed 2011-03-24 16:30 by r.david.murray. This issue is now closed.

File name Uploaded Description Edit
quoprimime_patch_body_encode.patch michael.henry, 2011-03-19 12:13 unit tests and fix for body_encode() line overflow bug
quoprimime_body_encode_algorithm.patch r.david.murray, 2011-03-23 21:43
Messages (5)
msg131407 - (view) Author: Michael Henry (michael.henry) * Date: 2011-03-19 12:13
The email module's body_encode() function (found in can generate oversized encoded lines that exceed
the maximum line length specified by the maxlinelen parameter.
The attached test case
'test_encode_trailing_space_at_maxlinelen' demonstrates the
problem.  When encoding the body 'abcd \n1234' with
maxlinelen=5, the erroneous output is 'abcd =\n\n1234', with 6
characters in the first output line.

The attached patch provides unit tests and a fix for

Note: This patch depends on application of the patch in Issue
msg131925 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-03-23 21:43
Michael, in general your approach looks sound and is much easier to read and comprehend than the original code (which, as the comments say, was never refined from the original quick and dirty hack).  However, rather than dynamically defining sub-functions each time body_encode is called, I've moved the body-construction logic almost completely out of body_encode into a helper object.  I think this further clarifies the algorithm.  I also used a simpler approach to end-of-list detection (enumerate).  That change is more a matter of taste, but does have the advantage of taking fewer lines of code.

If you have time to review this and double check my changes (the tests pass, at least), that would be great.  Otherwise I'll just go ahead and apply it.
msg131963 - (view) Author: Michael Henry (michael.henry) * Date: 2011-03-24 10:41

Your patch looks fine to me.  I like putting the logic is a
separate class as you've done.  I looked in itertools for
something to perform the job of the each_last() generator I'd
had in my patch, but I didn't see anything.  I like the idea of
encapsulating the test logic of (index + 1 == len(sequence)) in
some way, as each_last() does, rather than having the caller
calculate it.  If that capability exists somewhere in the Python
standard library, it would be my choice to use that.  If it has
to be built just for this test, though, perhaps it's not worth
the extra lines of code to define each_last().

Michael Henry
msg131994 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-03-24 16:01
I turns out that issue 5803 has a patch that also fixes this bug, and the algorithm used there is even more efficient than the one you've developed here.  However, it is also not compatible with the email5 version of quoprimime.  It could be adapted, but I think I'm going to put off considering that until I can take a deeper look at why encode_body takes string as its input.  So I'm going to apply this patch in the meantime.
msg132002 - (view) Author: Roundup Robot (python-dev) Date: 2011-03-24 16:29
New changeset 37ba11d806c5 by R David Murray in branch '3.1':
#11606: improved body_encode algorithm, no longer produces overlong lines

New changeset b801d55a9979 by R David Murray in branch '3.2':
Merge #11606: improved body_encode algorithm, no longer produces overlong lines

New changeset 0c40f4939174 by R David Murray in branch 'default':
Merge #11606: improved body_encode algorithm, no longer produces overlong lines
Date User Action Args
2011-03-24 16:30:16r.david.murraysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2011-03-24 16:29:42python-devsetnosy: + python-dev
messages: + msg132002
2011-03-24 16:01:15r.david.murraysetmessages: + msg131994
versions: - Python 2.7
2011-03-24 10:41:07michael.henrysetmessages: + msg131963
2011-03-23 21:43:49r.david.murraysetfiles: + quoprimime_body_encode_algorithm.patch
nosy: + barry
messages: + msg131925

2011-03-19 14:11:38r.david.murraysetassignee: r.david.murray
versions: + Python 3.1, Python 2.7, Python 3.2, Python 3.3
type: behavior
stage: patch review
2011-03-19 12:13:16michael.henrycreate