classification
Title: Header.decode_header eats up spaces
Type: behavior Stage: resolved
Components: email Versions: Python 3.3
process
Status: closed Resolution: duplicate
Dependencies: Superseder: decode_header does not follow RFC 2047
View: 1079
Assigned To: Nosy List: BreamoreBoy, alexanderweb, barry, georg.brandl, mgoutell, r.david.murray, runtux
Priority: normal Keywords:

Created on 2006-04-10 10:33 by mgoutell, last changed 2012-06-03 16:14 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
emailheader.diff georg.brandl, 2007-05-16 12:51 review
Messages (10)
msg28181 - (view) Author: Mathieu Goutelle (mgoutell) Date: 2006-04-10 10:33
The Header.decode_header function eats up spaces in
non-encoded part of a header.

See the following source:
# -*- coding: iso-8859-1 -*-
from email.Header import Header, decode_header
h = Header('Essai ', None)
h.append('éè', 'iso-8859-1')
print h
print decode_header(h)

This prints:
Essai =?iso-8859-1?q?=E9=E8?=
[('Test', None), ('\xe9\xe8', 'iso-8859-1')]

This should print:
Essai =?iso-8859-1?q?=E9=E8?=
[('Test ', None), ('\xe9\xe8', 'iso-8859-1')]
       ^ This space disappears

This appears in Python 2.3 but the source code of the
function didn't change in 2.4 so the same problem
should still exist. Bug "[ 1372770 ] email.Header
should preserve original FWS" may be linked to that one
although I'm not sure this is exactly the same.

This patch (not extensively tested though) seems to
solve this problem:

--- /usr/lib/python2.3/email/Header.py  2005-09-05
00:20:03.000000000 +0200
+++ Header.py   2006-04-10 12:27:27.000000000 +0200
@@ -90,7 +90,7 @@
             continue
         parts = ecre.split(line)
         while parts:
-            unenc = parts.pop(0).strip()
+            unenc = parts.pop(0).rstrip()
             if unenc:
                 # Should we continue a long line?
                 if decoded and decoded[-1][1] is None:
msg28182 - (view) Author: Alexander Schremmer (alexanderweb) Date: 2006-05-12 22:28
Logged In: YES 
user_id=254738

I can confirm this bug and have been bitten by it as well.
msg28183 - (view) Author: Mathieu Goutelle (mgoutell) Date: 2007-05-16 09:25
Hello,
Any news about this bug. It seems still there in 2.5 after a one year notice...
Regards,
msg28184 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2007-05-16 12:51
I propose the attached patch. RFC 2047 specifies to ignore whitespace between encoded-words, but IMHO not between ordinary text and encoded-words.
File Added: emailheader.diff
msg28185 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2007-05-16 13:08
IIRC, I tried the OP's patch and it broke too many of the email package's test suite.  I made an attempt at fixing the problem to be much more RFC compliant, but couldn't get the test suite to pass completely.  This points to a much deeper problem with email package header management.  I don't think the problem is a bug, I think it's a design flaw.
msg114651 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-22 09:11
Would someone like to comment on Georg's patch.
msg114722 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-08-23 01:25
Georg's patch no longer applies to py3k.  I ported it, but the result is not functional.  It causes extra spaces during header generation, because it is there that email4/5 "deals" with "ignoring" spaces between encoded words by *adding* spaces when they are adjacent to non-encoded words.  (In general email4/5 compresses runs of whitespace into single spaces.)  I tried fixing that, but then ran in to the fact that header parsing/generation currently depends on the whitespace compression in order to handle the header folding cases.  So, the logic used for header parsing and generation in emai5 does not allow for an easy patch to fix this bug.  I'm deferring it to email6, where I an rewriting the header parser/generator.
msg150459 - (view) Author: Ralf Schlatterbeck (runtux) * Date: 2012-01-02 15:50
I've been bitten by this too (in python up to 2.7 in roundup the bug-tracker). We're currently using a workaround that re-inserts spaces, see git on roundup.sourceforge.net file mailgw.py method _decode_header_to_utf8

RFC2047 even has a test-case at the end, it specifies:

encoded form                                displayed as
(=?ISO-8859-1?Q?a?= b)                      (a b)

note the space between 'a' and 'b' above. Spaces between non-encoded and encoded parts should be preserved. And it's probably a good idea to put the examples from the RFC into the regression test.
msg150470 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-02 17:55
Antoine, I marked this for Python 3.3 only because there is no good way to fix it in 2.7/3.2.  (If someone comes up with a way I'll be happy to review it, though!)
msg162219 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-03 16:14
This is fixed by the fix in issue 1079.  Ralf found a *relatively* backward compatible way to fix it, but since the point is preserving whitespace that wasn't preserved before, there is an unavoidable behavior change, so it can't be backported.
History
Date User Action Args
2012-06-03 16:14:50r.david.murraysetstatus: open -> closed
superseder: decode_header does not follow RFC 2047
messages: + msg162219

resolution: duplicate
stage: needs patch -> resolved
2012-05-24 14:43:55r.david.murraysetassignee: r.david.murray ->
components: + email, - Library (Lib)
2012-01-02 17:55:24r.david.murraysetmessages: + msg150470
versions: - Python 2.7, Python 3.2
2012-01-02 15:58:40pitrousetversions: + Python 2.7, Python 3.2
2012-01-02 15:50:44runtuxsetnosy: + runtux
messages: + msg150459
2010-09-05 23:02:59r.david.murraysetpriority: high -> normal
2010-08-23 01:26:00r.david.murraysetstage: patch review -> needs patch
messages: + msg114722
versions: + Python 3.3, - Python 3.1, Python 2.7, Python 3.2
2010-08-22 09:11:57BreamoreBoysetversions: + Python 3.2, - Python 2.6, Python 3.0
nosy: + BreamoreBoy

messages: + msg114651

stage: test needed -> patch review
2010-05-05 13:32:44barrysetassignee: barry -> r.david.murray

nosy: + r.david.murray
2009-03-21 03:25:08ajaksu2setstage: test needed
type: behavior
versions: + Python 2.6, Python 3.0, Python 3.1, Python 2.7, - Python 2.3
2006-04-10 10:33:54mgoutellcreate