classification
Title: 'UnstructuredTokenList' object has no attribute '_fold_as_ew'
Type: behavior Stage: resolved
Components: email Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Zhang Wenbo, archibald, barry, berker.peksag, r.david.murray, rslinckx, touilleMan, vtemian
Priority: normal Keywords: patch

Created on 2016-06-06 10:25 by touilleMan, last changed 2018-11-17 14:54 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
test_mail_2.py touilleMan, 2016-06-06 10:25
test2.py Zhang Wenbo, 2017-06-12 07:31
Pull Requests
URL Status Linked Edit
PR 3488 merged r.david.murray, 2017-09-11 02:07
PR 4693 merged python-dev, 2017-12-03 23:53
Messages (11)
msg267525 - (view) Author: Leblond Emmanuel (touilleMan) * Date: 2016-06-06 10:25
The folding code for email headers crashes whenever it hits line _header_value_parser.py:432 given it try to call method `_fold_as_ew`.

```
(venv)➜  vigiechiro-api git:(in2p3) ✗ python test_mail.py  
Traceback (most recent call last):
  File "test_mail.py", line 36, in <module>
    mail.send(msg)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 492, in send
    message.send(connection)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 427, in send
    connection.send(self)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 190, in send
    message.as_bytes() if PY3 else message.as_string(),
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 385, in as_bytes
    return self._message().as_bytes()
  File "/usr/lib/python3.5/email/message.py", line 179, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/lib/python3.5/email/generator.py", line 195, in _write
    self._write_headers(msg)
  File "/usr/lib/python3.5/email/generator.py", line 422, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
  File "/usr/lib/python3.5/email/policy.py", line 190, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "/usr/lib/python3.5/email/policy.py", line 204, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib/python3.5/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 1228, in _fold
    rest._fold(folded)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 432, in _fold
    part._fold_as_ew(folded)
AttributeError: 'UnstructuredTokenList' object has no attribute '_fold_as_ew'
> /usr/lib/python3.5/email/_header_value_parser.py(432)_fold()
-> part._fold_as_ew(folded)
```

Given folding depends a lot on the processed value, this bug only trigger in very specific encoded headers (I've try to reproduce this bug with various values but only the one which originally made me find the bug worked so far).

I've isolated the bug into a simple test case (see enclosed file).

It seems this part of the code has been rewritten between Python2 and 3 (the file _header_value_parser.py doesn't exists in Python2). So unsurprisingly the bug doesn't trigger in Python2.

Replacing `part._fold_as_ew(folded)` by `part._fold(folded)` seems to fix the bug, but given my poor knowledge of this code, this fix probably produce unwanted behaviors.
msg295751 - (view) Author: Zhang Wenbo (Zhang Wenbo) Date: 2017-06-12 07:31
I think I find a case which can reproduct this bug: 
the UnstructuredTokenList has a ValueTerminal which is long enough(its encoded word is longer than policy.maxlen ) and not at the beginning.

For examples, the following subjects can cause this bug:

- 'Re: 我只是一个有那么一点点长的主题而已'
- '回复: 回复:【北京大学-2017年招生信息公告】'

So, the code can not fold a ValueTerminal correctly. 
Seems the method `_fold_as_ew` was designed to do this, but it is missing.
msg301456 - (view) Author: Raphael Slinckx (rslinckx) Date: 2017-09-06 08:43
It happens regularly for emails with long header lines for me too, which makes the python3 email module quite unusable.

A workaround which seems to help a bit is to set refold_source='none' on the email policy, although it seems to keep happening, less often.

Is the fix proposed by Emmanuel a proper fix ? I'm not knowledgeable enough on this subject...
msg301481 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-09-06 16:49
I started rewriting the header folder in response to this bug, because the root problem is a bit deeper than just _fold_as_ew being missing. I will probably work on it some more this week, but if I don't think I'm going to get it finished I'll look at putting in a fix that doesn't address the other problems but does address this one.
msg301796 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-09-10 00:31
I managed to finish the rewrite, and it does fix this issue.  I'll probably post the PR tomorrow.
msg307528 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-12-03 23:51
New changeset 85d5c18c9d83a1d54eecc4c2ad4dce63194107c6 by R. David Murray in branch 'master':
bpo-27240 Rewrite the email header folding algorithm. (#3488)
https://github.com/python/cpython/commit/85d5c18c9d83a1d54eecc4c2ad4dce63194107c6
msg307530 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-12-04 00:46
New changeset a87ba60fe56ae2ebe80ab9ada6d280a6a1f3d552 by R. David Murray (Miss Islington (bot)) in branch '3.6':
bpo-27240 Rewrite the email header folding algorithm. (GH-3488) (#4693)
https://github.com/python/cpython/commit/a87ba60fe56ae2ebe80ab9ada6d280a6a1f3d552
msg307642 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-12-05 04:43
All three examples in test2.py correctly folded (doesn't hit the ``except Exception as e:`` branch anymore) in current master. Can this issue be closed now?
msg307659 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-12-05 12:21
Huh, I thought I had closed it.
msg330016 - (view) Author: Vlad Temian (vtemian) Date: 2018-11-17 00:58
Is there any chance that this fix is gonna be ported to 3.5 as well?
msg330030 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-11-17 14:54
No.  3.5 is in security-fix-only mode.
History
Date User Action Args
2018-11-17 14:54:32r.david.murraysetmessages: + msg330030
2018-11-17 00:58:16vtemiansetnosy: + vtemian
messages: + msg330016
2017-12-05 12:21:29r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg307659

stage: patch review -> resolved
2017-12-05 04:43:58berker.peksagsetnosy: + berker.peksag
messages: + msg307642
2017-12-04 00:46:25r.david.murraysetmessages: + msg307530
2017-12-03 23:53:07python-devsetpull_requests: + pull_request4605
2017-12-03 23:51:43r.david.murraysetmessages: + msg307528
2017-09-11 02:07:55r.david.murraysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request3478
2017-09-10 00:31:36r.david.murraysetmessages: + msg301796
2017-09-06 16:49:01r.david.murraysetmessages: + msg301481
versions: + Python 3.7, - Python 3.5
2017-09-06 08:43:39rslinckxsetnosy: + rslinckx
messages: + msg301456
2017-06-12 07:31:34Zhang Wenbosetfiles: + test2.py

messages: + msg295751
2017-06-12 06:02:18Zhang Wenbosetnosy: + Zhang Wenbo
2017-05-11 13:49:03archibaldsetnosy: + archibald
2016-06-06 11:19:26SilentGhostsetstage: needs patch
type: behavior
versions: - Python 3.2, Python 3.3, Python 3.4
2016-06-06 10:25:59touilleMancreate