Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'UnstructuredTokenList' object has no attribute '_fold_as_ew' #71427

Closed
touilleMan mannequin opened this issue Jun 6, 2016 · 11 comments
Closed

'UnstructuredTokenList' object has no attribute '_fold_as_ew' #71427

touilleMan mannequin opened this issue Jun 6, 2016 · 11 comments
Labels
3.7 (EOL) end of life topic-email type-bug An unexpected behavior, bug, or error

Comments

@touilleMan
Copy link
Mannequin

touilleMan mannequin commented Jun 6, 2016

BPO 27240
Nosy @warsaw, @bitdancer, @berkerpeksag, @touilleMan, @ArchibaldArt, @rslinckx, @vtemian
PRs
  • bpo-27240 Rewrite the email header folding algorithm. #3488
  • [3.6] bpo-27240 Rewrite the email header folding algorithm. (GH-3488) #4693
    		•
    Files
  • test_mail_2.py
  • test2.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-12-05.12:21:29.872>
created_at = <Date 2016-06-06.10:25:59.132>
labels = ['type-bug', '3.7', 'expert-email']
title = "'UnstructuredTokenList' object has no attribute '_fold_as_ew'"
updated_at = <Date 2018-11-17.14:54:32.694>
user = 'https://github.com/touilleMan'

    bugs.python.org fields:

    activity = <Date 2018-11-17.14:54:32.694>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-05.12:21:29.872>
closer = 'r.david.murray'
components = ['email']
creation = <Date 2016-06-06.10:25:59.132>
creator = 'touilleMan'
dependencies = []
files = ['43257', '46941']
hgrepos = []
issue_num = 27240
keywords = ['patch']
message_count = 11.0
messages = ['267525', '295751', '301456', '301481', '301796', '307528', '307530', '307642', '307659', '330016', '330030']
nosy_count = 8.0
nosy_names = ['barry', 'r.david.murray', 'berker.peksag', 'touilleMan', 'archibald', 'Zhang Wenbo', 'rslinckx', 'vtemian']
pr_nums = ['3488', '4693']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27240'
versions = ['Python 3.6', 'Python 3.7']
    @touilleMan
    Copy link
    Mannequin Author

    touilleMan mannequin commented Jun 6, 2016

    The folding code for email headers crashes whenever it hits line _header_value_parser.py:432 given it try to call method _fold_as_ew.

    (venv)➜  vigiechiro-api git:(in2p3) ✗ python test_mail.py  
Traceback (most recent call last):
  File "test_mail.py", line 36, in <module>
    mail.send(msg)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 492, in send
    message.send(connection)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 427, in send
    connection.send(self)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 190, in send
    message.as_bytes() if PY3 else message.as_string(),
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 385, in as_bytes
    return self._message().as_bytes()
  File "/usr/lib/python3.5/email/message.py", line 179, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/lib/python3.5/email/generator.py", line 195, in _write
    self._write_headers(msg)
  File "/usr/lib/python3.5/email/generator.py", line 422, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
  File "/usr/lib/python3.5/email/policy.py", line 190, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "/usr/lib/python3.5/email/policy.py", line 204, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib/python3.5/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 1228, in _fold
    rest._fold(folded)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 432, in _fold
    part._fold_as_ew(folded)
AttributeError: 'UnstructuredTokenList' object has no attribute '_fold_as_ew'
> /usr/lib/python3.5/email/_header_value_parser.py(432)_fold()
-> part._fold_as_ew(folded)

    Given folding depends a lot on the processed value, this bug only trigger in very specific encoded headers (I've try to reproduce this bug with various values but only the one which originally made me find the bug worked so far).

    I've isolated the bug into a simple test case (see enclosed file).

    It seems this part of the code has been rewritten between Python2 and 3 (the file _header_value_parser.py doesn't exists in Python2). So unsurprisingly the bug doesn't trigger in Python2.

    Replacing part._fold_as_ew(folded) by part._fold(folded) seems to fix the bug, but given my poor knowledge of this code, this fix probably produce unwanted behaviors.

    @touilleMan touilleMan mannequin added the topic-email label Jun 6, 2016
    @SilentGhost SilentGhost mannequin added the type-bug An unexpected behavior, bug, or error label Jun 6, 2016
    @ZhangWenbo
    Copy link
    Mannequin

    ZhangWenbo mannequin commented Jun 12, 2017

    I think I find a case which can reproduct this bug:
    the UnstructuredTokenList has a ValueTerminal which is long enough(its encoded word is longer than policy.maxlen ) and not at the beginning.

    For examples, the following subjects can cause this bug:

    • 'Re: 我只是一个有那么一点点长的主题而已'
    • '回复: 回复:【北京大学-2017年招生信息公告】'

    So, the code can not fold a ValueTerminal correctly.
    Seems the method _fold_as_ew was designed to do this, but it is missing.

    @rslinckx
    Copy link
    Mannequin

    rslinckx mannequin commented Sep 6, 2017

    It happens regularly for emails with long header lines for me too, which makes the python3 email module quite unusable.

    A workaround which seems to help a bit is to set refold_source='none' on the email policy, although it seems to keep happening, less often.

    Is the fix proposed by Emmanuel a proper fix ? I'm not knowledgeable enough on this subject...

    @bitdancer
    Copy link
    Member

    I started rewriting the header folder in response to this bug, because the root problem is a bit deeper than just _fold_as_ew being missing. I will probably work on it some more this week, but if I don't think I'm going to get it finished I'll look at putting in a fix that doesn't address the other problems but does address this one.

    @bitdancer bitdancer added the 3.7 (EOL) end of life label Sep 6, 2017
    @bitdancer
    Copy link
    Member

    I managed to finish the rewrite, and it does fix this issue. I'll probably post the PR tomorrow.

    @bitdancer
    Copy link
    Member

    New changeset 85d5c18 by R. David Murray in branch 'master':
    bpo-27240 Rewrite the email header folding algorithm. (bpo-3488)
    85d5c18

    @bitdancer
    Copy link
    Member

    New changeset a87ba60 by R. David Murray (Miss Islington (bot)) in branch '3.6':
    bpo-27240 Rewrite the email header folding algorithm. (GH-3488) (bpo-4693)
    a87ba60

    @berkerpeksag
    Copy link
    Member

    All three examples in test2.py correctly folded (doesn't hit the ``except Exception as e:`` branch anymore) in current master. Can this issue be closed now?

    @bitdancer
    Copy link
    Member

    Huh, I thought I had closed it.

    @vtemian
    Copy link
    Mannequin

    vtemian mannequin commented Nov 17, 2018

    Is there any chance that this fix is gonna be ported to 3.5 as well?

    @bitdancer
    Copy link
    Member

    No. 3.5 is in security-fix-only mode.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life topic-email type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @berkerpeksag @bitdancer