Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exception in BytesGenerator.flatten #71445

Closed
frispete mannequin opened this issue Jun 7, 2016 · 8 comments
Closed

Exception in BytesGenerator.flatten #71445

frispete mannequin opened this issue Jun 7, 2016 · 8 comments
Labels
topic-email

Comments

@frispete
Copy link
Mannequin

frispete mannequin commented Jun 7, 2016

BPO 27258
Nosy @warsaw, @bitdancer, @berkerpeksag, @pslacerda, @iritkatriel
Files
  • flatten-exception.mail
  • email_flatten.py
  • flatten-no-exception.mail
  • flatten-no-exception.mail
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-12-12.16:35:16.824>
created_at = <Date 2016-06-07.19:52:43.623>
labels = ['expert-email']
title = 'Exception in BytesGenerator.flatten'
updated_at = <Date 2021-12-12.16:35:16.823>
user = 'https://bugs.python.org/frispete'

    bugs.python.org fields:

    activity = <Date 2021-12-12.16:35:16.823>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-12.16:35:16.824>
closer = 'iritkatriel'
components = ['email']
creation = <Date 2016-06-07.19:52:43.623>
creator = 'frispete'
dependencies = []
files = ['43288', '43415', '43456', '43457']
hgrepos = []
issue_num = 27258
keywords = []
message_count = 8.0
messages = ['267736', '268645', '268658', '268673', '268759', '268807', '268810', '408397']
nosy_count = 6.0
nosy_names = ['barry', 'frispete', 'r.david.murray', 'berker.peksag', 'Pedro Lacerda', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue27258'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']
    @frispete
    Copy link
    Mannequin Author

    frispete mannequin commented Jun 7, 2016

    Attached mail, parsed with email.message_from_binary_file results in:

    Traceback (most recent call last):
  File "./mail_filter.py", line 616, in <module>
    ret = main.run()
  File "./mail_filter.py", line 605, in run
    self.process(fp)
  File "./mail_filter.py", line 589, in process
    self.save_message(msg, self._fname + '.out')
  File "./mail_filter.py", line 103, in save_message
    ofd.write(msg.as_bytes())
  File "/usr/lib64/python3.4/email/message.py", line 179, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib64/python3.4/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/lib64/python3.4/email/generator.py", line 195, in _write
    self._write_headers(msg)
  File "/usr/lib64/python3.4/email/generator.py", line 422, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
  File "/usr/lib64/python3.4/email/policy.py", line 190, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "/usr/lib64/python3.4/email/policy.py", line 204, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib64/python3.4/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 1228, in _fold
    rest._fold(folded)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 338, in _fold
    if folded.append_if_fits(part, tstr):
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 149, in append_if_fits
    token._fold(self)
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 324, in _fold
    for part in self.parts:
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 254, in parts
    if token.startswith_fws():
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
  File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws
    return self[0].startswith_fws()
IndexError: list index out of range

    when flattened with BytesGenerator.

    @frispete frispete mannequin added the topic-email label Jun 7, 2016
    @pslacerda
    Copy link
    Mannequin

    pslacerda mannequin commented Jun 16, 2016

    I was unable to reproduce this bug using the following snippet

        import email, sys
    from email.generator import BytesGenerator
    from email.mime.text import MIMEText

    fp = open('flatten-exception.mail', 'rb')
    email.message_from_binary_file(fp)
    bs = BytesGenerator(sys.stdout.buffer)
    bs.flatten(MIMEText('msg', 'plain', 'utf-8'))

    @berkerpeksag
    Copy link
    Member

    Thanks for helping to triage this, Pedro. I think there is a typo in your example: email.message_from_binary_file(fp) needs to be passed to bs.flatten().

    With the following script I'm also unable to reproduce the issue in Python 3.4+:

        import email
    import email.generator
    import sys

    with open('flatten-exception.mail', 'rb') as f:
        msg = email.message_from_binary_file(f)
        gen = email.generator.BytesGenerator(sys.stdout.buffer)
        gen.flatten(msg)

    Hans-Peter, could you share a reproducer with us? Thanks!

    @frispete
    Copy link
    Mannequin Author

    frispete mannequin commented Jun 16, 2016

    Sorry guys for not providing this earlier.

    It turned out, that the sub optimal behaviour is related to a unfortunate policy choice: email.policy.SMTP.

    @pslacerda
    Copy link
    Mannequin

    pslacerda mannequin commented Jun 18, 2016

    Seems that token.has_fws evaluates to True in the following condition

        if token.has_fws:

    causing token._fold(self) where isn't needed and raising the exception. Hope it helps!

    By the way, why the _header_value_parser.py was removed from the repository?
    https://github.com/python/cpython/blob/master/Lib/email/_header_value_parser.py#L144

    @pslacerda
    Copy link
    Mannequin

    pslacerda mannequin commented Jun 18, 2016

    Now the file is back! If any previous header has a newline before the value the error will not happen. But even with the output correct it isn't as expected.

    1 similar comment
    @pslacerda
    Copy link
    Mannequin

    pslacerda mannequin commented Jun 18, 2016

    Now the file is back! If any previous header has a newline before the value the error will not happen. But even with the output correct it isn't as expected.

    @iritkatriel
    Copy link
    Member

    I am also unable to reproduce the issue on 3.11 with the attached script and input.

    Please create a new issue if this problem still exists in a current python version (>= 3.9).

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    topic-email
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @berkerpeksag @iritkatriel