Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zipfile: tuple IndexError on extract #81174

Closed
alter-bug-tracer mannequin opened this issue May 21, 2019 · 11 comments
Closed

zipfile: tuple IndexError on extract #81174

alter-bug-tracer mannequin opened this issue May 21, 2019 · 11 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@alter-bug-tracer
Copy link
Mannequin

alter-bug-tracer mannequin commented May 21, 2019

BPO 36993
Nosy @berkerpeksag, @serhiy-storchaka, @matrixise, @miss-islington, @danifus, @alter-bug-tracer
PRs
  • bpo-36993: Improve error detection of extra field in ZipFile #13727
  • bpo-36993: Improve error reporting for zipfiles with bad zip64 extra … #14656
  • [3.7] bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656) #16979
  • [3.8] bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656) #16980
    		•
    Files
  • index_tuple.zip
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-10-29.08:12:07.290>
created_at = <Date 2019-05-21.12:32:49.258>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'zipfile: tuple IndexError on extract'
updated_at = <Date 2019-10-29.08:12:07.283>
user = 'https://github.com/alter-bug-tracer'

    bugs.python.org fields:

    activity = <Date 2019-10-29.08:12:07.283>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-29.08:12:07.290>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2019-05-21.12:32:49.258>
creator = 'alter-bug-tracer'
dependencies = []
files = ['48348']
hgrepos = []
issue_num = 36993
keywords = ['patch']
message_count = 11.0
messages = ['343038', '343152', '344181', '344193', '344196', '345194', '347522', '355623', '355625', '355626', '355630']
nosy_count = 6.0
nosy_names = ['berker.peksag', 'serhiy.storchaka', 'matrixise', 'miss-islington', 'dhillier', 'alter-bug-tracer']
pr_nums = ['13727', '14656', '16979', '16980']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36993'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
    @alter-bug-tracer
    Copy link
    Mannequin Author

    alter-bug-tracer mannequin commented May 21, 2019

    The following code throws an IndexError when attempting to extract a malformed archive (attached):

    import zipfile
import sys

zf = zipfile.ZipFile(sys.argv[1])
for info in zf.infolist():
  zf.extract(info.filename)
    Result:
Traceback (most recent call last):
  File "code.py", line 4, in <module>
    zf = zipfile.ZipFile(sys.argv[1])
  File "/usr/local/lib/python3.8/zipfile.py", line 1230, in __init__
    self._RealGetContents()
  File "/usr/local/lib/python3.8/zipfile.py", line 1353, in _RealGetContents
    x._decodeExtra()
  File "/usr/local/lib/python3.8/zipfile.py", line 480, in _decodeExtra
    self.file_size = counts[idx]
IndexError: tuple index out of range

    @alter-bug-tracer alter-bug-tracer mannequin added 3.7 (EOL) end of life 3.8 only security fixes labels May 21, 2019
    @matrixise
    Copy link
    Member

    unzip index_tuple.zip -x
    Archive: index_tuple.zip

    caution: zipfile comment truncated
    error [index_tuple.zip]: missing 3992977728 bytes in zipfile
    (attempting to process anyway)
    skipping: zipfile_extract/ unsupported compression method 211

    I think the issue is not with Python but with your ZIP file. Did you try to uncompress it with unzip?\

    Thank you

    @berkerpeksag
    Copy link
    Member

    This report is valid. Serhiy has improved error reporting of the extra field in feccdb2.

    counts can indeed be an empty tuple:

    elif ln == 0:
    counts = ()

    If I'm reading section 4.5.3 of https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT correctly, I think we need to raise BadZipFile if ln == 0.

    @berkerpeksag berkerpeksag added the stdlib Python modules in the Lib dir label Jun 1, 2019
    @berkerpeksag berkerpeksag reopened this Jun 1, 2019
    @berkerpeksag berkerpeksag added type-bug An unexpected behavior, bug, or error and removed invalid labels Jun 1, 2019
    @serhiy-storchaka
    Copy link
    Member

    It is not enough. IndexError can be raised for ln == 8 or 16 when file_size, compress_size and header_offset are all set to 0xffffffff.

    @berkerpeksag
    Copy link
    Member

    @alter-bug-tracer, could you please create test files for the cases Serhiy has just mentioned?

    @alter-bug-tracer
    Copy link
    Mannequin Author

    alter-bug-tracer mannequin commented Jun 11, 2019

    @berker.peksag, first of all sorry for the late reply.
    We are not sure that we know how to do that. Our tests are generated automatically. What we can do is retest the lib with your temporary fixes in place, to see if they fix all the problems our software can detect. Would that help you?

    @danifus
    Copy link
    Mannequin

    danifus mannequin commented Jul 9, 2019

    I've pushed a PR which adds a test that generates corrupt zip64 files with different combinations of zip64 extra data lengths and zip64 flags (which determines how many fields are required in the extra data).

    It now raises a BadZipFile with a message naming the first missing field.

    @serhiy-storchaka
    Copy link
    Member

    New changeset da6ce58 by Serhiy Storchaka (Daniel Hillier) in branch 'master':
    bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)
    da6ce58

    @miss-islington
    Copy link
    Contributor

    New changeset f7d50f8 by Miss Skeleton (bot) in branch '3.7':
    bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)
    f7d50f8

    @miss-islington
    Copy link
    Contributor

    New changeset 3801b26 by Miss Skeleton (bot) in branch '3.8':
    bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)
    3801b26

    @serhiy-storchaka
    Copy link
    Member

    Thank you for your contribution Daniel.

    @serhiy-storchaka serhiy-storchaka added the 3.9 only security fixes label Oct 29, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @berkerpeksag @matrixise @serhiy-storchaka @miss-islington