classification
Title: zipfile: tuple IndexError on extract
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alter-bug-tracer, berker.peksag, dhillier, matrixise, miss-islington, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-05-21 12:32 by alter-bug-tracer, last changed 2019-10-29 08:12 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
index_tuple.zip alter-bug-tracer, 2019-05-21 12:32
Pull Requests
URL Status Linked Edit
PR 13727 open berker.peksag, 2019-06-01 16:13
PR 14656 merged dhillier, 2019-07-09 04:35
PR 16979 merged miss-islington, 2019-10-29 07:25
PR 16980 merged miss-islington, 2019-10-29 07:26
Messages (11)
msg343038 - (view) Author: alter-bug-tracer (alter-bug-tracer) * Date: 2019-05-21 12:32
The following code throws an IndexError when attempting to extract a malformed archive (attached):

import zipfile
import sys

zf = zipfile.ZipFile(sys.argv[1])
for info in zf.infolist():
  zf.extract(info.filename)

Result:
Traceback (most recent call last):
  File "code.py", line 4, in <module>
    zf = zipfile.ZipFile(sys.argv[1])
  File "/usr/local/lib/python3.8/zipfile.py", line 1230, in __init__
    self._RealGetContents()
  File "/usr/local/lib/python3.8/zipfile.py", line 1353, in _RealGetContents
    x._decodeExtra()
  File "/usr/local/lib/python3.8/zipfile.py", line 480, in _decodeExtra
    self.file_size = counts[idx]
IndexError: tuple index out of range
msg343152 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-05-22 08:13
unzip index_tuple.zip -x
Archive:  index_tuple.zip

caution:  zipfile comment truncated
error [index_tuple.zip]:  missing 3992977728 bytes in zipfile
  (attempting to process anyway)
   skipping: zipfile_extract/        unsupported compression method 211


I think the issue is not with Python but with your ZIP file. Did you try to uncompress it with unzip?\

Thank you
msg344181 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2019-06-01 16:13
This report is valid. Serhiy has improved error reporting of the extra field in feccdb2a249a71be330765be77dee57121866779.

counts can indeed be an empty tuple:

    elif ln == 0:
        counts = ()

If I'm reading section 4.5.3 of https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT correctly, I think we need to raise BadZipFile if ln == 0.
msg344193 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-01 17:12
It is not enough. IndexError can be raised for ln == 8 or 16 when file_size, compress_size and header_offset are all set to 0xffffffff.
msg344196 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2019-06-01 18:07
@alter-bug-tracer, could you please create test files for the cases Serhiy has just mentioned?
msg345194 - (view) Author: alter-bug-tracer (alter-bug-tracer) * Date: 2019-06-11 06:33
@berker.peksag, first of all sorry for the late reply. 
We are not sure that we know how to do that. Our tests are generated automatically. What we can do is retest the lib with your temporary fixes in place, to see if they fix all the problems our software can detect. Would that help you?
msg347522 - (view) Author: Daniel Hillier (dhillier) * Date: 2019-07-09 06:29
I've pushed a PR which adds a test that generates corrupt zip64 files with different combinations of zip64 extra data lengths and zip64 flags (which determines how many fields are required in the extra data).

It now raises a BadZipFile with a message naming the first missing field.
msg355623 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-10-29 07:24
New changeset da6ce58dd5ac109485af45878fca6bfd265b43e9 by Serhiy Storchaka (Daniel Hillier) in branch 'master':
bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)
https://github.com/python/cpython/commit/da6ce58dd5ac109485af45878fca6bfd265b43e9
msg355625 - (view) Author: miss-islington (miss-islington) Date: 2019-10-29 07:43
New changeset f7d50f8f997fbfce1556991a3700826536871fe7 by Miss Skeleton (bot) in branch '3.7':
bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)
https://github.com/python/cpython/commit/f7d50f8f997fbfce1556991a3700826536871fe7
msg355626 - (view) Author: miss-islington (miss-islington) Date: 2019-10-29 07:44
New changeset 3801b2699eb9441ca31c6ec8fa956fc0fe755ef7 by Miss Skeleton (bot) in branch '3.8':
bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)
https://github.com/python/cpython/commit/3801b2699eb9441ca31c6ec8fa956fc0fe755ef7
msg355630 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-10-29 08:12
Thank you for your contribution Daniel.
History
Date User Action Args
2019-10-29 08:12:07serhiy.storchakasetstatus: open -> closed
versions: + Python 3.9
messages: + msg355630

resolution: fixed
stage: patch review -> resolved
2019-10-29 07:44:10miss-islingtonsetmessages: + msg355626
2019-10-29 07:43:40miss-islingtonsetnosy: + miss-islington
messages: + msg355625
2019-10-29 07:26:02miss-islingtonsetpull_requests: + pull_request16506
2019-10-29 07:25:54miss-islingtonsetpull_requests: + pull_request16505
2019-10-29 07:24:21serhiy.storchakasetmessages: + msg355623
2019-07-09 06:29:38dhilliersetnosy: + dhillier
messages: + msg347522
2019-07-09 04:35:07dhilliersetpull_requests: + pull_request14463
2019-06-11 06:33:06alter-bug-tracersetmessages: + msg345194
2019-06-01 18:07:35berker.peksagsetmessages: + msg344196
2019-06-01 17:12:04serhiy.storchakasetmessages: + msg344193
2019-06-01 16:13:45berker.peksagsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request13611
2019-06-01 16:13:14berker.peksagsetstatus: closed -> open

type: behavior
components: + Library (Lib)
versions: - Python 3.6
nosy: + berker.peksag, serhiy.storchaka

messages: + msg344181
resolution: not a bug -> (no value)
stage: resolved -> needs patch
2019-05-22 08:13:30matrixisesetstatus: open -> closed

nosy: + matrixise
messages: + msg343152

resolution: not a bug
stage: resolved
2019-05-21 12:32:49alter-bug-tracercreate