classification
Title: zipfile: tuple IndexError on extract
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alter-bug-tracer, berker.peksag, dhillier, matrixise, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-05-21 12:32 by alter-bug-tracer, last changed 2019-07-09 06:29 by dhillier.

Files
File name Uploaded Description Edit
index_tuple.zip alter-bug-tracer, 2019-05-21 12:32
Pull Requests
URL Status Linked Edit
PR 13727 open berker.peksag, 2019-06-01 16:13
PR 14656 open dhillier, 2019-07-09 04:35
Messages (7)
msg343038 - (view) Author: alter-bug-tracer (alter-bug-tracer) * Date: 2019-05-21 12:32
The following code throws an IndexError when attempting to extract a malformed archive (attached):

import zipfile
import sys

zf = zipfile.ZipFile(sys.argv[1])
for info in zf.infolist():
  zf.extract(info.filename)

Result:
Traceback (most recent call last):
  File "code.py", line 4, in <module>
    zf = zipfile.ZipFile(sys.argv[1])
  File "/usr/local/lib/python3.8/zipfile.py", line 1230, in __init__
    self._RealGetContents()
  File "/usr/local/lib/python3.8/zipfile.py", line 1353, in _RealGetContents
    x._decodeExtra()
  File "/usr/local/lib/python3.8/zipfile.py", line 480, in _decodeExtra
    self.file_size = counts[idx]
IndexError: tuple index out of range
msg343152 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-05-22 08:13
unzip index_tuple.zip -x
Archive:  index_tuple.zip

caution:  zipfile comment truncated
error [index_tuple.zip]:  missing 3992977728 bytes in zipfile
  (attempting to process anyway)
   skipping: zipfile_extract/        unsupported compression method 211


I think the issue is not with Python but with your ZIP file. Did you try to uncompress it with unzip?\

Thank you
msg344181 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2019-06-01 16:13
This report is valid. Serhiy has improved error reporting of the extra field in feccdb2a249a71be330765be77dee57121866779.

counts can indeed be an empty tuple:

    elif ln == 0:
        counts = ()

If I'm reading section 4.5.3 of https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT correctly, I think we need to raise BadZipFile if ln == 0.
msg344193 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-06-01 17:12
It is not enough. IndexError can be raised for ln == 8 or 16 when file_size, compress_size and header_offset are all set to 0xffffffff.
msg344196 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2019-06-01 18:07
@alter-bug-tracer, could you please create test files for the cases Serhiy has just mentioned?
msg345194 - (view) Author: alter-bug-tracer (alter-bug-tracer) * Date: 2019-06-11 06:33
@berker.peksag, first of all sorry for the late reply. 
We are not sure that we know how to do that. Our tests are generated automatically. What we can do is retest the lib with your temporary fixes in place, to see if they fix all the problems our software can detect. Would that help you?
msg347522 - (view) Author: Daniel Hillier (dhillier) * Date: 2019-07-09 06:29
I've pushed a PR which adds a test that generates corrupt zip64 files with different combinations of zip64 extra data lengths and zip64 flags (which determines how many fields are required in the extra data).

It now raises a BadZipFile with a message naming the first missing field.
History
Date User Action Args
2019-07-09 06:29:38dhilliersetnosy: + dhillier
messages: + msg347522
2019-07-09 04:35:07dhilliersetpull_requests: + pull_request14463
2019-06-11 06:33:06alter-bug-tracersetmessages: + msg345194
2019-06-01 18:07:35berker.peksagsetmessages: + msg344196
2019-06-01 17:12:04serhiy.storchakasetmessages: + msg344193
2019-06-01 16:13:45berker.peksagsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request13611
2019-06-01 16:13:14berker.peksagsetstatus: closed -> open

type: behavior
components: + Library (Lib)
versions: - Python 3.6
nosy: + berker.peksag, serhiy.storchaka

messages: + msg344181
resolution: not a bug ->
stage: resolved -> needs patch
2019-05-22 08:13:30matrixisesetstatus: open -> closed

nosy: + matrixise
messages: + msg343152

resolution: not a bug
stage: resolved
2019-05-21 12:32:49alter-bug-tracercreate