classification
Title: zipfile.ZipFile.close() doesn't respect allowZip64
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: bgilbert, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-06-25 05:56 by bgilbert, last changed 2014-09-23 20:31 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
zipfile64_additional_checks.patch serhiy.storchaka, 2014-09-05 12:34 review
Messages (6)
msg221525 - (view) Author: Benjamin Gilbert (bgilbert) Date: 2014-06-25 05:56
The ZipFile documentation says:

> If allowZip64 is True (the default) zipfile will create ZIP files that
> use the ZIP64 extensions when the zipfile is larger than 2 GiB. If it
> is false zipfile will raise an exception when the ZIP file would
> require ZIP64 extensions.

ZipFile.close() will write ZIP64 central directory records if e.g. a member's local file header starts at an offset > 2 GB, or if there are more than 65535 files in the archive.  It will do this even if allowZip64 is False, whereas the documentation implies that it should raise an exception in that case.
msg221543 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-06-25 12:16
Do you want to provide a patch?
msg222476 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-07-07 17:02
An offset is checked in write() and writestr() methods and an exception is raised if an offset is >= 2 GiB and ZIP64 extension is not allowed. I don't see a possibility for a bug.
msg222489 - (view) Author: Benjamin Gilbert (bgilbert) Date: 2014-07-07 18:19
Here are the cases where close() will generate a ZIP64 archive and an exception will never be raised:

1. There are more than 65535 files in the archive.

2. The start of the central directory is at > 2 GB.

3. The central directory size is > 2 GB.

#1 could be checked from write/writestr.  #2 could possibly be checked from write/writestr by looking at the file offset after writing the compressed data.  #3 cannot be checked until close, but I'm not sure if it can ever occur without triggering one of the other checks first.
msg226420 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-09-05 12:34
Thank you Benjamin for explanation.

Here is a patch which adds all three required checks in the close() method and adds a check for files count in write methods. Implementing other checks in write methods is too laborious.

As far as test_zipfile64 is disabled, for manual testing you need temporary comment out the "support.requires()" statement and run only selected tests:

./python -m test.regrtest -v -m OtherTests test_zipfile64
msg227372 - (view) Author: Roundup Robot (python-dev) Date: 2014-09-23 18:41
New changeset 8a010ca89094 by Serhiy Storchaka in branch '2.7':
Issue #21866: ZipFile.close() no longer writes ZIP64 central directory
https://hg.python.org/cpython/rev/8a010ca89094

New changeset 8f25d118ce38 by Serhiy Storchaka in branch '3.4':
Issue #21866: ZipFile.close() no longer writes ZIP64 central directory
https://hg.python.org/cpython/rev/8f25d118ce38

New changeset d361d2176121 by Serhiy Storchaka in branch 'default':
Issue #21866: ZipFile.close() no longer writes ZIP64 central directory
https://hg.python.org/cpython/rev/d361d2176121
History
Date User Action Args
2014-09-23 20:31:16serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2014-09-23 18:41:04python-devsetnosy: + python-dev
messages: + msg227372
2014-09-05 12:34:11serhiy.storchakasetfiles: + zipfile64_additional_checks.patch
keywords: + patch
messages: + msg226420

stage: needs patch -> patch review
2014-07-07 18:19:19bgilbertsetstatus: pending -> open

messages: + msg222489
2014-07-07 17:02:30serhiy.storchakasetstatus: open -> pending

messages: + msg222476
2014-06-25 12:16:35serhiy.storchakasetversions: - Python 3.1, Python 3.2, Python 3.3
nosy: + serhiy.storchaka

messages: + msg221543

assignee: serhiy.storchaka
stage: needs patch
2014-06-25 05:56:09bgilbertcreate