classification
Title: shutil.make_archive makes invalid directory entries
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: SilentGhost, benjamin.peterson, gumblex, larry, python-dev, serhiy.storchaka, tarek
Priority: release blocker Keywords: easy, patch

Created on 2015-11-14 09:25 by gumblex, last changed 2015-12-19 08:11 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
storedirectory.patch gumblex, 2015-11-14 10:52 review
storedirectory_test.patch gumblex, 2015-11-22 06:50 zipfile patch with test review
Messages (15)
msg254647 - (view) Author: Dingyuan Wang (gumblex) * Date: 2015-11-14 09:25
The _make_zipfile in shutil uses ZIP_DEFLATED compression by default, and the fix introduced by #24982 adds directory entries. In zipfile.ZipFile.write, directories is added as 0 file_size, 0 compress_size, regardless of the compression method. Deflate will compress an empty string as \x03\x00, thus the directory entries become incorrect.

The command line interface of zipfile is correct. Shutil can be fixed as zipfile.main. As a directory entry with compression methods other than ZIP_STORED is meaningless, zipfile.write and (maybe) zipfile.writestr should always write a ZIP_STORED header for directory entries to avoid the above problem occuring by programming mistakes.
msg254648 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2015-11-14 09:28
Can you provide a test?
msg254649 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-14 10:11
Do you want to provide a patch?
msg254650 - (view) Author: Dingyuan Wang (gumblex) * Date: 2015-11-14 10:13
$ mkdir foo; touch foo/a.txt; python3 -c "import shutil; shutil.make_archive('foo', 'zip', base_dir='foo')"; unzip -t foo.zip
Archive:  foo.zip
    testing: foo/                    
  error:  invalid compressed data to inflate
    testing: foo/a.txt                OK
At least one error was detected in foo.zip.

(This affects 2.7, 3.4+)
msg254651 - (view) Author: Dingyuan Wang (gumblex) * Date: 2015-11-14 10:52
My patch for this.
msg255054 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-21 12:19
Could you add a test Dingyuan? You can take test_zipfile_vs_zip in test_shutil as an example.

The patch changes both shutil and zipfile. Aren't only changes to zipfile needed?
msg255084 - (view) Author: Dingyuan Wang (gumblex) * Date: 2015-11-22 06:50
Yes, patching zipfile is enough. I wrote a test using `unzip -t` to check the zip. ZipFile.testzip can't detect this kind of error because zlib.decompressobj(-15) will decode b'' to b'' without errors.
msg255097 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-22 12:56
New changeset e7d1df56f70b by Serhiy Storchaka in branch '3.4':
Issue #25624: ZipFile now always writes a ZIP_STORED header for directory
https://hg.python.org/cpython/rev/e7d1df56f70b

New changeset 0f8d426e0d0c by Serhiy Storchaka in branch '3.5':
Issue #25624: ZipFile now always writes a ZIP_STORED header for directory
https://hg.python.org/cpython/rev/0f8d426e0d0c

New changeset 09ed44192995 by Serhiy Storchaka in branch 'default':
Issue #25624: ZipFile now always writes a ZIP_STORED header for directory
https://hg.python.org/cpython/rev/09ed44192995

New changeset de5582c569ff by Serhiy Storchaka in branch '2.7':
Issue #25624: ZipFile now always writes a ZIP_STORED header for directory
https://hg.python.org/cpython/rev/de5582c569ff
msg255098 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-22 13:01
Thank you for your contribution Dingyuan.

Benjamin, I think this fix should be in 2.7.11. It was a regression in 2.7.11 that had made shutil.make_archive() unusable with the zip format.
msg255124 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-23 03:05
New changeset 43208b0f2535 by Benjamin Peterson in branch '2.7':
Issue #25624: ZipFile now always writes a ZIP_STORED header for directory entries. Patch by Dingyuan Wang.
https://hg.python.org/cpython/rev/43208b0f2535
msg255128 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-23 06:29
Thank you Benjamin.
msg255135 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-11-23 07:28
This change didn't make it into 3.5.1.  It will be released with 3.5.2.  I've already updated the Misc/NEWS entry.
msg255137 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-23 08:02
Larry, I think this patch is enough important to go into 3.5.1 final. It fixes a regression in common operation.
msg256054 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-12-07 08:43
Larry, is this fix included in 3.5.1?
msg256720 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-12-19 07:59
Like I said--no.
History
Date User Action Args
2015-12-19 08:11:01serhiy.storchakasetstatus: open -> closed
2015-12-19 07:59:54larrysetmessages: + msg256720
2015-12-07 08:43:09serhiy.storchakasetmessages: + msg256054
2015-12-07 08:39:47serhiy.storchakasetstatus: closed -> open
2015-11-23 08:02:35serhiy.storchakasetmessages: + msg255137
2015-11-23 07:28:30larrysetmessages: + msg255135
2015-11-23 06:29:30serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg255128

stage: test needed -> resolved
2015-11-23 03:05:38python-devsetmessages: + msg255124
2015-11-22 13:01:06serhiy.storchakasetpriority: normal -> release blocker
nosy: + benjamin.peterson, larry
messages: + msg255098

2015-11-22 12:56:55python-devsetnosy: + python-dev
messages: + msg255097
2015-11-22 06:50:11gumblexsetfiles: + storedirectory_test.patch

messages: + msg255084
2015-11-21 12:19:58serhiy.storchakasetmessages: + msg255054
stage: needs patch -> test needed
2015-11-14 10:52:44gumblexsetfiles: + storedirectory.patch
keywords: + patch
messages: + msg254651
2015-11-14 10:13:26gumblexsetmessages: + msg254650
versions: + Python 2.7, Python 3.4
2015-11-14 10:11:22serhiy.storchakasetkeywords: + easy
assignee: serhiy.storchaka
messages: + msg254649

stage: needs patch
2015-11-14 09:28:24SilentGhostsetnosy: + SilentGhost, tarek, serhiy.storchaka

messages: + msg254648
versions: + Python 3.6
2015-11-14 09:25:24gumblexcreate