Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

shutil.make_archive makes invalid directory entries #69810

Closed
gumblex mannequin opened this issue Nov 14, 2015 · 15 comments
Closed

shutil.make_archive makes invalid directory entries #69810

gumblex mannequin opened this issue Nov 14, 2015 · 15 comments
Assignees
@serhiy-storchaka
Labels
easy release-blocker stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@gumblex
Copy link
Mannequin

gumblex mannequin commented Nov 14, 2015

BPO 25624
Nosy @larryhastings, @benjaminp, @tarekziade, @serhiy-storchaka
Files
  • storedirectory.patch
  • storedirectory_test.patch: zipfile patch with test
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-12-19.08:11:01.507>
created_at = <Date 2015-11-14.09:25:24.400>
labels = ['easy', 'type-bug', 'library', 'release-blocker']
title = 'shutil.make_archive makes invalid directory entries'
updated_at = <Date 2015-12-19.08:11:01.507>
user = 'https://bugs.python.org/gumblex'

    bugs.python.org fields:

    activity = <Date 2015-12-19.08:11:01.507>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-12-19.08:11:01.507>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-11-14.09:25:24.400>
creator = 'gumblex'
dependencies = []
files = ['41039', '41120']
hgrepos = []
issue_num = 25624
keywords = ['patch', 'easy']
message_count = 15.0
messages = ['254647', '254648', '254649', '254650', '254651', '255054', '255084', '255097', '255098', '255124', '255128', '255135', '255137', '256054', '256720']
nosy_count = 7.0
nosy_names = ['larry', 'benjamin.peterson', 'tarek', 'SilentGhost', 'python-dev', 'serhiy.storchaka', 'gumblex']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25624'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']
    @gumblex
    Copy link
    Mannequin Author

    gumblex mannequin commented Nov 14, 2015

    The _make_zipfile in shutil uses ZIP_DEFLATED compression by default, and the fix introduced by bpo-24982 adds directory entries. In zipfile.ZipFile.write, directories is added as 0 file_size, 0 compress_size, regardless of the compression method. Deflate will compress an empty string as \x03\x00, thus the directory entries become incorrect.

    The command line interface of zipfile is correct. Shutil can be fixed as zipfile.main. As a directory entry with compression methods other than ZIP_STORED is meaningless, zipfile.write and (maybe) zipfile.writestr should always write a ZIP_STORED header for directory entries to avoid the above problem occuring by programming mistakes.

    @gumblex gumblex mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Nov 14, 2015
    @SilentGhost
    Copy link
    Mannequin

    SilentGhost mannequin commented Nov 14, 2015

    Can you provide a test?

    @serhiy-storchaka
    Copy link
    Member

    Do you want to provide a patch?

    @serhiy-storchaka serhiy-storchaka self-assigned this Nov 14, 2015
    @gumblex
    Copy link
    Mannequin Author

    gumblex mannequin commented Nov 14, 2015 

    $ mkdir foo; touch foo/a.txt; python3 -c "import shutil; shutil.make_archive('foo', 'zip', base_dir='foo')"; unzip -t foo.zip
Archive:  foo.zip
    testing: foo/                    
  error:  invalid compressed data to inflate
    testing: foo/a.txt                OK
At least one error was detected in foo.zip.

    (This affects 2.7, 3.4+)

    @gumblex
    Copy link
    Mannequin Author

    gumblex mannequin commented Nov 14, 2015

    My patch for this.

    @serhiy-storchaka
    Copy link
    Member

    Could you add a test Dingyuan? You can take test_zipfile_vs_zip in test_shutil as an example.

    The patch changes both shutil and zipfile. Aren't only changes to zipfile needed?

    @gumblex
    Copy link
    Mannequin Author

    gumblex mannequin commented Nov 22, 2015

    Yes, patching zipfile is enough. I wrote a test using unzip -t to check the zip. ZipFile.testzip can't detect this kind of error because zlib.decompressobj(-15) will decode b'' to b'' without errors.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Nov 22, 2015

    New changeset e7d1df56f70b by Serhiy Storchaka in branch '3.4':
    Issue bpo-25624: ZipFile now always writes a ZIP_STORED header for directory
    https://hg.python.org/cpython/rev/e7d1df56f70b

    New changeset 0f8d426e0d0c by Serhiy Storchaka in branch '3.5':
    Issue bpo-25624: ZipFile now always writes a ZIP_STORED header for directory
    https://hg.python.org/cpython/rev/0f8d426e0d0c

    New changeset 09ed44192995 by Serhiy Storchaka in branch 'default':
    Issue bpo-25624: ZipFile now always writes a ZIP_STORED header for directory
    https://hg.python.org/cpython/rev/09ed44192995

    New changeset de5582c569ff by Serhiy Storchaka in branch '2.7':
    Issue bpo-25624: ZipFile now always writes a ZIP_STORED header for directory
    https://hg.python.org/cpython/rev/de5582c569ff

    @serhiy-storchaka
    Copy link
    Member

    Thank you for your contribution Dingyuan.

    Benjamin, I think this fix should be in 2.7.11. It was a regression in 2.7.11 that had made shutil.make_archive() unusable with the zip format.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Nov 23, 2015

    New changeset 43208b0f2535 by Benjamin Peterson in branch '2.7':
    Issue bpo-25624: ZipFile now always writes a ZIP_STORED header for directory entries. Patch by Dingyuan Wang.
    https://hg.python.org/cpython/rev/43208b0f2535

    @serhiy-storchaka
    Copy link
    Member

    Thank you Benjamin.

    @larryhastings
    Copy link
    Contributor

    This change didn't make it into 3.5.1. It will be released with 3.5.2. I've already updated the Misc/NEWS entry.

    @serhiy-storchaka
    Copy link
    Member

    Larry, I think this patch is enough important to go into 3.5.1 final. It fixes a regression in common operation.

    @serhiy-storchaka
    Copy link
    Member

    Larry, is this fix included in 3.5.1?

    @larryhastings
    Copy link
    Contributor

    Like I said--no.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @serhiy-storchaka serhiy-storchaka

    Labels
    easy release-blocker stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @serhiy-storchaka @larryhastings