Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distutils: ZIP files don't include directory entries #78919

Closed
serhiy-storchaka opened this issue Sep 19, 2018 · 9 comments
Closed

Distutils: ZIP files don't include directory entries #78919

serhiy-storchaka opened this issue Sep 19, 2018 · 9 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

BPO 34738
Nosy @merwok, @serhiy-storchaka, @dstufft, @miss-islington
PRs
  • bpo-34738: Add directory entries in ZIP files created by distutils. #9419
  • [3.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) #10941
  • [3.6] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) #10942
  • [2.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419). #10950
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-12-05.22:10:17.851>
created_at = <Date 2018-09-19.13:47:27.502>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = "Distutils: ZIP files don't include directory entries"
updated_at = <Date 2018-12-05.22:10:17.851>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2018-12-05.22:10:17.851>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-05.22:10:17.851>
closer = 'serhiy.storchaka'
components = ['Distutils']
creation = <Date 2018-09-19.13:47:27.502>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 34738
keywords = ['patch']
message_count = 9.0
messages = ['325761', '325776', '325777', '325778', '331140', '331150', '331156', '331162', '331179']
nosy_count = 5.0
nosy_names = ['eric.araujo', 'mrabarnett', 'serhiy.storchaka', 'dstufft', 'miss-islington']
pr_nums = ['9419', '10941', '10942', '10950']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34738'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']
    @serhiy-storchaka
    Copy link
    Member Author

    ZIP files created by distutils don't include entries for directories. For example, if the directory structure is

    dist/
    [file1](https://bugs.python.org/file1).py
    sub/
        [file2](https://bugs.python.org/file2).py

    the ZIP file will contain entries 'dist/file1.py' and 'dist/sub/file2.py', but not 'dist/' and 'dist/sub/'.

    This only when use the zipfile module. If it is not available for some reasons, the zip command is used. In that case directory entries are created. The are created also in TAR files.

    All other methods of packing a tree to a ZIP in the stdlib (shutil.make_archive(), zipfile CLI, zipapp, Tools/msi/make_zip.py) create directory entries too.

    zipimport doesn't support implicit namespace packages if directory entries are not added (bpo-14905).

    The proposed PR makes distutils adding directory entries in a ZIP file.

    @serhiy-storchaka serhiy-storchaka added 3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Sep 19, 2018
    @mrabarnett
    Copy link
    Mannequin

    mrabarnett mannequin commented Sep 19, 2018

    I don't see a problem with this. If the zip file has 'dist/file1.py' then you know to create a directory when unzipping. If you want to indicate that there's an empty directory 'foo', then put 'foo/' in the zip file.

    @serhiy-storchaka
    Copy link
    Member Author

    There are two problems:

    1. This is inconsistent with TAR files and the zip utility.

    2. This is not compatible with zipimport.

    @merwok
    Copy link
    Member

    merwok commented Sep 19, 2018

    IMO inconsistency with tarfile is not very relevant.

    About the zip utility: are you saying that the unix unzip command is not able to inspect or extract distutils sdists?

    About zipimport, the question is: are sdists meant to be directly importable (IMO no).

    That said, if it is a minor change with no downsides, I wouldn’t oppose it. I just wanted to be clear on the reasons / benefits.

    @serhiy-storchaka
    Copy link
    Member Author

    The inconsistency is not just with the tarfile module, but with other method of creating ZIP archives.

    The unix unzip command creates directories if needed, but I am not sure about other third-party utilities. Since all know ZIP creators creates entries for directories, I would not be surprised if there are ZIP readers that doesn't work without explicit entries for directories. Zipimport is an example of this.

    Matthew mentioned other case. Empty directories currently are added to the TAR archive, but are omitted in the ZIP archive (unless you use an external zip tool). TAR archives are common on Unix, ZIP archives are common on Windows, and we can get different source distributions for Unix and Windows. You need to add a fake empty file to be sure that the directory will be added in the distribution. Some old DOS archives needed this trick, but the ZIP format supports empty directories.

    The downside of adding entries for directories is that this increases the size of the archive. But the difference is pretty small.

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset 67a93b3 by Serhiy Storchaka in branch 'master':
    bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419)
    67a93b3

    @miss-islington
    Copy link
    Contributor

    New changeset 53bed18 by Miss Islington (bot) in branch '3.7':
    bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419)
    53bed18

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset e0c2046 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
    bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) (GH-10942)
    e0c2046

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset b2742ba by Serhiy Storchaka in branch '2.7':
    [2.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419). (GH-10950)
    b2742ba

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @merwok @serhiy-storchaka @miss-islington