classification
Title: Distutils: ZIP files don't include directory entries
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, miss-islington, mrabarnett, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2018-09-19 13:47 by serhiy.storchaka, last changed 2018-12-05 22:10 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9419 merged serhiy.storchaka, 2018-09-19 13:53
PR 10941 merged miss-islington, 2018-12-05 19:46
PR 10942 merged miss-islington, 2018-12-05 19:46
PR 10950 merged serhiy.storchaka, 2018-12-05 20:37
Messages (9)
msg325761 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-19 13:47
ZIP files created by distutils don't include entries for directories. For example, if the directory structure is

    dist/
        file1.py
        sub/
            file2.py

the ZIP file will contain entries 'dist/file1.py' and 'dist/sub/file2.py', but not 'dist/' and 'dist/sub/'.

This only when use the zipfile module. If it is not available for some reasons, the zip command is used. In that case directory entries are created. The are created also in TAR files.

All other methods of packing a tree to a ZIP in the stdlib (shutil.make_archive(), zipfile CLI, zipapp, Tools/msi/make_zip.py) create directory entries too.

zipimport doesn't support implicit namespace packages if directory entries are not added (issue14905).

The proposed PR makes distutils adding directory entries in a ZIP file.
msg325776 - (view) Author: Matthew Barnett (mrabarnett) * (Python triager) Date: 2018-09-19 17:39
I don't see a problem with this. If the zip file has 'dist/file1.py' then you know to create a directory when unzipping. If you want to indicate that there's an empty directory 'foo', then put 'foo/' in the zip file.
msg325777 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-19 17:57
There are two problems:

1. This is inconsistent with TAR files and the zip utility.

2. This is not compatible with zipimport.
msg325778 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2018-09-19 18:19
IMO inconsistency with tarfile is not very relevant.

About the zip utility: are you saying that the unix unzip command is not able to inspect or extract distutils sdists?

About zipimport, the question is: are sdists meant to be directly importable (IMO no).

That said, if it is a minor change with no downsides, I wouldn’t oppose it.  I just wanted to be clear on the reasons / benefits.
msg331140 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-05 19:05
The inconsistency is not just with the tarfile module, but with other method of creating ZIP archives.

The unix unzip command creates directories if needed, but I am not sure about other third-party utilities. Since all know ZIP creators creates entries for directories, I would not be surprised if there are ZIP readers that doesn't work without explicit entries for directories. Zipimport is an example of this.

Matthew mentioned other case. Empty directories currently are added to the TAR archive, but are omitted in the ZIP archive (unless you use an external zip tool). TAR archives are common on Unix, ZIP archives are common on Windows, and we can get different source distributions for Unix and Windows. You need to add a fake empty file to be sure that the directory will be added in the distribution. Some old DOS archives needed this trick, but the ZIP format supports empty directories.

The downside of adding entries for directories is that this increases the size of the archive. But the difference is pretty small.
msg331150 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-05 19:46
New changeset 67a93b3a0b3814e97ef9d077b21325fc8ce351b2 by Serhiy Storchaka in branch 'master':
bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419)
https://github.com/python/cpython/commit/67a93b3a0b3814e97ef9d077b21325fc8ce351b2
msg331156 - (view) Author: miss-islington (miss-islington) Date: 2018-12-05 20:29
New changeset 53bed18d93233b030bb5b2637daf1b5e87548ef1 by Miss Islington (bot) in branch '3.7':
bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419)
https://github.com/python/cpython/commit/53bed18d93233b030bb5b2637daf1b5e87548ef1
msg331162 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-05 20:44
New changeset e0c2046d82f6db45a7f3d3ee5f555c3e2487a74d by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419) (GH-10942)
https://github.com/python/cpython/commit/e0c2046d82f6db45a7f3d3ee5f555c3e2487a74d
msg331179 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-05 22:02
New changeset b2742ba5f9ce8a6108202e0645662f2b58da423b by Serhiy Storchaka in branch '2.7':
[2.7] bpo-34738: Add directory entries in ZIP files created by distutils. (GH-9419). (GH-10950)
https://github.com/python/cpython/commit/b2742ba5f9ce8a6108202e0645662f2b58da423b
History
Date User Action Args
2018-12-05 22:10:17serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-12-05 22:02:15serhiy.storchakasetmessages: + msg331179
2018-12-05 20:44:43serhiy.storchakasetmessages: + msg331162
2018-12-05 20:37:51serhiy.storchakasetpull_requests: + pull_request10200
2018-12-05 20:29:33miss-islingtonsetnosy: + miss-islington
messages: + msg331156
2018-12-05 19:46:47miss-islingtonsetpull_requests: + pull_request10194
2018-12-05 19:46:38miss-islingtonsetpull_requests: + pull_request10193
2018-12-05 19:46:28serhiy.storchakasetmessages: + msg331150
2018-12-05 19:05:45serhiy.storchakasetmessages: + msg331140
2018-09-19 18:19:43eric.araujosetmessages: + msg325778
2018-09-19 17:57:24serhiy.storchakasetmessages: + msg325777
2018-09-19 17:39:05mrabarnettsetnosy: + mrabarnett
messages: + msg325776
2018-09-19 13:53:23serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request8839
2018-09-19 13:47:27serhiy.storchakacreate