classification
Title: ZIP does not support timestamps before 1980
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Dormouse759, petr.viktorin, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2018-07-11 14:35 by petr.viktorin, last changed 2018-08-31 14:44 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8270 merged Dormouse759, 2018-07-13 11:43
PR 8725 merged Dormouse759, 2018-08-10 11:41
Messages (16)
msg321460 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2018-07-11 14:35
The ZIP format cannot handle times before 1980.
Issue6090 provided a nice error message for trying to add such files.

I'm seeing a system for reproducible builds that sets mtime to 1970 (zero UNIX timestamp), resulting in files that Python can't package into (zip-based) wheels.

At least here on Fedora, the `zip` command-line utility silently bumps old timestamps to 1980-01-01. Of course, silently corrupting data would not be good default behavior for Python.
But in many cases timestamps don't matter. It would be nice to give ZipFile and ZipFile.write() a `strict_timestamps=True` keyword argument that could be turned off.
msg321464 - (view) Author: Marcel Plch (Dormouse759) * Date: 2018-07-11 14:40
I'm going to have a closer look at this and try to add the option.
msg321595 - (view) Author: Marcel Plch (Dormouse759) * Date: 2018-07-13 11:45
I have created a PR for this: https://github.com/python/cpython/pull/8270
msg321597 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-13 12:04
There are ZIP extensions which allow to save timestamps before 1980 and with better than 2-seconds resolution. It would be better to use this feature. But first we need to resolve issue17681.
msg321598 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2018-07-13 12:22
I'm not sure the extensions will solve this problem fully.
If an implementation that doesn't support these extensions, how does it handle them?

For example, if Python 3.8 implements the extensions, I use py3.8 to create a zipfile containing old timestamps, and then want to uncompress the file using Python 2.7, how do I avoid losing the timestamp information?
msg321600 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-13 12:37
You can't, because Python 2.7 doesn't support it. But you will be able to pack files in a ZIP archive and extract them without a loss on 3.8 and with a loss of a timestamp on 2.7. And you will be able to use a third-party utilities for extracting files without a loss.
msg321607 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2018-07-13 13:05
Which third-party utilities support these? As I said, AFAIK `zip` on my system does not.

I assume the loss of data is the reason we have an error now -- if that wasn't a concern, zipfile could just silently bump the timestamp to 1980.

When the extensions are implemented, `strict_timestamps=False` should *additionally* use the extension, but default should still be to raise the error (to prevent metadata loss when decompressing with older/simpler tools).
msg322949 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-02 13:04
New changeset a2fe1e52eb94c41d9ebce1ab284180d7b1faa2a4 by Victor Stinner (Marcel Plch) in branch 'master':
bpo-34097: Add support for zipping files older than 1980-01-01 (GH-8270)
https://github.com/python/cpython/commit/a2fe1e52eb94c41d9ebce1ab284180d7b1faa2a4
msg322950 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-02 13:05
Thank you Petr Viktorin for the bug report and thanks to Marcel Plch for the implementation of the new strict_timestamps keyword-only parameter!
msg322954 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-08-02 13:23
If add a new option, I prefer to add it to the ZipFile constructor, similarly to allowZip64. Initially allowZip64 was False by default, because not all third-party tools supported ZIP64 extension. Later the default was changed to True, but you still can force raising an error if the archive is not compatible with old tools. I think the same policy should be applied here. Add the ability of writing date before 1980, and add an option for raising an error for date before 1980.
msg322957 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-02 14:23
Serhiy:
>  If add a new option, I prefer to add it to the ZipFile constructor, similarly to allowZip64. 

Aha, that would make sense. I'm not sure that it's useful to control the parameter per added file, it's enough to control the parameter per ZIP archive.

Marcel: would you mind to try to move the strict_timestamps parameter  from ZipFile.write() to ZipFile constructor?
msg322960 - (view) Author: Marcel Plch (Dormouse759) * Date: 2018-08-02 14:31
It seems reasonable, I'll have a look at it.
msg323023 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-08-03 05:20
Tests are failed on some buildbots. See issue34325.
msg323095 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-03 23:09
> Tests are failed on some buildbots. See issue34325.

Marcel Plch already fixed it: commit 7b41dbad78c6b03ca2f98800a92a1977d3946643. The two buildbots are back to green.
msg324423 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-31 14:43
New changeset 77b112cd56a50232abcdbf28f9aba88dc5d33ad3 by Victor Stinner (Marcel Plch) in branch 'master':
bpo-34097: Polish API design (GH-8725)
https://github.com/python/cpython/commit/77b112cd56a50232abcdbf28f9aba88dc5d33ad3
msg324424 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-31 14:44
Thanks Petr Viktorin for reporting this issue and thanks Marcel Plch for the fix!
History
Date User Action Args
2018-08-31 14:44:46vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg324424

stage: patch review -> resolved
2018-08-31 14:43:38vstinnersetmessages: + msg324423
2018-08-10 11:41:39Dormouse759setstage: resolved -> patch review
pull_requests: + pull_request8209
2018-08-03 23:09:34vstinnersetmessages: + msg323095
2018-08-03 05:20:43serhiy.storchakasetmessages: + msg323023
2018-08-02 14:31:42Dormouse759setmessages: + msg322960
2018-08-02 14:23:24vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg322957
2018-08-02 13:23:00serhiy.storchakasetmessages: + msg322954
2018-08-02 13:05:41vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg322950

stage: patch review -> resolved
2018-08-02 13:04:56vstinnersetnosy: + vstinner
messages: + msg322949
2018-07-13 13:05:01petr.viktorinsetmessages: + msg321607
2018-07-13 12:37:48serhiy.storchakasetmessages: + msg321600
2018-07-13 12:22:14petr.viktorinsetmessages: + msg321598
2018-07-13 12:04:25serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg321597
2018-07-13 11:45:19Dormouse759setmessages: + msg321595
2018-07-13 11:43:45Dormouse759setkeywords: + patch
stage: patch review
pull_requests: + pull_request7805
2018-07-11 14:40:11Dormouse759setnosy: + Dormouse759
messages: + msg321464
2018-07-11 14:35:58petr.viktorincreate