classification
Title: Embedded zipfile fields dependent on absolute position
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: alanmcintyre, python-dev, serhiy.storchaka, spoo, twouters
Priority: normal Keywords: patch

Created on 2016-02-05 08:52 by spoo, last changed 2017-05-06 12:11 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
zipfile_concatenated.patch serhiy.storchaka, 2016-05-13 19:14 review
Pull Requests
URL Status Linked Edit
PR 1484 merged serhiy.storchaka, 2017-05-06 11:13
PR 1485 merged serhiy.storchaka, 2017-05-06 11:54
PR 1486 merged serhiy.storchaka, 2017-05-06 11:57
Messages (14)
msg259642 - (view) Author: (spoo) Date: 2016-02-05 08:52
Example:

from zipfile import ZipFile
with open('a.zipp', 'wb') as base:
    base.write(b'old\n')
    with ZipFile(base, 'a') as myzip:
        myzip.write('eggs.txt')

If the embedded zip portion of the file is extracted (first four bytes deleted), some fields will be incorrect in the resultant file - commenting out line 3 produces a file that can serve as a comparison.  These differences cause issues opening with some zip library implementations.

My best guess is that this is related to this line: https://github.com/python/cpython/blob/master/Lib/zipfile.py#L1459
msg259644 - (view) Author: (spoo) Date: 2016-02-05 08:59
I'm not familiar with iOS development, but I'd hazard a guess that "some zip library implementations" means a (the official?) iOS zip library.
msg265484 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-13 19:14
Yes, this may be a bug. After appending all offsets are absolute file positions. Here is a patch that presumably fixes this issue. Unfortunately I can't write tests for this issue. ZipFile is too lenient and all written tests are passes with unpatched code. Please test the patch manually.
msg267095 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-06-03 12:27
spoo, can you confirm that the patch fixes your issue?
msg267107 - (view) Author: (spoo) Date: 2016-06-03 15:39
On 06/03/2016 09:27 PM, Serhiy Storchaka wrote:
> 
> Serhiy Storchaka added the comment:
> 
> spoo, can you confirm that the patch fixes your issue?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26293>
> _______________________________________
> 

Yes, sorry, and thank you for looking into this!  It may take me a few
days to be able to test it.
msg267598 - (view) Author: (spoo) Date: 2016-06-07 06:25
Thank you for looking into this!  I don't have my tablet to test with at
the moment, but testing locally with zipinfo there used to be a warning
about missing bytes and now there is none.  I will test with the tablet
in a week or two when I get it back.
msg278079 - (view) Author: (spoo) Date: 2016-10-04 18:59
I confirmed this fixes the issue loading zips on an iPad.
msg278262 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-07 19:25
New changeset 005704f5634f by Serhiy Storchaka in branch '3.5':
Issue #26293: Fixed writing ZIP files that starts not from the start of the
https://hg.python.org/cpython/rev/005704f5634f

New changeset b06e75ae1981 by Serhiy Storchaka in branch '3.6':
Issue #26293: Fixed writing ZIP files that starts not from the start of the
https://hg.python.org/cpython/rev/b06e75ae1981

New changeset 64a19fe3a16a by Serhiy Storchaka in branch 'default':
Issue #26293: Fixed writing ZIP files that starts not from the start of the
https://hg.python.org/cpython/rev/64a19fe3a16a
msg278264 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-07 20:13
New changeset 9a99a88301ef by Serhiy Storchaka in branch '2.7':
Issue #26293: Fixed writing ZIP files that starts not from the start of the
https://hg.python.org/cpython/rev/9a99a88301ef
msg278265 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-07 20:15
Thank you for your report and testing.
msg292912 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-03 16:10
Unfortunately we should revert this change. It breaks other cases. See issue29094.
msg293155 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-06 11:46
New changeset 3763ea865cee5bbabcce11cd577811135e0fc747 by Serhiy Storchaka in branch 'master':
Revert bpo-26293 for zipfile breakage. See also bpo-29094. (#1484)
https://github.com/python/cpython/commit/3763ea865cee5bbabcce11cd577811135e0fc747
msg293157 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-06 12:10
New changeset 70dc6a7a0b7f104d87512556fca242c2ca96a010 by Serhiy Storchaka in branch '3.6':
[3.6] Revert bpo-26293 for zipfile breakage. See also bpo-29094. (GH-1484). (#1485)
https://github.com/python/cpython/commit/70dc6a7a0b7f104d87512556fca242c2ca96a010
msg293159 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-06 12:11
New changeset c8faabce6ef318f3b425c6defd846e274d61e2ef by Serhiy Storchaka in branch '3.5':
[3.5] Revert bpo-26293 for zipfile breakage. See also bpo-29094. (GH-1484). (#1486)
https://github.com/python/cpython/commit/c8faabce6ef318f3b425c6defd846e274d61e2ef
History
Date User Action Args
2017-05-06 12:11:22serhiy.storchakasetmessages: + msg293159
2017-05-06 12:10:52serhiy.storchakasetmessages: + msg293157
2017-05-06 11:57:37serhiy.storchakasetpull_requests: + pull_request1588
2017-05-06 11:54:20serhiy.storchakasetpull_requests: + pull_request1586
2017-05-06 11:46:03serhiy.storchakasetmessages: + msg293155
2017-05-06 11:13:01serhiy.storchakasetpull_requests: + pull_request1584
2017-05-03 16:10:55serhiy.storchakasetmessages: + msg292912
2017-05-03 16:08:26serhiy.storchakasetpull_requests: - pull_request1050
2017-03-31 16:36:32dstufftsetpull_requests: + pull_request1050
2016-10-07 20:15:03serhiy.storchakasetstatus: open -> closed
versions: + Python 3.7
messages: + msg278265

resolution: fixed
stage: patch review -> resolved
2016-10-07 20:13:15python-devsetmessages: + msg278264
2016-10-07 19:25:25python-devsetnosy: + python-dev
messages: + msg278262
2016-10-04 18:59:54spoosetmessages: + msg278079
2016-06-07 06:25:15spoosetmessages: + msg267598
2016-06-03 15:39:42spoosetmessages: + msg267107
2016-06-03 12:27:05serhiy.storchakasetmessages: + msg267095
2016-05-13 19:14:12serhiy.storchakasetfiles: + zipfile_concatenated.patch
versions: + Python 2.7, Python 3.5
messages: + msg265484

keywords: + patch
stage: patch review
2016-02-05 09:43:39serhiy.storchakasetassignee: serhiy.storchaka

nosy: + twouters, alanmcintyre, serhiy.storchaka
2016-02-05 08:59:20spoosetmessages: + msg259644
2016-02-05 08:52:35spoocreate