classification
Title: buffer overflow in Zipfile when wrinting more than 2gig file
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: zipfile writes incorrect local file header for large files in zip64
View: 9720
Assigned To: Nosy List: Paul, amaury.forgeotdarc, enlavin, lambacck, nadeem.vawda, segfault42
Priority: normal Keywords:

Created on 2009-07-07 17:10 by segfault42, last changed 2011-11-03 12:17 by nadeem.vawda. This issue is now closed.

Files
File name Uploaded Description Edit
zipfile.py segfault42, 2009-07-07 17:10 zipfile.py lib
Messages (11)
msg90242 - (view) Author: (segfault42) Date: 2009-07-07 17:10
Hello, 

I have a problem with the librairy zipfile.py
http://svn.python.org/view/python/trunk/Lib/zipfile.py?revision=73565&view=markup


Zinfo structure limit the size of a file to an int max value with the
ZIP64_LIMIT value ( equal to "(1 << 31) - 1 " so to 2147483647 . 

The problem is happening when you write a big file in the line 1095 : 

self.fp.write(struct.pack("<lLL", zinfo.CRC, zinfo.compress_size,
                 zinfo.file_size))

zinfo.file_size is limited to a int  size and if you have a file bigger
than ZIP64_LIMIT you make a buffer overflow even if you set the flag
allowZip64 to true.
msg90243 - (view) Author: (segfault42) Date: 2009-07-07 17:14
look like issue 1182788
msg90261 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-07-08 09:00
I don't see how it can be a buffer overflow. Or is it an exception
raised by the struct.pack function?
msg90282 - (view) Author: (segfault42) Date: 2009-07-08 21:01
yes it's zinfo.file_size which is bigger than the long specify in the 
struct.pack

There's must have a solution with the extra header because a lot of tools 
can zip big file and these zip file can be open by zipfile.py

it's easy to reproduice with a big file of 3 gig.

i think that the problem come from that the write methode do not take 
care of the flag allowZip64
msg92142 - (view) Author: (segfault42) Date: 2009-09-01 18:58
still no one to help on this problem ? is someone has some idea ?
msg92147 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-09-01 23:44
I did reproduce the problem, but I'm sorry I don't have the time to fix 
it. However, I will review any proposed patch.
msg121826 - (view) Author: Chris Lambacher (lambacck) * Date: 2010-11-21 01:20
This should be closed as a dup of #1182788 which the OP identified as being the same bug and which is now fixed due to the implementation. of ZIP64.
msg146505 - (view) Author: Paul (Paul) Date: 2011-10-27 16:47
This is a problem with python2.7 as well.  A change in struct between python2.6 and 2.7 raises an exception on overflow instead of silently allowing it.  This prevents zipping any file larger than 4.5G.  This exception concurs when writing the 32-bit headers (which are not used on large files anyway)

The patch should be simple.  Just wrap line 1100: 
...struct.pack("<LLL",...
with a try: except: to revert to the old behavior.   Alternatively, check if size is bigger than ZIP64_LIMIT and set to anything less than ZIP64_LIMIT.
msg146527 - (view) Author: Paul (Paul) Date: 2011-10-27 22:26
I attempted to "re-allow overflow" in the struct(...) call by replacing `zinfo.file_size` with `ZIP64_LIMIT % zinfo.file_size` in zipfile.py, and successfully produced a compressed file from a 10G file, but the resulting compressed file could not be uncompressed and was deemed "invalid" by any unzip util I tried.
msg146911 - (view) Author: Miguel Hernández Martos (enlavin) Date: 2011-11-03 09:26
I think it's a dup of http://bugs.python.org/issue9720 

That issue has a patch that allows the generation of zip files with >2GB files.
msg146922 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2011-11-03 12:17
Marking as duplicate.
History
Date User Action Args
2011-11-03 12:17:17nadeem.vawdasetstatus: open -> closed
superseder: zipfile writes incorrect local file header for large files in zip64
messages: + msg146922

type: crash -> behavior
resolution: duplicate
stage: needs patch -> resolved
2011-11-03 09:26:36enlavinsetnosy: + enlavin
messages: + msg146911
2011-10-27 22:26:37Paulsetmessages: + msg146527
2011-10-27 22:17:33pitrousetversions: + Python 3.2, Python 3.3, - Python 2.4, Python 3.0
2011-10-27 16:49:35ezio.melottisetnosy: + nadeem.vawda
2011-10-27 16:47:49Paulsetnosy: + Paul

messages: + msg146505
versions: + Python 2.7
2010-11-21 01:20:39lambaccksetnosy: + lambacck
messages: + msg121826
2009-09-01 23:44:13amaury.forgeotdarcsetmessages: + msg92147
stage: needs patch
2009-09-01 18:58:33segfault42setmessages: + msg92142
2009-07-08 21:01:38segfault42setmessages: + msg90282
2009-07-08 09:00:42amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg90261
2009-07-07 17:14:30segfault42settype: crash
messages: + msg90243
2009-07-07 17:10:57segfault42create