zipfile complains about "Bad magic number for central directory" when I
give it files over 2GB. I believe the problem is that the offset for the
central directory should be read as an unsigned long rather than as a
signed long. Modifying structEndArchive from "<4s4H2lH" to "<4s4H2LH"
(note the capital L) should probably fix it. When the offset is >2^31
you get a negative offset and the code fails to find the central
directory. I'll appreciate it if someone more knowledgeable looks at the
problem and the suggested fix, Thanks, Alon

What Python version exactly are you using? This might have been fixed in
2.5.2, with r60117.

Hi,
I'm using 2.5.2 (r252:60911),
Thanks,
Alon

On Sun, 2008-08-10 at 17:51 +0000, Martin v. Löwis wrote:
> Martin v. Löwis <martin@v.loewis.de> added the comment:
> 
> What Python version exactly are you using? This might have been fixed in
> 2.5.2, with r60117.
> 
> ----------
> nosy: +loewis
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue3535>
> _______________________________________

Do you have a public URL for such a zip file?

Hi Antoine,
The problem happens for files between 2GB and 4GB. I can't really send
you a link to such a big file. To reproduce the problem, you can
generate one. I created (and attach) a tiny C program that helps
generate one. If you want to, you can run it, save its output to a file
and then add it to a zip file (it should compress around 12%). The
resulting zip file will fail to open from python using the zipfile
package because of the bug I mentioned. Please let me know whether this
is enough information to reproduce,
Thanks,
Alon

On Mon, 2008-08-11 at 17:30 +0000, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> Do you have a public URL for such a zip file?
> 
> ----------
> nosy: +pitrou
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue3535>
> _______________________________________

> The problem happens for files between 2GB and 4GB. I can't really send
> you a link to such a big file. To reproduce the problem, you can
> generate one.

The problem is that the "zip" command fails to create a zip file larger
than 2GB (I get "zip I/O error: Invalid argument"). And even if it
didn't fail the internal structure of the zip file might not be exactly
the same as with other compression tools. That's why I was asking you
for an existing file.

If I give you an ssh/sftp access somewhere, would you be able to upload
such a file?

Antoine,
I had a similar problem with zip version 2.32, but this is fixed in
version 3.0 (or on 64-bit architectures). Would you be able to give it a
try with the newer version (which can be obtained from info-zip.org)?
Unfortunately, my upload bandwidth will not allow me to upload such a
big file.
Thanks,
Alon

On Wed, 2008-08-13 at 22:11 +0000, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> > The problem happens for files between 2GB and 4GB. I can't really send
> > you a link to such a big file. To reproduce the problem, you can
> > generate one.
> 
> The problem is that the "zip" command fails to create a zip file larger
> than 2GB (I get "zip I/O error: Invalid argument"). And even if it
> didn't fail the internal structure of the zip file might not be exactly
> the same as with other compression tools. That's why I was asking you
> for an existing file.
> 
> If I give you an ssh/sftp access somewhere, would you be able to upload
> such a file?
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue3535>
> _______________________________________

Alon, can you try with the following patch? It seems to fix it here.

Alan, do you have an opinion on this?

Your patch seems like a better way to detect whether a file is written
as Zip64, and it seems to be able to properly handle extracting a >2GB
file from a >2GB archive, so I'd vote to include it.  

I tested it with r66233, using a file made from the output of large.c,
zipped with the built-in archiver on OS X 10.4.11.  All regression tests
pass, including test_zipfile64, on both Linux and OS X.

Alan, do you have commit access? Otherwise the patch needs approval from
another core developer.

No, I don't have commit access at the moment.

I also agree with the patch. This seems the correct way to detect the
Zip64 format.

Fixed in r66240, r66241. Thanks!