New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Within zipfile, use of zlib.crc32 raises OverflowError at argument-parsing time on large strings #67495
Comments
Reproduction steps: --- $ python2.7 -c "import zlib;zlib.crc32('a'*(1<<31))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
OverflowError: size does not fit in an int We ran into this bug in zlib.crc32 when using zipfile.writestr() with a very large string; as soon as zipfile tried to write the crc checksum, it raised this error. Python 3 does not appear to suffer from this bug. |
This bug prevents zipfile's writestr() from writing large data (longer than UINT_MAX) to a 64-bit zip file. The zlib.crc32 function which, as written, cannot accept input with a size larger than an unsigned int. https://hg.python.org/cpython/file/94ec4d8cf104/Modules/zlibmodule.c#l964 Python 3 has updated this to call the zlib crc32 function multiple times in this situation: https://hg.python.org/cpython/file/93888975606b/Modules/zlibmodule.c#l1210 so the fix exists, we just need to do this in 2.7. |
Unfortunately, fixing just zlib.crc32 isn't quite enough for our purposes. We still will see OverflowErrow in zipfile if compression is selected. Demonstration code: ############################################
import zipfile
## Possible workaround: monkey-patch crc32 from binascii?!
import binascii
zipfile.crc32 = binascii.crc32
content = 'a'*(1<<31)
filename = '/tmp/zip_test.zip'
zf = zipfile.ZipFile(filename, "w",
compression=zipfile.ZIP_DEFLATED,
allowZip64=True)
zf.writestr('big', content)
zf.close()
zf = zipfile.ZipFile(filename, "r", allowZip64=True)
print zf.open('big').read() == content
############################################# This will raise the following error under Python 2.7.6: #############################################
$ python zip_test.py
Traceback (most recent call last):
File "zip_test.py", line 13, in <module>
zf.writestr('big', content)
File "/usr/lib/python2.7/zipfile.py", line 1228, in writestr
bytes = co.compress(bytes) + co.flush()
OverflowError: size does not fit in an int
############################################# If we use compression=zipfile.ZIP_STORED, we don't see this error, but it kind of misses a major point of using zipfile. |
This appears to have been fixed by at least bpo-27130's https://hg.python.org/cpython/rev/2192edcfea02 recent commits. greg:cpython/build27$ ./python -c "import zlib;zlib.crc32('a'*(1<<31))" |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: