classification
Title: Add support for bzip2 compression to the zipfile module
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alanmcintyre, jcea, loewis, nadeem.vawda, pitrou, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2012-03-20 11:30 by serhiy.storchaka, last changed 2012-05-01 05:59 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
bzip2_in_zip.patch serhiy.storchaka, 2012-03-20 11:30
bzip2_in_zip_review.patch loewis, 2012-03-20 15:08 Regenerate patch with cpython base revision review
bzip2_in_zip_tests.patch serhiy.storchaka, 2012-03-21 09:11 review
bzip2_in_zip_2.patch serhiy.storchaka, 2012-03-22 16:55 review
bzip2_in_zip_3.patch serhiy.storchaka, 2012-03-23 16:55 review
Messages (17)
msg156394 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-20 11:30
ZIP File Format Specification (http://www.pkware.com/documents/casestudies/APPNOTE.TXT) supports bzip2 compression since at least 2003. Since bzip2 contained in Python standart library, it would be nice to add support for these method in zipfile. This will allow to process more foreign zip files and create more compact distributives.

The proposed patch adds new method ZIP_BZIP2, which is automatically detecting when unpacking and that can be used for packing.
msg156419 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-03-20 15:23
Can you please submit a contributor form?

http://python.org/psf/contrib/contrib-form/
http://python.org/psf/contrib/
msg156427 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-03-20 16:09
The patch looks good. Can you also provide a test case?
msg156436 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-20 17:03
I am working on this. Should I add tests to test_zipfile.py or create new test_zipfile_bzip2.py?

It would add a note that the bzip2 compression can understand not all programs (and do not understand the older versions of Python), but understands the Info-Unzip? My English is not enough for the documentation.
msg156445 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-03-20 18:37
Please add it to test_zipfile.

As for the documentation, I propose the wording

"bzip2 compression was added to the zip file format in 2001. However, even more recent tools (including older Python releases) may not support it, causing either refusal to process the zip file altogether, or faiilure to extract individual files."

I'm not a native speaker of English, either. Feel free to put things through Google translate; some native speaker will pick up the text and correct it.
msg156482 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-21 09:08
Thanks to the tests, I found the error. Since the bzip2 is block algorithm, decompressor need to eat a certain amount of data, so it began to return data. Now when reading small chunks turns out premature end of data. I'm working on a fix.
msg156594 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-22 16:55
All errors are fixed. All tests are passed. Unfortunately, the patch was more than expected. This is necessary for correct and effective work with large bzip2 buffers (for other codecs can also be a profit).
msg156643 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2012-03-23 10:05
[Adding Alan McIntyre, who is listed as zipfile's maintainer.]


I haven't yet had a chance to properly familiarize myself with the
zipfile module, but I did notice an issue in the changes to ZipExtFile's
read() method. The existing code uses the b"".join() idiom for linear-
time concatenation, but the patch replaces it with a version that does
"buf += data" after each read. CPython can (I think) do this efficiently,
but it can be much slower on other implementations.


Martin:
> As for the documentation, I propose the wording
>
> "bzip2 compression was added to the zip file format in 2001. However, even more recent tools (including older Python releases) may not support it, causing either refusal to process the zip file altogether, or faiilure to extract individual files."

How about this?

"The zip format specification has included support for bzip2 compression
since 2001. However, some tools (including older Python releases) do not
support it, and may either refuse to process the zip file altogether, or
fail to extract individual files."
msg156646 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-23 10:40
> The existing code uses the b"".join() idiom for linear-
> time concatenation, but the patch replaces it with a version that does
> "buf += data" after each read.

You made a mess. The existing code uses ``buf += data``, but I allowed myself 
to replace it with the ``b"".join()`` idiom. The bzip2 codec has to deal with 
large pieces of data, now this may be important. In read1 still used ``buf += 
data``, but not in loop, there is a concatenation of the only two pieces.

> "The zip format specification has included support for bzip2 compression

Thank you. Can you offer the variant with including both bzip2 and lzma 
(supported since 2006)? I put him in the upcoming patch that adds support for 
lzma compression  to the zipfile module.
msg156647 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2012-03-23 10:52
> You made a mess. The existing code uses ``buf += data``, but I allowed myself 
> to replace it with the ``b"".join()`` idiom. The bzip2 codec has to deal with 
> large pieces of data, now this may be important. In read1 still used ``buf += 
> data``, but not in loop, there is a concatenation of the only two pieces.

My mistake; I confused the bodies of read() and read1().


> Thank you. Can you offer the variant with including both bzip2 and lzma
> (supported since 2006)? I put him in the upcoming patch that adds support for
> lzma compression  to the zipfile module.

"The zip format specification has included support for bzip2 compression
since 2001, and for LZMA compression since 2006. However, some tools
(including older Python releases) do not support these compression
methods, and may either refuse to process the zip file altogether,
or fail to extract individual files."
msg156670 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-03-23 16:55
Fixed regeression in decompression.

Nadeem Vawda, we both were wrong. `buf += data` is noticeably faster `b''.join()` in CPython.
msg158564 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-04-17 18:19
What's the status of your contrib form?
msg158618 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-04-18 13:23
> `buf += data` is noticeably faster `b''.join()` in CPython.

Perhaps because your system's memory allocator is extremely good (or buf is always very small), but b''.join() is far more robust.
Another alternative is accumulating in a bytearray, since it uses overallocation for linear time appending.
msg158741 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-04-19 20:25
> What's the status of your contrib form?

Oops. I put this off for a detailed study and forgotten.

I will send the form, as only get to the printer and the scanner.
msg158742 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-04-19 20:25
> Perhaps because your system's memory allocator is extremely good (or buf is always very small), but b''.join() is far more robust.
> Another alternative is accumulating in a bytearray, since it uses overallocation for linear time appending.

I thought, that it was in special optimization, mentioned in the
python-dev, but could not find this in the code. Perhaps it had not been
implemented.

In this particular case, the bytes appending is performed only once (and
probably a lot of appending with b''). Exceptions are possible only in
pathological cases, for example when compressed data is much larger
uncompressed data. The current implementation uses `buf += data`, if
someone wants to change it, then it's not me.
msg159743 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-01 05:58
New changeset 028e8e0b03e8 by Martin v. Löwis in branch 'default':
Issue #14371: Support bzip2 in zipfile module.
http://hg.python.org/cpython/rev/028e8e0b03e8
msg159744 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-01 05:59
Thanks for the patch!
History
Date User Action Args
2013-12-21 21:00:23belopolskylinkissue20048 dependencies
2012-05-01 05:59:21loewissetstatus: open -> closed
resolution: fixed
messages: + msg159744
2012-05-01 05:58:54python-devsetnosy: + python-dev
messages: + msg159743
2012-04-19 20:25:46serhiy.storchakasetmessages: + msg158742
2012-04-19 20:25:21serhiy.storchakasetmessages: + msg158741
2012-04-18 13:23:47pitrousetnosy: + pitrou
messages: + msg158618
2012-04-17 18:19:13loewissetmessages: + msg158564
2012-04-17 17:56:40jceasetnosy: + jcea
2012-03-23 16:55:32serhiy.storchakasetfiles: + bzip2_in_zip_3.patch

messages: + msg156670
2012-03-23 10:52:30nadeem.vawdasetmessages: + msg156647
2012-03-23 10:40:09serhiy.storchakasetmessages: + msg156646
2012-03-23 10:05:30nadeem.vawdasetnosy: + alanmcintyre
messages: + msg156643
2012-03-22 17:22:58pitrousetnosy: + nadeem.vawda

stage: patch review
2012-03-22 16:55:26serhiy.storchakasetfiles: + bzip2_in_zip_2.patch

messages: + msg156594
2012-03-21 09:11:34serhiy.storchakasetfiles: + bzip2_in_zip_tests.patch
2012-03-21 09:08:00serhiy.storchakasetmessages: + msg156482
2012-03-20 18:37:28loewissetmessages: + msg156445
2012-03-20 17:03:23serhiy.storchakasetmessages: + msg156436
2012-03-20 16:09:50loewissetmessages: + msg156427
2012-03-20 15:23:46loewissetnosy: + loewis
messages: + msg156419
2012-03-20 15:08:41loewissetfiles: + bzip2_in_zip_review.patch
2012-03-20 11:30:40serhiy.storchakacreate