classification
Title: LZMACompressor and LZMADecompressor raise exceptions if given empty strings twice
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Decorater, benfogle, nadeem.vawda, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-07-15 01:03 by benfogle, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
lzma.patch benfogle, 2016-07-15 01:03 review
lzma_2.patch benfogle, 2016-10-31 01:52 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (7)
msg270451 - (view) Author: Benjamin Fogle (benfogle) * Date: 2016-07-15 01:03
To reproduce:

>>> import lzma
>>> c = lzma.LZMACompressor()
>>> c.compress(b'')
b'\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F'
>>> c.compress(b'')
b''
>>> c.compress(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_lzma.LZMAError: Insufficient buffer space
>>> d = lzma.LZMADecompressor()
>>> d.decompress(b'')
b''
>>> d.decompress(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_lzma.LZMAError: Insufficient buffer space

This can occur anytime during compression/decompression when an empty string is passed twice in a row.

The problem is that that liblzma interprets a call to lzma_code() with avail_in == 0 as a buffer full condition. The second time in a row it encounters this, it returns LZMA_BUF_ERROR as per documentation. The attached patch prevents this condition from occurring.
msg270452 - (view) Author: Decorater (Decorater) * Date: 2016-07-15 01:08
Why you passing decompress without passing compress before decompressing it again?

Also I would expect that it would show the same compress result trice in a row on the 1st test. This must definately be a issue.
msg270455 - (view) Author: Benjamin Fogle (benfogle) * Date: 2016-07-15 01:58
The above code demonstrates two separate issues. One with the decompressor, and one with the compressor.

In the compressor example, the first output differs from the rest because it is a file header which is always emitted. That behavior is correct.
msg279737 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-30 18:06
Thank you for your report and patch Benjamin. Seems your patch fixes the problem in default case. But the compressor still fails with the raw format.

>>> import lzma
>>> FILTERS_RAW_4 = [{"id": lzma.FILTER_DELTA, "dist": 4},
...                  {"id": lzma.FILTER_X86, "start_offset": 0x40},
...                  {"id": lzma.FILTER_LZMA2, "preset": 4, "lc": 2}]
>>> c = lzma.LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
>>> c.compress(b'')
b''
>>> c.compress(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_lzma.LZMAError: Insufficient buffer space
msg279758 - (view) Author: Benjamin Fogle (benfogle) * Date: 2016-10-31 01:52
Ah, thank you. Good catch. I have reworked the patch to handle both cases.
msg279764 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-31 06:36
New changeset b06f15507978 by Serhiy Storchaka in branch '3.5':
Issue #27517: LZMA compressor and decompressor no longer raise exceptions if
https://hg.python.org/cpython/rev/b06f15507978

New changeset fb64c7a81010 by Serhiy Storchaka in branch '3.6':
Issue #27517: LZMA compressor and decompressor no longer raise exceptions if
https://hg.python.org/cpython/rev/fb64c7a81010

New changeset 98c078fca8e0 by Serhiy Storchaka in branch 'default':
Issue #27517: LZMA compressor and decompressor no longer raise exceptions if
https://hg.python.org/cpython/rev/98c078fca8e0
msg279765 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-31 06:36
Thank you for you contribution Benjamin.
History
Date User Action Args
2017-03-31 16:36:25dstufftsetpull_requests: + pull_request990
2016-10-31 06:36:55serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg279765

stage: patch review -> resolved
2016-10-31 06:36:09python-devsetnosy: + python-dev
messages: + msg279764
2016-10-31 01:52:23benfoglesetfiles: + lzma_2.patch

messages: + msg279758
2016-10-30 18:06:41serhiy.storchakasetmessages: + msg279737
2016-10-30 17:23:11serhiy.storchakasetassignee: serhiy.storchaka
2016-10-25 18:45:01serhiy.storchakasetversions: + Python 3.7
2016-07-15 03:21:52serhiy.storchakasetnosy: + nadeem.vawda, serhiy.storchaka
stage: patch review

versions: - Python 3.3, Python 3.4
2016-07-15 01:58:33benfoglesetmessages: + msg270455
2016-07-15 01:08:43Decoratersetnosy: + Decorater
messages: + msg270452
2016-07-15 01:03:30benfoglecreate