classification
Title: zlib.error: Error -2 while flushing: inconsistent stream state
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Ellison Marks, Jeremy Heiner, martin.panter, serhiy.storchaka, takluyver, xiang.zhang
Priority: normal Keywords:

Created on 2017-04-07 14:21 by Jeremy Heiner, last changed 2017-04-16 09:05 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1041 merged serhiy.storchaka, 2017-04-08 06:34
PR 1092 merged serhiy.storchaka, 2017-04-12 13:11
Messages (6)
msg291275 - (view) Author: Jeremy Heiner (Jeremy Heiner) Date: 2017-04-07 14:21
I had some statements inside a `with` statement to write data to an entry in a
ZipFile. It worked great. I added a second `with` statement containing almost
exactly the same statements. That still worked great.

I refactored those common statements into a function and called that function
from the two `with` statements... and got an exception:

  zlib.error: Error -2 while flushing: inconsistent stream state

I can't figure out why it matters whether the writing happens in the `with` or
in the function called by the `with`, but here's a trimmed-down version of the
code that demonstrates the problem:

--------------------------------------------------------------------------------
#!/usr/bin/env python

import io, pprint, zipfile
from zipfile import ZIP_DEFLATED

def printLiteral( data, out ) :
        encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
        pprint.pprint( data, stream=encoder )

data = { 'not' : 'much', 'just' : 'some K \N{RIGHTWARDS WHITE ARROW} V pairs' }

with zipfile.ZipFile( 'zzz.zip', mode='w', compression=ZIP_DEFLATED ) as myzip :

    with myzip.open( 'this one works', 'w' ) as out :
        encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
        pprint.pprint( data, stream=encoder )

    with myzip.open( 'this one fails', 'w' ) as out :
        printLiteral( data, out )

        print( 'printed but entry still open' )
    print( 'entry has been closed but not file' )
print( 'zip file has been closed' )
--------------------------------------------------------------------------------

And here's the output on my Arch Linux 64bit with package `python 3.6.0-2`...
A co-worker sees the same behavior on MacOS 10.11.6 Python 3.6.1 :

--------------------------------------------------------------------------------
printed but entry still open
Traceback (most recent call last):
  File "zzz.py", line 21, in <module>
    print( 'printed but entry still open' )
  File "/usr/lib/python3.6/zipfile.py", line 995, in close
    buf = self._compressor.flush()
zlib.error: Error -2 while flushing: inconsistent stream state
--------------------------------------------------------------------------------

I tried debugging this in PyDev but got lost. Turning off the compression makes
the exception go away.
msg291309 - (view) Author: Ellison Marks (Ellison Marks) Date: 2017-04-08 00:26
At a guess, when encoder goes out of scope, it closes the underlying file object. Then, on exiting the with block, the zipfile tries to take some action with the closed file and errors out?
msg291310 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-04-08 01:22
It looks like the zip entry writer object may not expect its “close” method to be called multiple times. Other file objects tend to allow this with no ill effect. Adding Serhiy and Thomas who implemented the writer object in Issue 26039.

The first time it is called will be when the context manager (“with” statement) exits.

The second time will be when the TextIOWrapper is garbage-collected. This is documented behaviour <https://docs.python.org/3.6/library/io.html#io.IOBase.__del__>, but I think it is a design mistake of the IOBase hierarchy; see Issue 19829.

A reasonable workaround would be to call encoder.detach() after you have finished writing the zip entry.
msg291319 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-08 06:33
I agree, that close() should be an idempotent operation. Proposed patch fixes this.
msg291550 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-12 13:03
New changeset 4c0d9ea995da595e90e08813b89510de59907802 by Serhiy Storchaka in branch 'master':
bpo-30017: Allowed calling the close() method of the zip entry writer object (#1041)
https://github.com/python/cpython/commit/4c0d9ea995da595e90e08813b89510de59907802
msg291747 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-16 09:04
New changeset 8e5b52a8da07e781bda50ba0a7065b1058495a37 by Serhiy Storchaka in branch '3.6':
bpo-30017: Allowed calling the close() method of the zip entry writer object (#1041) (#1092)
https://github.com/python/cpython/commit/8e5b52a8da07e781bda50ba0a7065b1058495a37
History
Date User Action Args
2017-04-16 09:05:54serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-04-16 09:04:47serhiy.storchakasetmessages: + msg291747
2017-04-12 13:11:39serhiy.storchakasetpull_requests: + pull_request1234
2017-04-12 13:03:25serhiy.storchakasetmessages: + msg291550
2017-04-08 06:34:32serhiy.storchakasetpull_requests: + pull_request1194
2017-04-08 06:33:40serhiy.storchakasetversions: + Python 3.7
messages: + msg291319

assignee: serhiy.storchaka
components: + Library (Lib)
stage: needs patch -> patch review
2017-04-08 01:23:00martin.pantersetnosy: + serhiy.storchaka, martin.panter, takluyver

messages: + msg291310
stage: needs patch
2017-04-08 00:26:14Ellison Markssetnosy: + Ellison Marks
messages: + msg291309
2017-04-07 17:22:30xiang.zhangsetnosy: + xiang.zhang
2017-04-07 14:32:24Jeremy Heinersettitle: zlib -> zlib.error: Error -2 while flushing: inconsistent stream state
2017-04-07 14:21:06Jeremy Heinercreate