The stream encoder for the zlib_codec doesn't use the incremental encoder, so it has limited usefulness in practice. This is easiest to show with an example.

Here is the behavior with the stream encoder:

>>> filelike = io.BytesIO()
>>> wrapped = codecs.getwriter('zlib_codec')(filelike)
>>> wrapped.write(b'hello')
>>> filelike.getvalue()
b'x\x9c\xab\x00\x00\x00y\x00y'
>>> wrapped.write(b'x')
>>> filelike.getvalue()
b'x\x9c\xab\x00\x00\x00y\x00yx\x9c\xab\x00\x00\x00y\x00y'
>>>

However, this is the behavior of the incremental encoder:

>>> ienc = codecs.getincrementalencoder('zlib_codec')()
>>> ienc.encode(b'x')
b'x\x9c'
>>> ienc.encode(b'x', final=True)
b'\xab\xa8\x00\x00\x01j\x00\xf1'
>>>

The stream encoder is apparently encoding each write as an individual block, but the incremental encoder buffers until it gets a block that's large enough to be meaningfully compressed.

Fixing this may require addressing a separate issue with stream encoders. Unlike with the GzipFile module, closing a stream encoder closes the underlying file. If this underlying file is a BytesIO, then closing makes it free its buffer, making it impossible to get at the completed file.

Andrew, could you possibly write a patch and a test for 3.3?

It looks like encodings/zlib_codec.py defines a custom IncrementalEncoder and IncrementalDecoder, but its StreamWriter and StreamReader rely on the standard implementation of codecs.StreamWriter and codecs.StreamReader.

One solution might be to have zlib_codec.StreamWriter inherit from zlib_codec.IncrementalEncoder instead of from zlib_encoder.Codec. I'm not familiar enough with the codecs library to know whether this is the best approach.

Unfortunately, there are 120 codec files in the encodings directory, and it's unclear how many of them would need to be modified. Based on the number of them that implement StreamWriter as "class StreamWriter(Codec,codecs.StreamWriter)", it looks like it might be a lot of them. Was each of these 120 files hand-written?

See also issue bpo-7475.

The stream part of the codecs isn't used that much in Python 3 any more, so I'm not sure if this is worth fixing.

The corresponding stream reader has a related issue which I mentioned in bpo-20132

Even if all these issues aren’t worth fixing, I think the documentation should at least say which codecs work fully (e.g. most text encodings), which ones work suboptimally (e.g. UTF-7), and which ones only work for single-shot encoding and decoding.

See bpo-23231 for a proposal which should make the incremental codec API compatible with a generic StreamReader/Writer class.

I discovered that many of the codec files are generated by gencodec.py, not hand-written. However when I tried regenerating them, I found a handful had diverged from the generated code, but others were more or less true to the generated code.

Here is a patch to implement the zlib-codec and bz2-codec StreamWriter classes based on their IncrementalEncoder classes. It depends on my patch for bpo-23231, though I guess it could be tweaked to work around that if desired.

New patch that also fixes StreamWriter.writelines() in general for the byte codecs

On 15.01.2015 05:43, Martin Panter wrote:

New patch that also fixes StreamWriter.writelines() in general for the byte codecs

Could you explain this new undocumented class ?

+class _IncrementalBasedWriter(StreamWriter):
+ """Generic StreamWriter implementation.
+
+ The _EncoderClass attribute must be set to an IncrementalEncoder
+ class to use.
+ """
+
+ def __init__(self, stream, errors='strict'):
+ super().__init__(stream, errors)
+ self._encoder = self._Encoder(errors)
+
+ def write(self, object):
+ self.stream.write(self._encoder.encode(object))
+
+ def reset(self):
+ self.stream.write(self._encoder.encode(final=True))
+

Note that the doc-string mentions a non-existing attribute and there
are doc-string missing for the other methods.

The purpose appears to be a StreamWriter which works with
an IncrementalEncoder. A proper name would thus be
IncrementalStreamWriter which provides an .encode()
method which adapts the signature of the incremental encoder
to the one expected for StreamWriters and Codecs.

Sorry, I changed the name of the attribute and forgot to update the doc string. Its new name was _Encoder.

Your description was fairly accurate. I am adding patch v3, with an expanded the doc string. Hopefully that explains it a bit better. Since it is just implementing the documented StreamWriter API, I only added brief descriptions of the methods pointing back to the StreamWriter methods.

Here is patch v4. The stream writer is now automatically generated by default by the CodecInfo constructor if no custom stream writer parameter is supplied. The base64 and quopri codecs are adjusted to also use this default stream writer to help with bpo-20132.