HZ and ISO-2022 family codecs may generate an escape sequence at the end of a stream. For example, the HZ codec uses '~{' to switchs from ASCII to GB2312, and '~}' resets the encoding to ASCII. At the end of a stream, the encoding should be reset to ASCII. '\u804a'.encode('hz') returns b'~{AD~}', which is correct.

Incremental encoders generate also the escape sequence if the last call to encode() is done using final=True.

It would be nice to be able to generate the escape sequence without the final flag, because sometimes you don't know which call to encode() is the last one. For example if you write data in a file, you may want to write the escape sequence at the end when the file is closed.

I propose to change the reset() method of incremental encoders: they may return a bytes object to close the stream. For example, the reset() method of the HZ codec may returns b'~}' if the encoder was using GB2312 (if it emited previously b'~{').

So the 3 following code should returns b'~{AD~}':

• '\u804a'.encode('hz')
• encoder = codecs.lookup('hz').incrementalencoder(); encoder.encode('\u804a', final=True)
• encoder = codecs.lookup('hz').incrementalencoder(); encoder.encode('\u804a') + encoder.reset()

For backward compatibility, reset() returns None if there is no pending buffer or any escape sequence.

--

This proposition comes from bpo-12000: Armin Rigo noticed that the reset method of the incremental encoders of CJK codecs calls the decoder reset function. Extract of Modules/cjkcodecs/multibytecodec.c:

static PyObject *
mbiencoder_reset(MultibyteIncrementalEncoderObject *self)
{
    if (self->codec->decreset != NULL &&
        self->codec->decreset(&self->state, self->codec->config) != 0)
        return NULL;
    self->pendingsize = 0;

    Py_RETURN_NONE;
}

I suppose that codec->encreset() is not called here because we need an output buffer, and there is no such buffer. Or it's just a copy-paste failure.

--

I am not sure that it is really useful to emit b'~}' at the end of a HZ stream, but the change is simple and if you don't care of b'~}': just ignore the result of reset() (as everybody does today, so it doesn't hurt).

--

Only HZ and ISO-2022 encodings implement the reset method of their incremental encoder. For example, encodings of the UTF family don't need to emit bytes at reset.

For the maximum length of reset() output: HZ may generates 2 bytes (b'~}') and ISO-2022 may generates 4 bytes (b'\x0F' + b'\x1F(B').

--

See also the issue bpo-12100.

Do we need an additional method? It seems that this reset() could also be written encoder.encode('', final=True)

Amaury Forgeot d'Arc wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

Do we need an additional method? It seems that this reset() could also be written encoder.encode('', final=True)

+1

I think that's a much more natural way to implement "finalize the
encoding output without adding any data to it".

Le mercredi 25 mai 2011 à 08:23 +0000, Marc-Andre Lemburg a écrit :

> Do we need an additional method? It seems that this reset() could
> also be written encoder.encode('', final=True)

+1

I think that's a much more natural way to implement "finalize the
encoding output without adding any data to it".

And so, reset() should discard the output? I can easily adapt my patch
to discard the output (but still call encreset() instead of decreset()).

cjk_encreset.patch: Fix multibytecodec.MultibyteIncrementalEncoder.reset(), call encreset() instead of decreset(). Improve also incremental encoder tests: reset the encoder using .encode('', final=True) and check the output.

I also prefer to reuse the existing API instead of changing reset() API just for a corner case.

New cjk_encreset.patch looks good to me.

As with bpo-12100 this likely wasn't reported before because decoders are robust against escape sequence oddities.

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Le mercredi 25 mai 2011 à 08:23 +0000, Marc-Andre Lemburg a écrit :
>>> Do we need an additional method? It seems that this reset() could
>>> also be written encoder.encode('', final=True)
>>
>> +1
>>
>> I think that's a much more natural way to implement "finalize the
>> encoding output without adding any data to it".
> 
> And so, reset() should discard the output? I can easily adapt my patch
> to discard the output (but still call encreset() instead of decreset()).

I'm not sure what you mean by "discard the output".

Calling .reset() should still add the closing sequence to the output
buffer, if needed.

The purpose of .reset() is flush all data and put the codec into a
clean state (comparable to the state you get when you start using
it).

I'm not sure what you mean by "discard the output".

Calling .reset() should still add the closing sequence to the output
buffer, if needed.

Incremental encoders don't have any buffer.

Python 3.3a0 (default:3c7792ec4547+, May 26 2011, 00:24:51) 
>>> import codecs

>>> enc=codecs.lookup('hz').incrementalencoder()
>>> enc.encode('\u8000')
b'~{R+'
>>> enc.reset()

>>> enc.encode('\u8000')
b'~{R+'
>>> enc.encode('', final=True)
b'~}'

>>> import io
>>> buffer=io.BytesIO()
>>> enc=codecs.lookup('hz').streamwriter(buffer)
>>> enc.write('\u8000')
>>> buffer.getvalue()
b'~{R+'
>>> enc.reset()
>>> buffer.getvalue()
b'~{R+~}'

IncrementalEncoder.reset() may mention that the output is discarded:

.. method:: reset()

  Reset the encoder to the initial state. The output is discarded: 
  use the ``encode('', final=True)`` method to reset the encoder 
  and to get the output.

+1 on the documentation changes.

New changeset 61aaec67b521 by Victor Stinner in branch 'default':
Close bpo-12171: IncrementalEncoder.reset() of CJK codecs (multibytecodec) calls
http://hg.python.org/cpython/rev/61aaec67b521