classification
Title: Reset method of the incremental encoders of CJK codecs calls the decoder reset function
Type: Stage: resolved
Components: Library (Lib), Unicode Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, arigo, doerwalter, gz, hyeshik.chang, lemburg, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2011-05-24 21:30 by vstinner, last changed 2011-05-30 20:57 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
cjk_reset_result.patch vstinner, 2011-05-24 21:30 review
cjk_encreset.patch vstinner, 2011-05-25 22:31 review
Messages (10)
msg136794 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-24 21:30
HZ and ISO-2022 family codecs may generate an escape sequence at the end of a stream. For example, the HZ codec uses '~{' to switchs from ASCII to GB2312, and '~}' resets the encoding to ASCII. At the end of a stream, the encoding should be reset to ASCII. '\u804a'.encode('hz') returns b'~{AD~}', which is correct.

Incremental encoders generate also the escape sequence if the last call to encode() is done using final=True.

It would be nice to be able to generate the escape sequence without the final flag, because sometimes you don't know which call to encode() is the last one. For example if you write data in a file, you may want to write the escape sequence at the end when the file is closed.

I propose to change the reset() method of incremental encoders: they may return a bytes object to close the stream. For example, the reset() method of the HZ codec may returns b'~}' if the encoder was using GB2312 (if it emited previously b'~{').

So the 3 following code should returns b'~{AD~}':

 * '\u804a'.encode('hz')
 * encoder = codecs.lookup('hz').incrementalencoder(); encoder.encode('\u804a', final=True)
 * encoder = codecs.lookup('hz').incrementalencoder(); encoder.encode('\u804a') + encoder.reset()

For backward compatibility, reset() returns None if there is no pending buffer or any escape sequence.

--

This proposition comes from #12000: Armin Rigo noticed that the reset method of the incremental encoders of CJK codecs calls the decoder reset function. Extract of Modules/cjkcodecs/multibytecodec.c:

static PyObject *
mbiencoder_reset(MultibyteIncrementalEncoderObject *self)
{
    if (self->codec->decreset != NULL &&
        self->codec->decreset(&self->state, self->codec->config) != 0)
        return NULL;
    self->pendingsize = 0;

    Py_RETURN_NONE;
}

I suppose that codec->encreset() is not called here because we need an output buffer, and there is no such buffer. Or it's just a copy-paste failure.

--

I am not sure that it is really useful to emit b'~}' at the end of a HZ stream, but the change is simple and if you don't care of b'~}': just ignore the result of reset() (as everybody does today, so it doesn't hurt).

--

Only HZ and ISO-2022 encodings implement the reset method of their incremental encoder. For example, encodings of the UTF family don't need to emit bytes at reset.

For the maximum length of reset() output: HZ may generates 2 bytes (b'~}') and ISO-2022 may generates 4 bytes (b'\x0F' + b'\x1F(B').

--

See also the issue #12100.
msg136796 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-05-24 21:48
Do we need an additional method? It seems that this reset() could also be written encoder.encode('', final=True)
msg136831 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-05-25 08:23
Amaury Forgeot d'Arc wrote:
> 
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Do we need an additional method? It seems that this reset() could also be written encoder.encode('', final=True)

+1

I think that's a much more natural way to implement "finalize the
encoding output without adding any data to it".
msg136908 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-25 21:52
Le mercredi 25 mai 2011 à 08:23 +0000, Marc-Andre Lemburg a écrit :
> > Do we need an additional method? It seems that this reset() could
> > also be written encoder.encode('', final=True)
> 
> +1
> 
> I think that's a much more natural way to implement "finalize the
> encoding output without adding any data to it".

And so, reset() should discard the output? I can easily adapt my patch
to discard the output (but still call encreset() instead of decreset()).
msg136915 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-25 22:31
cjk_encreset.patch: Fix multibytecodec.MultibyteIncrementalEncoder.reset(), call encreset() instead of decreset(). Improve also incremental encoder tests: reset the encoder using .encode('', final=True) and check the output.

I also prefer to reuse the existing API instead of changing reset() API just for a corner case.
msg136918 - (view) Author: Martin (gz) * Date: 2011-05-25 22:48
New cjk_encreset.patch looks good to me.

As with issue 12100 this likely wasn't reported before because decoders are robust against escape sequence oddities.
msg136919 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-05-25 23:42
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Le mercredi 25 mai 2011 à 08:23 +0000, Marc-Andre Lemburg a écrit :
>>> Do we need an additional method? It seems that this reset() could
>>> also be written encoder.encode('', final=True)
>>
>> +1
>>
>> I think that's a much more natural way to implement "finalize the
>> encoding output without adding any data to it".
> 
> And so, reset() should discard the output? I can easily adapt my patch
> to discard the output (but still call encreset() instead of decreset()).

I'm not sure what you mean by "discard the output".

Calling .reset() should still add the closing sequence to the output
buffer, if needed.

The purpose of .reset() is flush all data and put the codec into a
clean state (comparable to the state you get when you start using
it).
msg136922 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-26 00:14
> I'm not sure what you mean by "discard the output".
> 
> Calling .reset() should still add the closing sequence to the output
> buffer, if needed.

Incremental encoders don't have any buffer.

Python 3.3a0 (default:3c7792ec4547+, May 26 2011, 00:24:51) 
>>> import codecs

>>> enc=codecs.lookup('hz').incrementalencoder()
>>> enc.encode('\u8000')
b'~{R+'
>>> enc.reset()

>>> enc.encode('\u8000')
b'~{R+'
>>> enc.encode('', final=True)
b'~}'

>>> import io
>>> buffer=io.BytesIO()
>>> enc=codecs.lookup('hz').streamwriter(buffer)
>>> enc.write('\u8000')
>>> buffer.getvalue()
b'~{R+'
>>> enc.reset()
>>> buffer.getvalue()
b'~{R+~}'

IncrementalEncoder.reset() may mention that the output is discarded:

   .. method:: reset()

      Reset the encoder to the initial state. The output is discarded: 
      use the ``encode('', final=True)`` method to reset the encoder 
      and to get the output.
msg136946 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2011-05-26 11:43
+1 on the documentation changes.
msg137326 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-05-30 20:57
New changeset 61aaec67b521 by Victor Stinner in branch 'default':
Close #12171: IncrementalEncoder.reset() of CJK codecs (multibytecodec) calls
http://hg.python.org/cpython/rev/61aaec67b521
History
Date User Action Args
2011-05-30 20:57:33python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg137326

resolution: fixed
stage: resolved
2011-05-26 11:43:09doerwaltersetnosy: + doerwalter
messages: + msg136946
2011-05-26 00:14:09vstinnersetmessages: + msg136922
2011-05-25 23:42:28lemburgsetmessages: + msg136919
2011-05-25 22:48:37gzsetmessages: + msg136918
2011-05-25 22:31:04vstinnersetfiles: + cjk_encreset.patch

messages: + msg136915
2011-05-25 21:52:43vstinnersetmessages: + msg136908
2011-05-25 08:23:15lemburgsetmessages: + msg136831
title: Reset method of the incremental encoders of CJK codecs calls the decoder reset function -> Reset method of the incremental encoders of CJK codecs calls the decoder reset function
2011-05-24 21:48:20amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg136796
2011-05-24 21:30:09vstinnercreate