This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: open doesn't call IncrementalEncoder with final=True
Type: behavior Stage:
Components: IO Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: lemburg Nosy List: doerwalter, haney, lemburg, martin.panter, vstinner
Priority: normal Keywords:

Created on 2018-12-29 06:00 by haney, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
test.py haney, 2018-12-29 06:00
Messages (2)
msg332701 - (view) Author: David Haney (haney) * Date: 2018-12-29 06:00
The implementation of open relies on a codecs' IncrementalEncoder, however it never calls `encode` with final=True. This appears to violate the documentation for IncrementalEncoder.encode which states that the last call to encode _must_ set final=True.

The attached test case demonstrates this behavior. A codec "delayed" is implemented that holds the last encoded string until the next call to `encode`, at which point it returns the encoded string. When final=True, both the previous and current string are returned.

When `codecs.iterencode` is used to encode a sequence of strings, the encode function is called for each element in the sequence, with final=False. encode is then called a final time with an empty string and final=True.

When `open` is used to open a file stream for the encoding, each call to `write` calls `encode` with final=False, however it never calls `encode` with final=True, and it doesn't appear there's an API for forcing it to occur (for instance `flush` and `close` do not).
msg332740 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2018-12-30 00:06
FWIW this happens with the built-in IDNA codec, and Amaury gave a demonstration under <https://bugs.python.org/issue17404#msg184045>.

I don’t think the “TextIOWrapper.flush” method should use final=True, but “close” and “detach” probably should.
History
Date User Action Args
2022-04-11 14:59:09adminsetgithub: 79792
2019-01-04 22:12:34vstinnersetnosy: + vstinner
2018-12-30 00:06:01martin.pantersetnosy: + martin.panter
messages: + msg332740
2018-12-29 07:00:06serhiy.storchakasetassignee: lemburg

nosy: + lemburg, doerwalter
2018-12-29 06:00:15haneycreate