This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: codecs: StreamWriter issues with stateful codecs after a seek or with append mode
Type: Stage: resolved
Components: Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords:

Created on 2011-07-07 11:14 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (3)
msg139969 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-07-07 11:14
The following code fails with an AssertionError('###\ufeffdef'):

import codecs
_open = codecs.open
#_open = open
filename = "test"
with _open(filename, 'w', encoding='utf_16') as f:
    f.write('abc')
    pos = f.tell()
with _open(filename, 'w', encoding='utf_16') as f:
    f.seek(pos)
    f.write('def')
    f.seek(0)
    f.write('###')
with _open(filename, 'r', encoding='utf_16') as f:
    content = f.read()
    assert content == '###def', ascii(content)

It is a bug in StreamWriter.seek(): it should update the encoder state to not write a new BOM. It has to be fixed in the StreamWriter class of each stateful codec, or a stateful StreamWriter class should be implemented in the codecs module.

Python supports the following stateful codecs:

 * cp932
 * cp949
 * cp950
 * euc_jis_2004
 * euc_jisx2003
 * euc_jp
 * euc_kr
 * gb18030
 * gbk
 * hz
 * iso2022_jp
 * iso2022_jp_1
 * iso2022_jp_2
 * iso2022_jp_2004
 * iso2022_jp_3
 * iso2022_jp_ext
 * iso2022_kr
 * shift_jis
 * shift_jis_2004
 * shift_jisx0213
 * utf_8_sig
 * utf_16
 * utf_32

This bug has already been fixed in TextIOWrapper: issue #5006.
msg139970 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-07-07 11:20
There is a similar bug for append mode:

import codecs
_open = codecs.open
#_open = open
filename = "test"
with _open(filename, 'w', encoding='utf_16') as f:
    f.write('abc')
with _open(filename, 'a', encoding='utf_16') as f:
    f.write('def')
with _open(filename, 'r', encoding='utf_16') as f:
    content = f.read()
    assert content == 'abcdef', ascii(content)

This bug has also been fixed by the issue #5006 in the io module.
msg297127 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 01:39
Sorry, I just lost track of this issue and so will just close it.
History
Date User Action Args
2022-04-11 14:57:19adminsetgithub: 56721
2017-06-28 01:39:22vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg297127

stage: resolved
2011-07-07 11:20:54vstinnersetmessages: + msg139970
title: codecs: StreamWriter issue with stateful codecs after a seek -> codecs: StreamWriter issues with stateful codecs after a seek or with append mode
2011-07-07 11:14:35vstinnercreate