This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: TextIOWrapper.write writes utf BOM for every string
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.0
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: alexandre.vassalotti, erickt, gvanrossum
Priority: high Keywords:

Created on 2008-01-07 08:50 by erickt, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
io.py.patch erickt, 2008-01-07 08:50
Messages (3)
msg59438 - (view) Author: Erick Tryzelaar (erickt) Date: 2008-01-07 08:50
I was playing around with python 3's io functions, and I found that when trying to write to 
an encoded utf-16 file that TextIOWrapper.write re-writes the utf-16 bom for every string:

>>> f=open('foo', 'w', encoding='utf-16')
>>> print('1234', file=f)
>>> print('5678', file=f)
>>> open('foo', 'rb').read()
b'\xff\xfe1\x002\x003\x004\x00\xff\xfe\n\x00\xff\xfe5\x006\x007\x008\x00\xff\xfe\n\x00'
>>> open('foo', 'r', encoding='utf-16').read()
'1234\ufeff\n\ufeff5678\ufeff\n'
>>> 

With the attached patch, it appears to generate the correct file:

>>> f=open('foo', 'w', encoding='utf-16')
>>> print('1234', file=f)
>>> print('5678', file=f)
>>> open('foo', 'rb').read()
b'\xff\xfe1\x002\x003\x004\x00\n\x005\x006\x007\x008\x00\n\x00'
>>> open('foo', 'r', encoding='utf-16').read()
'1234\n5678\n'
>>>
msg59449 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-01-07 15:25
I was aware of this but tried to ignore it until it became important. :-(
msg59473 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-01-07 18:32
Committed fix in r59832. Thanks!

P.S. Guido, what this comment, in write(), is about?
# XXX What if we were just reading?
History
Date User Action Args
2022-04-11 14:56:29adminsetgithub: 46093
2008-01-07 18:32:38alexandre.vassalottisetstatus: open -> closed
2008-01-07 18:32:34alexandre.vassalottisetresolution: fixed
2008-01-07 18:32:21alexandre.vassalottisetmessages: + msg59473
2008-01-07 16:43:07georg.brandlsetpriority: normal -> high
2008-01-07 15:25:45gvanrossumsetpriority: normal
nosy: + gvanrossum
messages: + msg59449
2008-01-07 09:21:11georg.brandlsetassignee: alexandre.vassalotti
nosy: + alexandre.vassalotti
2008-01-07 08:50:33ericktcreate