Copy/paste of message79330 from the issue bpo-4862:
--------------

>>> f = open('utf16.txt', 'w', encoding='utf-16')
>>> f.write('abc')
3
>>> f.close()

>>> f = open('utf16.txt', 'a', encoding='utf-16')
>>> f.write('def')
3
>>> f.close()
>>> open('utf16.txt', 'r', encoding='utf-16').read()
'abc\ufeffdef'

Bug is reproductible with:

• Python 2.5 : charset utf-8-sig and utf-16 for codecs.open()
• trunk : charset utf-8-sig, utf-16 and utf-32 for codecs.open()
• py3k : charset utf-8-sig, utf-16 and utf-32 for open()

With utf-7 or utf-8, no BOM is written.

Note: with UTF-32, the BOM is 4 bytes long (0xff 0xfe 0x00 0x00 on
little endian) but it's still the character (BOM) \ufeff (little
endian).

See also issues bpo-5008 (f.tell()) and bpo-5016 (f.seekable()), not
directly related to this issue.

One possible solution would be to call tell() in the TextIOWrapper
constructor, and then decoder.setstate((b"", 0)) if the current pos is >0.

@pitrou: You're right, but the state have to be changed for the
encoder, not the decoder. I added the following code to TextIOWrapper
constructor (for the C and the Python version of io library):

        if self._seekable and self.writable():
            position = self.buffer.tell()
            if position != 0:
                self._encoder = self._get_encoder()
                self._encoder.setstate(0)

As for the C version of the patch:

• I'm not sure why you make self->ok handling more complicated than it was
• encodefunc should not be forced to NULL, otherwise it will yield a big
  decrease in write() performance

• I'm not sure why you make self->ok handling more complicated than it was

tell() requires self->ok=1. I choosed to reset self->ok to zero on error, but
it's maybe useless.

• encodefunc should not be forced to NULL, otherwise it will yield a big
  decrease in write() performance

self>encodefunc value have to be changed because <encoder>.setstate() changes
the encoding function: UTF-16 => UTF-16-LE or UTF-16-BE.

I don't know how to get the new value of self>encodefunc. I will try to write
another patch.

Le jeudi 26 mars 2009 à 17:26 +0000, STINNER Victor a écrit :

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

> - I'm not sure why you make self->ok handling more complicated than it was

tell() requires self->ok=1. I choosed to reset self->ok to zero on error, but
it's maybe useless.

You are calling tell() on the underlying binary buffer, not on the
TextIOWrapper object. So self->ok should have no impact. Am I missing
something?

self>encodefunc value have to be changed because <encoder>.setstate() changes
the encoding function: UTF-16 => UTF-16-LE or UTF-16-BE.

I know, but then the logic should probably be changed and use an
additional struct member to signal that the start of file has been
skipped.

Faster patch!

• add fast encoder for UTF-32, UTF-32-LE and UTF-32-BE (copy/paste of
  utf16 functions)
• move utf-8 before utf-16-* because utf8 more popular than utf16 :-p
• don't set self->encodefunc=NULL (loose all the encoder
  optimisation), but only fix self->encodefunc for two special cases:
  utf16=>utf16le or utf16be and utf32=>utf32le or utf32be
• remove self->ok: it was may be usefull for an older version of my
  patch, but it's not more needed

Hum, it's a detail, but is it a good idea to keep the reference the int(0)? Or
would it be better to release it at exit?

It's better now, although I think it's not good to duplicate the
encoding switch logic. It would be better to have a separate flag
indicate whether it's the start of stream or not. I'm gonna produce a
new patch, unless you beat me to it.

Also, I'm adding Amaury to the nosy list so that he tells us whether he
thinks the approach is sound.

Here is a new patch catering to more cases (seek()) in addition to just
opening in append mode.

seek() has also the problem? It's really hard to encode UTF-16/32
correctly...

Updated patch against py3k.

If no-one objects, I'm going to commit this in a couple of days.

As I said on IRC a few days ago, I think the patch is ready to go.

Ok, committed in r72635.

Shouldn't this fix be back-ported to the 2.6 branch too?

I fixed bpo-6213 in 2.6 and 2.7, and so it's now possible to backport this fix to 2.6 => r83200