Hello,

The bug occurs when writing an XML file using the UTF-16 encoding.
The problem is that the etree encodes every string to utf-16 by itself - meaning, inserting the 0xfffe BOM before every string (tag, text, attribute name, etc.), causing a badly formed utf=16 strings.

A possible solution, which was offered by a co-worker of mine, was to use a utf-16 writer (from codecs.getwriter('utf-16') to write the file.

Best,

BugoK.

Fredrik, could you take a look at this?

ET's standard serializer currently only supports ASCII-compatible encodings.  See e.g.

    http://effbot.python-hosting.com/ticket/47

The best workaround for ET 1.2 (Python 2.5) is probably to serialize as "utf-8" and transcode:

    out = unicode(ET.tostring(elem), "utf-8").encode(...)

This is a bug in two halves.

1. Not all characters in the file are UTF-16. The initial xml header
isn't, and the individual < > etc characters are not. This is just a
matter of extending the methodology to encode all characters and not
just the textual bits. There is no work-around except a five-minute hack
of the ElementTree.write() method.

2. Every write has a BOM, so corrupting the file in a manner analogous
to bug 555360. This is a result of using string.encode() and is a
well-known feature. It can be worked around by using UTF-16LE or
UTF-16BE which do not prepend a BOM, but then the file doesn't have any
BOM. A complete solution would be to rewrite ElementTree.write() to use
a different encoding methodology such as StreamWriter.

I have made the above hack and work-around for my own use, and I can
report that it produces perfect UTF-16.

Would you provide a patch?

Here is a patch of my quick hack, more for interest than any suggestion
it gets used. Although it does produce good output so long as you avoid
the BOM.

The full solution is beyond my (very weak) Python skills. The character
encoding is tied in with XML character substitution (& etc. and
hexadecimal representation of multibyte characters). I could disentangle
it, but I probably wouldn't produce optimal Python, or indeed anything
that wouldn't inspire mirth and/or incredulity.

NB. The workaround suggested by Fredrik Lundh doesn't solve our
particular problems, since the downsize to UTF-8 causes the multi-byte
characters to be represented in hex. Our software doesn't read those. (I
know that's our problem.)

Could you provide a test case, so we can check if the upgrade proposed on #6472 solves this issue?

@Richard: Could you provide a test case for this, or do you consider it beyond your Python capabilities allowing for your comments on msg75875?

I can't produce an automated test, for want of time, but here is a demonstrator.

Grab the example XHTML from http://docs.python.org/library/xml.etree.elementtree.html#elementtree-objects or use some tiny ASCII-encoded xml file. Save it as "file.xml" in the same folder as bug-test.py attached here.

Execute bug-test.xml

file.xml is read and then written in UTF-16. The output file is then read and dumped to stdout as a byte-stream.

1. To be correct UTF-16, the output should start with 255 254, which should never occur in the rest of the file.

2. The rest of the output (including the first line) should alternate zeros with ASCII character codes.

3. The file output.xml should be loadable in a UTF16-capable text editor (eg jEdit), be recognised as UTF-16 and be identical in terms of content to file.xml

> Execute bug-test.xml

I meant bug-test.py, of course

@Florent: is this something you could pick up, I think it's out of my league.

As an example, here is the first two lines of output when I use Python 2.6.3:
60 63 120 109 108 32 118 101 114 115 105 111 110 61 39 49 46 48 39 32 101 110 99 111 100 105 110 103 61 39 85 84 70 45 49 54 39 63 62 10
60 255 254 104 0 116 0 109 0 108 0 62 255 254 10

Note:
No 255 254 at the start of the file, but several within it.
No zeros interspersing the first line and the odd one missing thereafter.

Python 3.1 improves the situation, the file looks more like utf-16, except that the BOM ("\xff\xfe") is repeated all the time, probably on every internal call to file.write().

Here is a test script that should work on both 2.7 and 3.1.

from io import BytesIO
from xml.etree.ElementTree import ElementTree
content = "<?xml version='1.0' encoding='UTF-16'?><html></html>"
input = BytesIO(content.encode('utf-16'))
tree = ElementTree()
tree.parse(input)
# Write content
output = BytesIO()
tree.write(output, encoding="utf-16")
assert output.getvalue().decode('utf-16') == content