XML defines the following chars as whitespace [1]::

S ::= (#x20 | #x9 | #xD | #xA)+

However the chars are not properly escaped into attributes, so they are converted into spaces as per attribute-value normalization [2]

    >>> data = '\x09\x0a\x0d\x20'
    >>> data
    '\t\n\r '

    >>> import  xml.etree.ElementTree as ET
    >>> e = ET.Element('x', attr=data)
    >>> s = ET.tostring(e)
    >>> s
    '<x attr="\t&#10;\r " />'

    >>> e1 = ET.fromstring(s)
    >>> data1 = e1.attrib['attr']
    >>> data1 == data
    False

    >>> data1
    ' \n  '

cElementTree suffers of the same bug::

    >>> import  xml.etree.cElementTree as cET
    >>> cET.fromstring(cET.tostring(cET.Element('a', attr=data))).attrib['attr']
    ' \n  '

but not the external library lxml.etree::

    >>> import lxml.etree as LET
    >>> LET.fromstring(LET.tostring(LET.Element('a', attr=data))).attrib['attr']
    '\t\n\r '

The bug is analogous to bpo-5752 but it refers to a different and independent module. Proper escaping should be added to the _escape_attrib() function into /xml/etree/ElementTree.py (and equivalent for cElementTree).

[1] http://www.w3.org/TR/REC-xml/#white
[2] http://www.w3.org/TR/REC-xml/#AVNormalize

Proper escaping should be added to the _escape_attrib() function into /xml/etree/ElementTree.py (and equivalent for cElementTree).

Agreed. Can you provide a patch?

No, I cannot. I take the fact there has been no answer for more than 18 months as an acknowledgement that the issue is not deemed important by Python maintainers: it's not important for me either. I'm not a heavy xml user: just knowing that the Python XML libraries are unreliable and that by default I should use lxml is a sufficient solution to my sporadic xml uses. Your mileage should vary.

Here is a patch. Please note that in your example \r is replaced by \n per 2.11: http://www.w3.org/TR/REC-xml/#sec-line-ends
Also, the patch is only for ElementTree, I will investigate cElementTree but no promises.

I sort of realized, does this mean lxml.etree would now be the offender, for not following 2.11 and leaving the \r as-is?

The patch seems reason, though it needs a test.

Here is a patch with a unit test for the new escaping functionality. I believe it covers all the new cases. Additional code is not required for cElementTree as the serialisation code is all Python.

Stefan, can you opine on the patches and whether they should be backported?

Patch and test look correct. They fix a bug that produces incorrect output, so I vote for backporting them. Most code won't see the difference as whitespace control characters are rare in attribute values. And code that uses them will benefit from correctness. Obviously, there might also be breakage in the rare case that code puts control characters into attribute values and expects them to disappear magically, but then it's the user code that is wrong here.

Only issue is that serialisation is slow already and this change slows it down a bit more. Every attribute value will now be searched 8 times instead of 5 times. I added a minor review comment that would normally reduce it to 7. timeit suggests to me that the overall overhead is still tiny, though, and thus acceptable:

$ python3.5 -m timeit -s "s = 'askhfalsdhfashldfsadf'" "'\n' in s"
10000000 loops, best of 3: 0.0383 usec per loop

$ python3.5 -m timeit -s "s = 'askhfalsdhfashldfsadf'" "s.replace('\n', 'y')"
10000000 loops, best of 3: 0.151 usec per loop

$ python3.5 -m timeit -s "s = 'askhfalsdhfashldfsadf'; rep=s.replace" "rep('\n', 'y')"
10000000 loops, best of 3: 0.12 usec per loop

Raymond, you might have meant me when assigning the ticket and not Stefan Krah, but since I'm actually not a core dev, I can't commit the patch myself.

See my last comment, though, I reviewed the patch and it should get committed.

New changeset 0a5596315cf0 by Raymond Hettinger in branch '3.5':
Issue bpo-17582: xml.etree.ElementTree nows preserves whitespaces in attributes
https://hg.python.org/cpython/rev/0a5596315cf0

Done.

I don't see newlines currently preserved in attributes:

   elem = ET.parse(StringIO('<test a="   \nab\n    "/>')).getroot()
   print(ET.tostring(elem))

Also see the later fix in bpo-39011, where the EOL normalisation in attribute text was removed again. This change was applied in Py3.9.