Encoding a (well-formed) Unicode string containing a non-BMP character, using the xmlcharrefreplace error handler, will produce two XML entities for surrogate codepoints instead of one entity for the actual character.

Here's a transcript (Python 2.7.3, x86_64):


  >>> b = '\xf0\x9f\x92\x9d'
  >>> u = b.decode('utf8')
  >>> u
  u'\U0001f49d'
  >>> u.encode('ascii', errors='xmlcharrefreplace')
  '��'
  >>> ( u'\U0001f49d' ).encode('ascii', errors='xmlcharrefreplace')
  '��'
  >>> list(u)
  [u'\ud83d', u'\udc9d']
  >>> u.encode('utf8', errors='xmlcharrefreplace')
  '\xf0\x9f\x92\x9d'

The utf8 bytestring is correctly decoded, and the print representation shows one single Unicode character. Encoding using xmlcharrefreplace produces two XML entities, which is wrong[1]: a single non-BMP character should be represented in XML as a single entity reference, in this case presumably '💝'.

As the last two lines show, I'm using a narrow build (so the unicode strings are represented internally in UTF-16, I guess). Converting the string back to utf8 does the right thing, and emits a single UTF8 sequence representing the supplementary-plane codepoint.

(FWIW, the backslashreplace error handler also emits a surrogate pair, but I don't know if there is a complete specification for what that handler does, so it's possible that it's not wrong.)

[1] http://www.w3.org/International/questions/qa-escapes#bytheway

Attached patch against 3.2 seems to fix the problem.

Note that there's similar code in  charmap_encoding_error, PyUnicode_EncodeCharmap, PyUnicode_TranslateCharmap, and PyUnicode_EncodeDecimal, however I'm not sure how to reach these paths.

Thanks to the PEP 393, this issue is already fixed in Python 3.3.

$ ./python 
Python 3.3.0rc1+ (default:ba2c1def3710+, Sep  3 2012, 23:20:25) 
[GCC 4.6.3 20120306 (Red Hat 4.6.3-2)] on linux
>>> ( u'\U0001f49d' ).encode('ascii', errors='xmlcharrefreplace')
b'💝'

I prefer a little different (simpler for me) form:

                for (p = collstart; p < collend;) {
                    Py_UCS4 ch = *p++;
                    if ((0xD800 <= ch && ch <= 0xDBFF) &&
                        (p < collend) &&
                        (0xDC00 <= *p && *p <= 0xDFFF)) {
                        ch = ((((ch & 0x03FF) << 10) |
                               ((Py_UCS4)*p++ & 0x03FF)) + 0x10000);
                    }
                    str += sprintf(str, "&#%d;", (int)ch);
                }

And please look at the loop above ("determine replacement size"). It should be corrected too. It will be simpler to use a buffer with static size (``char buffer[2+29+1+1];``) as in charmap encoder. Perhaps charmap encoder should be fixed too (and common code extracted to separate function).

I doubt about '\ud83d\udc9d' on wide build. Is it right to encode it as b'&#128157;' and not as b'&#55357;&#56477;'? This will be compatible with narrow build but will break compatibility with 3.3+. What is less evil?

> I doubt about '\ud83d\udc9d' on wide build. Is it right to encode it as 
> b'💝' and not as b'��'?

I don't think so.  IIRC surrogates are invalid in UTF-32, and certainly shouldn't be recombined.

> This will be compatible with narrow build but will break compatibility
> with 3.3+. What is less evil?

I think it's better to be compatible with 3.3+.  This is anyway a rather obscure corner case.

Do you want to propose a new patch?

> I think it's better to be compatible with 3.3+.  This is anyway a rather obscure corner case.

Well, we should not introduce new divergence between 3.2 wide build and 3.3.

> Do you want to propose a new patch?

I will do it.

Here is a patch which fixes xmlcharrefreplace error handling in other places.

Unfortunately multibyte asian encoders are broken yet. I'll open a separate issue for this.

Should we really invest time to fix bugs related to astral (non-BMP) characters with rare codecs and error handlers (CJK codecs, xmlcharrefreplace error handler)? Python 3.3 is released and has a much better support of astral characters (in many places). I don't know for CJK codecs: Python 3.3 still uses the legacy Unicode API for CJK codecs and so depend on the wchar_t type (which is 16 bits on Windows). I just fixed Python 3.4 to use the new Unicode API (PEP 393), which always support astral characters support and don't depend on the size of the wchar_t type.

I'm not against fixing Python 2.7, I'm just not interested.

I tend to agree with Victor: if you want to fix 2.7 go ahead, but if that's too much work it's OK with me to close this issue.

New changeset 719ee60fc5e2 by Serhiy Storchaka in branch '2.7':
Issue #15866: The xmlcharrefreplace error handler no more produces two XML
http://hg.python.org/cpython/rev/719ee60fc5e2