As discussed in issue 10521 and the sprawling "len(chr(i)) = 2?" thread [1] on python-dev, many functions in python library behave differently on narrow and wide builds.  While there are unavoidable differences such as the length of strings with non-BMP characters, many functions can work around these differences.  For example, the ord() function already produces integers over 0xFFFF when given a surrogate pair as a string of length two on a narrow build.  Other functions such as str.isalpha(), are not yet aware of surrogates.  See also issue9200.

A consensus is developing that non-BMP characters support on narrow builds is here to stay and that naive functions should be fixed.  Unfortunately, working with surrogates in python code is tricky because unicode C-API does not provide much support and existing examples of surrogate processing look like this:

-        while (u != uend && w != wend) {
-            if (0xD800 <= u[0] && u[0] <= 0xDBFF
-                && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
-            {
-                *w = (((u[0] & 0x3FF) << 10) | (u[1] & 0x3FF)) + 0x10000;
-                u += 2;
-            }
-            else {
-                *w = *u;
-                u++;
-            }
-            w++;
-        }

The attached patch introduces a Py_UNICODE_NEXT() macro that allows replacing the code above with two lines:

+        while (u != uend && w != wend)
+            *w++ = Py_UNICODE_NEXT(u, uend);

The patch also introduces a set of macros for manipulating the surrogates, but I have not started replacing more instances of verbose surrogate processing because I would like to first look for higher level abstractions such as Py_UNICODE_NEXT().  For example, there are many instances that can benefit from Py_UNICODE_PUT_NEXT(ptr, ch) macro that would put a UCS4 character ch into Py_UNICODE buffer pointed by ptr and advance ptr by 1 or 2 units as necessary.


[1] http://mail.python.org/pipermail/python-dev/2010-November/105908.html