As discussed in issue 10521 and the sprawling "len(chr(i)) = 2?" thread [1] on python-dev, many functions in python library behave differently on narrow and wide builds.  While there are unavoidable differences such as the length of strings with non-BMP characters, many functions can work around these differences.  For example, the ord() function already produces integers over 0xFFFF when given a surrogate pair as a string of length two on a narrow build.  Other functions such as str.isalpha(), are not yet aware of surrogates.  See also issue9200.

A consensus is developing that non-BMP characters support on narrow builds is here to stay and that naive functions should be fixed.  Unfortunately, working with surrogates in python code is tricky because unicode C-API does not provide much support and existing examples of surrogate processing look like this:

-        while (u != uend && w != wend) {
-            if (0xD800 <= u[0] && u[0] <= 0xDBFF
-                && 0xDC00 <= u[1] && u[1] <= 0xDFFF)
-            {
-                *w = (((u[0] & 0x3FF) << 10) | (u[1] & 0x3FF)) + 0x10000;
-                u += 2;
-            }
-            else {
-                *w = *u;
-                u++;
-            }
-            w++;
-        }

The attached patch introduces a Py_UNICODE_NEXT() macro that allows replacing the code above with two lines:

+        while (u != uend && w != wend)
+            *w++ = Py_UNICODE_NEXT(u, uend);

The patch also introduces a set of macros for manipulating the surrogates, but I have not started replacing more instances of verbose surrogate processing because I would like to first look for higher level abstractions such as Py_UNICODE_NEXT().  For example, there are many instances that can benefit from Py_UNICODE_PUT_NEXT(ptr, ch) macro that would put a UCS4 character ch into Py_UNICODE buffer pointed by ptr and advance ptr by 1 or 2 units as necessary.


[1] http://mail.python.org/pipermail/python-dev/2010-November/105908.html

In addition to the proposed Py_UNICODE_NEXT and Py_UNICODE_PUT_NEXT,  str.__format__ would also need a function that tells it how many Py_UNICODEs are needed to store a given Py_UCS4.

On Fri, Nov 26, 2010 at 7:27 PM, Eric Smith <report@bugs.python.org> wrote:
..
>
> In addition to the proposed Py_UNICODE_NEXT and Py_UNICODE_PUT_NEXT, > str.__format__ would also need a function that tells it how many Py_UNICODEs
> are needed to store a given Py_UCS4.

Yes, this functionality is currently hidden in

unicode_aswidechar(PyUnicodeObject *unicode,
                   wchar_t *w,
                   Py_ssize_t size):

/* Helper function for PyUnicode_AsWideChar() and
PyUnicode_AsWideCharString():
   convert a Unicode object to a wide character string.

   - If w is NULL: return the number of wide characters (including the
nul
     character) required to convert the unicode object. Ignore size argument.
.. */

and I believe is reimplemented in a few other places.

I'd need access to this without having to build a PyUnicodeObject, for efficiency. But it sounds like it does have the basic functionality I need.

For my use I'd really need it to take the result of Py_UNICODE_NEXT. Something like:
Py_ssize_t
Py_UNICODE_NUM_NEEDED(Py_UCS4 c)
and it would always return 1 or 2. Always 1 for a wide build, and for a narrow build 1 if c is in the BMP else 2. Choose a better name, of course.

On Fri, Nov 26, 2010 at 7:45 PM, Eric Smith <report@bugs.python.org> wrote:
..
> For my use I'd really need it to take the result of Py_UNICODE_NEXT. Something like:
> Py_ssize_t
> Py_UNICODE_NUM_NEEDED(Py_UCS4 c)
> and it would always return 1 or 2. Always 1 for a wide build, and for a narrow
> build 1 if c is in the BMP else 2. Choose a better name, of course.

Can you describe your use case in more detail?  Would
Py_UNICODE_PUT_NEXT() combined with
Py_UNICODE_CODEPOINT_COUNT(Py_UNICODE *begin, Py_UNICODE *end) solve it?

I don't like macro having a result and using multiple instructions using the evil magic trick (the ","). It's harder to maintain the code and harder to debug than a classical function.

Don't you think that modern compilers are able to inline the code? (If not, we may add the right C attribute/keyword)

The code will basically be:

  Py_UCS4 fill;

  parse_format_string(fmt, ..., &fill, ...);

  /* lots more code */

  if (fill_needed) {
    /* compute how many characters to reserve */
    space_needed = Py_UNICODE_NUM_NEEDED(fill) *
                      number_of_characters_to_fill;
  }

It would be most convenient (and require the fewest changes) if the computation could just use fill, instead of remembering the pointers to the beginning and end of fill.

Py_UNICODE_CODEPOINT_COUNT could be implemented with a primitive that does what I want.

On Fri, Nov 26, 2010 at 8:41 PM, STINNER Victor <report@bugs.python.org> wrote:
..
> I don't like macro having a result and using multiple instructions using the evil
> magic trick (the ","). It's harder to maintain the code and harder to debug than
> a classical function.
>

You are preaching to the choir.  In fact, my first version
(issue10521-unicode-next.diff attached to  issue10521) used a
function.   I would not worry about implementation at this point,
though.  Let's find the best abstraction first.

> Don't you think that modern compilers are able to inline the code?
> (If not, we may add the right C attribute/keyword)

Not in C.  In C++, I could use a reference to the pointer incremented
by the macro, but in C, I have to use an address.  Once you take an
address of a variable, the compiler will refuse to put it in a
register.  So no, I don't think we can write an ANSI C function that
will be as efficient as the macro.

The compiler's decision to inline something should not be related to its ability to put variables in a register.

But I definitely agree that we should get the abstraction right first and worry about the implementation later.

On Fri, Nov 26, 2010 at 9:22 PM, Eric Smith <report@bugs.python.org> wrote:
..
> But I definitely agree that we should get the abstraction right first and worry about
> the implementation later.

I am fairly happy with Py_UNICODE_NEXT() abstraction.  It's semantics
should be natural for users familiar with python iterators and the
fact that it expands to simply *ptr++ on wide builds makes it easy to
explain its usage.   I am note very happy about the end argument for
the following reasons:

1. Builtin "next()" takes the default value as a second argument.
Extension writers may expect the same from Py_UNICODE_NEXT().  The
name "end" should be self-explainatory though, especially to those
with an exposure to STL.

2. If  Py_UNICODE_NEXT() stays as a macro, an innocent looking
Py_UNICODE_NEXT(p, p + size) will have a hard to detect bug.  Can be
fixed by making Py_UNICODE_NEXT() a function.

I wonder whether it is best to prefix the new macros with an
underscore.  On one hand, we want to make this available to extension
writers, on the other hand, once more people start dealing with
non-BMP issues, a better abstraction may be found and we man not want
to maintain  Py_UNICODE_NEXT()  indefinitely.

Raymond,

I wonder if you would like to comment on the iterator analogy and/or on adding public names to C API.

Mark, can you opine on this?

Raymond Hettinger wrote:
> 
> Raymond Hettinger <rhettinger@users.sourceforge.net> added the comment:
> 
> Mark, can you opine on this?

Yes, I'll have a look later today.

I like the idea and thanks for putting work into this.

Some comments:
 
 * when using macro variables, always put the variables in parens in the expansion; this avoids precedence issues, weird syntax errors, etc. - even if it may not be necessary
 
 * a function would be cleaner, but since this code is very performance sensitive, I'd opt for the macro version, unless someone can prove that a function would be just as fast in benchmarks
 
 * the macros should be documented in the unicodeobject.h header file and clearly mention that ptr and end should be side-effect free and that ptr must an lvalue
 
 * please use the faster bitmask operators for joining surrogates, i.e. ucs4 = ((((high & 0x03FF) << 10) | (low & 0x03FF)) + 0x00010000);

 * the Py_UNICODE_JOIN_SURROGATES() macro should use Py_UCS4 as prefix since it returns Py_UCS4 values, i.e. Py_UCS4_JOIN_SURROGATES()

 * same for the Py_UNICODE_NEXT() macro, i.e. Py_UCS4_NEXT()
 
 * in order to make the macro easier to understand, please rename it to Py_UCS4_READ_CODE_POINT(); that's a little more typing, but still a lot less than without the macro :-)

 * this version should be slightly faster and is also easier to read:
 
#define Py_UCS4_READ_CODE_POINT(ptr, end) \
    ((Py_UNICODE_ISHIGHSURROGATE((ptr)[0]) && \
      (ptr) < (end) && \
      Py_UNICODE_ISLOWSURROGATE((ptr)[1])) ? \
      Py_UNICODE_JOIN_SURROGATES((ptr)++, (ptr)++) : \
      (Py_UCS4)*(ptr)++)

   I haven't tested it, but you get the idea.
   
BTW: You only focus on UCS2 builds. Please also make sure that these changes work on UCS4 builds, e.g. Py_UCS2_READ_CODE_POINT() will also work on UCS4 builds and join code points there.

Note that UCS4 builds currently don't join surrogates, so a high and low surrogates appear as two code points, which they are, but given the experience with UCS2 builds, may not be what the user expects. So for the purpose of consistency we should be careful with auto-joining surrogates in UCS2.

It does make sence for ord() and the various string methods, but should be done with care in other cases.

In any case, we should clearly document where these macros are used and warn about the implications of using them in the wrong places.

On Sat, Nov 27, 2010 at 5:03 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
.. [I'll respond to skipped when I update the patch]

> In any case, we should clearly document where these macros are used and
> warn about the implications of using them in the wrong places.

It may be best to start with _Py_UCS2_READ_CODE_POINT() (BTW, I like
the name because it naturally lead to Py_UCS2_WRITE_CODE_POINT()
counterpart.)  The leading underscore will probably not stop early
adopters from using it and we may get some user feedback if they ask
to make these macros public.

On Sat, Nov 27, 2010 at 5:03 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
>  * same for the Py_UNICODE_NEXT() macro, i.e. Py_UCS4_NEXT()
>
>  * in order to make the macro easier to understand, please rename it to
>  Py_UCS4_READ_CODE_POINT(); that's a little more typing, but still
>  a lot less than without the macro :-)

I am not sure Py_UCS4_ prefix is right here.  (I agree on *SURROGATE*
methods.)  The point of Py_UNICODE_NEXT(ptr, end) is that the pointers
ptr and end are Py_UNICODE* and the macro expands to *p++ on wide
builds.  Maybe Py_UNICODE_NEXT_USC4?

Alexander Belopolsky wrote:
> 
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> On Sat, Nov 27, 2010 at 5:03 PM, Marc-Andre Lemburg
> <report@bugs.python.org> wrote:
> ..
>>  * same for the Py_UNICODE_NEXT() macro, i.e. Py_UCS4_NEXT()
>>
>>  * in order to make the macro easier to understand, please rename it to
>>  Py_UCS4_READ_CODE_POINT(); that's a little more typing, but still
>>  a lot less than without the macro :-)
> 
> I am not sure Py_UCS4_ prefix is right here.  (I agree on *SURROGATE*
> methods.)  The point of Py_UNICODE_NEXT(ptr, end) is that the pointers
> ptr and end are Py_UNICODE* and the macro expands to *p++ on wide
> builds.  Maybe Py_UNICODE_NEXT_USC4?

The idea is that the first part refers to what the macro returns (Py_UCS4)
and the "read" part of the name refers to moving a pointer across
an array (any array of integers).

Note that the macro can also work on Py_UCS4 arrays (even in
UCS2 builds), so it's universal in that respect.

Perhaps we should allow ord() to work on surrogates in
UCS4 builds as well. That would reduce the number of
surprises.

> The idea is that the first part refers to what the macro
> returns (Py_UCS4) and the "read" part of the name refers 
> to moving a pointer across an array (any array of integers).

I thought the first part generally meant the type of the first parameter. Although I can go either way, especially if we add an underscore.

> * the Py_UNICODE_JOIN_SURROGATES() macro should use Py_UCS4 as prefix since it returns Py_UCS4 values, i.e. Py_UCS4_JOIN_SURROGATES()
> * same for the Py_UNICODE_NEXT() macro, i.e. Py_UCS4_NEXT()

I'm not so familiar with the prefix conventions, but wouldn't that lead users to think that this macro is for wide builds and that they have to use Py_UCS2_* macros for narrow builds? If these macros are supposed to abstract the build type maybe they should have a "neutral" prefix. (But if the conventions we use say otherwise I guess the best we can do is to document it properly).
 
> * in order to make the macro easier to understand, please rename it to Py_UCS4_READ_CODE_POINT(); that's a little more typing, but still a lot less than without the macro :-)

The term code point is not entirely correct here. High and low surrogates are code points too. The right term should be 'scalar value' (but that might be confusing). The 'READ' bit sounds fine though, maybe 'READ_NEXT'?

On Sat, Nov 27, 2010 at 5:41 PM, Ezio Melotti <report@bugs.python.org> wrote:
>
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
>
>> * the Py_UNICODE_JOIN_SURROGATES() macro should use Py_UCS4 as prefix since it returns Py_UCS4 values, i.e. Py_UCS4_JOIN_SURROGATES()
>> * same for the Py_UNICODE_NEXT() macro, i.e. Py_UCS4_NEXT()
>
> I'm not so familiar with the prefix conventions, but wouldn't that lead users to think that this macro is for wide builds and that they have to use Py_UCS2_* macros for narrow builds? If these macros are supposed to abstract the build type maybe they should have a "neutral" prefix. (But if the conventions we use say otherwise I guess the best we can do is to document it properly).

When I was using the name, I did not think about argument type.
Py_UNICODE_ is just the namespace prefix used by all macros in
unicodeobject.h. Case in point: Py_UNICODE_ISALPHA() and family that
take Py_UCS4.  (I know, there is a historical reason at work here, but
why fight it?)

Functions use PyUnicode_ prefix and build specific functions use
PyUnicodeUCSx_ prefix.   As far as I can tell, there are no macros
with Py_UCS4_ prefix.  The choices I like in the order of preference
are

1. Py_UNICODE_NEXT
2. Py_UNICODE_NEXT_UCS4
3. Py_UNICODE_READ_NEXT_UCS4

I can live with anything else, though.

I suggest Py_UNICODE_ADVANCE() to avoid false suggestion that the iterator protocol is being used.

> I suggest Py_UNICODE_ADVANCE() to avoid false suggestion that the
> iterator protocol is being used.

You can't use the iterator protocol on a non-PyObject, and Py_UNICODE_*
(as opposed to PyUnicode_*) suggests the macro operates on a raw array
of code points.

AFAIU the macro returns lone surrogates as they are, this means that:
  1) if the string contains only surrogate pairs, Py_UNICODE_NEXT will iterate on scalar values[0];
  2) if the string contains only lone surrogates, it will iterate on codepoints[1];
  3) if it contains both it will be half and half (i.e. scalar values if the surrogates are in pair, or falling back on codepoints if they aren't);
(for strings without surrogates, iterating on scalar values or codepoints is the same).

Is this semantic correct for all (or at least most of) the places where the macro will be used?
Would a stricter version (that rejects lone surrogates and iterates on scalar values only) be useful in addition or in alternative to Py_UNICODE_NEXT?

[0]: http://unicode.org/glossary/#unicode_scalar_value
[1]: http://unicode.org/glossary/#code_point

I am attaching a patch that defines Py_UNICODE_PUT_NEXT() macro (tentative name) and uses it to fix str.upper method.  The implementation of surrogate-aware str.upper shows that NEXT/PUT_NEXT abstractions may lead to somewhat inefficient code for "by codepoint" processing.  The issue is that once in in the process of reading the codepoint, it is determined whether the code point is BMP or non-BMP.  Testing the result again in order to write it is somewhat wasteful.  I don't think this would matter in practice, but would like to hear alternative opinions before moving further. (Please, don't argue over names - let's figure out the proper semantics first.)

On Sat, Nov 27, 2010 at 6:38 PM, Raymond Hettinger
<report@bugs.python.org> wrote:
..
> I suggest Py_UNICODE_ADVANCE() to avoid false suggestion that the iterator protocol is being used.
>

As a data point, ICU defines U16_NEXT() for similar purpose.  I also
like ICU terminology for surrogates ("lead" and "trail") better than
the backward "high" and "low".  The U16_APPEND()  suggests
Py_UNICODE_APPEND instead of PUT_NEXT (this one has a virtue of not
having "next" in the name as well.)  I still like NEXT better than
ADVANCE because it is shorter and has an obvious PREV counterpart that
we may want to add later.

Note that ICU uses U16_ prefix for these macros even when they operate
on 32-bit characters.

More at

http://icu-project.org/apiref/icu4c/utf16_8h.html
http://userguide.icu-project.org/strings

Alexander Belopolsky wrote:
> 
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> On Sat, Nov 27, 2010 at 6:38 PM, Raymond Hettinger
> <report@bugs.python.org> wrote:
> ..
>> I suggest Py_UNICODE_ADVANCE() to avoid false suggestion that the iterator protocol is being used.
>>
> 
> As a data point, ICU defines U16_NEXT() for similar purpose.  I also
> like ICU terminology for surrogates ("lead" and "trail") better than
> the backward "high" and "low". 

"High" and "low" are Unicode standard terms, so we should use
those.

Regarding Py_UCS4_READ_CODE_POINT: you're right that surrogates
are code points, so how about Py_UCS4_READ_NEXT() ?!

Regarding Py_UCS4_READ_NEXT() vs. Py_UNICODE_READ_NEXT(): the return
value of the macro is a Py_UCS4 value, not a Py_UNICODE value. The
first argument of the macro can be any array, not just Py_UNICODE*,
but also Py_UCS4* or even int*.

Py_UCS2_READ_NEXT() would be plain wrong :-) Also note that Python
does have a Py_UCS4 type; it doesn't have a Py_UCS2 type.

That's why we should use *Py_UCS4*_READ_NEXT().

Daniel,

While these macros should not affect ABI, I would appreciate your feedback in light of your work on issue 8654.

+1 on the general idea of abstracting out repeated code.

I will take a closer look at the details within the next few days.

In bltinmodule.c, it looks like some of the indentation doesn't line up?

Bikeshedding aside, it looks good to me.

I agree with Eric Smith that the first part macro name usually refers to the type of the first argument (or the type the first argument points to).  Examples:

 - Py_UNICODE_ISSPACE : Py_UNICODE -> int
 - Py_UNICODE_TOLOWER : Py_UNICODE -> Py_UNICODE
 - Py_UNICODE_strlen: Py_UNICODE * -> size_t

This is true elsewhere in the code as well:

 - PyList_GET_SIZE : PyListObject * -> Py_ssize_t

Yes, I know there are some unfortunate exceptions. ;-)

I agree that it would be nice if something in the name hinted that the return type was Py_UCS4 though.

Marc-Andre Lemburg wrote:
>  The first argument of the macro can be any array, not just
> Py_UNICODE*, but also Py_UCS4* or even int*.

It's true that macros in C do not have any type safety.  While technically passing a Py_UCS4 * will work, on a UCS2 build it would needlessly check the Py_UCS4 data for surrogates.  I think we should discourage that.

You can also technically pass a PyListObject * to PyTuple_GET_SIZE, but that's also not a good idea. ;-)

Alexander Belopolsky wrote:
> The issue is that once in in the process of reading the codepoint, it
> is determined whether the code point is BMP or non-BMP.  Testing the
> result again in order to write it is somewhat wasteful.  I don't
> think this would matter in practice, but would like to hear
> alternative opinions before moving further.

If the common pattern is:

         ch = Py_UNICODE_NEXT(rp, end);
         uc = Py_UNICODE_SOME_TRANSFORMATION(ch);
         Py_UNICODE_PUT_NEXT(wp, uc);

The second check for surrogates in Py_UNICODE_PUT_NEXT is necessary, unless you can prove that Py_UNICODE_SOME_TRANSFORMATION will never transform characters < 0x10000 into characters > 0x10000 or vice versa.  

Can we prove will always be the case, for current and future versions of Unicode, for all or almost-all of the transformations we care about?

Answering that question and figuring out what to do about it are probably more trouble than it's worth.  If a particularly point proves to be a bottleneck, we can always specialize the code there later.

On Fri, Dec 10, 2010 at 6:09 PM, Daniel Stutzbach
<report@bugs.python.org> wrote:
..
> The second check for surrogates in Py_UNICODE_PUT_NEXT is necessary, unless you can prove that
> Py_UNICODE_SOME_TRANSFORMATION will never transform characters < 0x10000 into characters >
> 0x10000 or vice versa.
>
> Can we prove will always be the case, for current and future versions of Unicode, for all or almost-all of the
> transformations we care about?
>
Certainly not for all, but for some important transformations, I
believe Unicode Standard does promise that the transformation  maps
BMP to BMP and supplements to supplements.  For example case folding
and normalization are two important examples.

> Answering that question and figuring out what to do about it are probably more trouble than it's worth.
>  If a particularly point proves to be a bottleneck, we can always specialize the code there later.

Agree.  It is even more likely that the applications that have to deal
with lots of supplementary characters will be better off using a wide
unicode build rather than such specialization.

On Sat, Nov 27, 2010 at 5:03 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
>  * this version should be slightly faster and is also easier to read:
>
> #define Py_UCS4_READ_CODE_POINT(ptr, end) \
..
>      Py_UNICODE_JOIN_SURROGATES((ptr)++, (ptr)++) : \
..
>   I haven't tested it, but you get the idea.

I don't think C guarantees the order of evaluation of the operands in
bitwise expressions such as the expansion of the JOIN_SURROGATES
macro.

I am attaching a patch for commit review.  I added an underscore prefix to all new macros.  This way I am not introducing new features and we will have a full release cycle to come up with better names.  i would just note that "next" terminology is consistent with PyDict_Next and _PySet_NextEntry.  The latter suggests that Py_UNICODE_NEXT_UCS4 may be a better choice.

Alexander Belopolsky wrote:
> 
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> I am attaching a patch for commit review.  I added an underscore prefix to all new macros.  This way I am not introducing new features and we will have a full release cycle to come up with better names.  i would just note that "next" terminology is consistent with PyDict_Next and _PySet_NextEntry.  The latter suggests that Py_UNICODE_NEXT_UCS4 may be a better choice.

I don't think this should go into 3.2. The macros have the potential
of subtly changing Python semantics when used in places that previously
did not support auto-joining surrogates. Let's wait for 3.3 with the
change.

Some comments:

* The macros still need some more attention to enhance their performance.

* For consistency, I'd choose names Py_UNICODE_READ_NEXT()
  and Py_UNICODE_WRITE_NEXT() instead of Py_UNICODE_NEXT() and
  Py_UNICODE_PUT_NEXT().

* Py_UNICODE_JOIN_SURROGATES() either needs to go away completely
  (and be integrated straight into the other macros), or be renamed
  to Py_UCS4_JOIN_SURROGATES(), since it doesn't return Py_UNICODE
  values

* The macros need to be carefully documented, both in unicodeobject.h
  and the general docs.

* Your _Py_UNICODE_PUT_NEXT() implementation is missing a few casts
  to turn ch into a Py_UNICODE/Py_UCS4 value.

* Same for your _Py_UNICODE_NEXT() to make sure that the return
  value is indeed a Py_UNICODE value.

* In general, we should probably be clear on the allowed input
  and define the output types in the documentation.

> Let's wait for 3.3 with the change.

Definitely.

On Wed, Dec 29, 2010 at 10:00 AM, Georg Brandl <report@bugs.python.org> wrote:
..
>
>> Let's wait for 3.3 with the change.
>
> Definitely.

Does this also mean that the numerous surrogates related bugs should
wait until 3.3 as well? (See issues #9200 and #10521.)

This patch was just a stepping stone for the bug fixes.   I
deliberately kept the code changes to the minimum sufficient to
demonstrate and test the new macros.  I would not mind restricting the
patch further by limiting it to the header file changes so that the
macros can be used to fix bugs.  Fixing the bugs in the old verbose
style does not seem feasible.

Note that surrogate bugs are not as exotic as they seem.  For example,
on a wide build I can do

42

but on a narrow build,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    𝐀 = 42
       ^
SyntaxError: invalid character in identifier

So at the moment, narrow and wide builds implement two different languages.

That bug already strikes me as quite exotic.

You need to at least address Marc-Andre's remarks, and to give an overview of what else you'd like to change as well, and how this could affect semantics.

Remember that the next release is already a release candidate.

On Wed, Dec 29, 2010 at 7:19 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
> * The macros still need some more attention to enhance their performance.
>
Although I made your suggested change from '-' to '&', I seriously
doubt that this would make any difference on modern CPUs.  Why do you
think these macros are performance critical?  Users with lots of
supplementary characters in their files are probably better off with a
wide build where Py_UNICODE_NEXT() is just *ptr++ and can hardly be
further optimized.  Higher performance algorithms are possible, but
those should probably do some loop unrolling and/or process more than
one character at a time.  At this point, however it is too soon to
worry about optimization before we even know where these macros will
be used.

> * For consistency, I'd choose names Py_UNICODE_READ_NEXT()
>  and Py_UNICODE_WRITE_NEXT() instead of Py_UNICODE_NEXT() and
>  Py_UNICODE_PUT_NEXT().
>

I would leave it for you and Raymond to reach a consensus.  My
understanding is that Raymond does not want "next" in the name, so
your suggestion still conflicts with that.  I would mildly prefer
GET/PUT over READ/WRITE because the latter suggests multiple
characters.

As discussed before, the macro prefix does not imply the return value.
 Compare this to Py_UNICODE_ISSPACE() and friends or pretty much any
other Py_UNICODE_ macro.   Note that I added a leading underscore to
Py_UNICODE_JOIN_SURROGATES and other new macros, so there is no
immediate pressure to get the names perfect.

> * The macros need to be carefully documented, both in unicodeobject.h
>  and the general docs.
>

I've added a description above _Py_UNICODE_*NEXT macros.  I would
really like to see these macros in private use for a while before they
are published for general audience.  I'll add a comment describing
_Py_UNICODE_JOIN_SURROGATES.  The remaining macros seem to be fairly
self-explanatory (unlike, say Py_UNICODE_ISDIGIT or Py_UNICODE_ISTITLE
which are not documented in unicodeobject.h.)

Explicit downcasts would probably make sense, for example *(ptr)++ =
(Py_UNICODE)ch instead of *(ptr)++ = ch, but I don't think we need
explicit casts say in Py_UCS4 code = (ch) - 0x10000; where they can
mask coding errors.

I also looked for the use of casts elsewhere in unicodeobject.h and
the following does not look right:

#define Py_UNICODE_ISSPACE(ch) \
    ((ch) < 128U ? _Py_ascii_whitespace[(ch)] : _PyUnicode_IsWhitespace(ch))

It looks like this won't work right if ch is a signed char.

> * Same for your _Py_UNICODE_NEXT() to make sure that the return
>  value is indeed a Py_UNICODE value.
>

The return value of _Py_UNICODE_NEXT()  is *not* Py_UNICODE, it is
Py_UCS4 and as far as I can see, every conditional branch in narrow
case has an explicit cast.  In the wide case, I don't think we want an
explicit cast because ptr should already be Py_UCS4* and if it is not,
it may be a coding error that we don't want to mask.

> * In general, we should probably be clear on the allowed input
>  and define the output types in the documentation.

I agree.  I'll add a note that ptr and end should be Py_UNICODE*.  I
am not sure what we should say about ch argument.  If we add casts,
the macro will accept anything, but we should probably document it as
expecting Py_UCS4.

On Sat, Nov 27, 2010 at 5:24 PM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
..
> Perhaps we should allow ord() to work on surrogates in
> UCS4 builds as well. That would reduce the number of
> surprises.
>

This is an interesting idea, however, having surrogates in UCS4 builds
will sooner or later lead to surprises such as

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in
position 0: surrogates not allowed

I though UCS4 (or more properly, UTF-32) did not allow encoding of
surrogate code points.

It is somewhat bothersome that a valid string literal such as
'\uD800\uDC00' in narrow build is subtly invalid in wide build.  It
would probably be better if  '\uD800\uDC00'  was either rejected on a
wide build, or interpreted as a single character so that

True

on any build.

The example in my previous message should have been:

>>> '\U00010000' == '\uD800\uDC00'
True

On Wed, Dec 29, 2010 at 11:36 AM, Georg Brandl <report@bugs.python.org> wrote:
..
> That bug already strikes me as quite exotic.
>
Would it look as exotic if presented like this?

  File "<stdin>", line 1
    𐌀 = 5
       ^
SyntaxError: invalid character in identifier
(works on a wide build)

Note that with few exceptions, pretty much anything you can do with
supplementary characters will produce different results in wide and
narrow builds.  This includes all character type methods (isalpha,
isdigit, etc.), transformations such as case folding or normalization,
text formatting, etc, etc.

When I suggested on python-dev that supplementary character support on
narrow builds is not worth violating fundamental invariants such as
len(chr(i)) == 1, pretty much everyone said that Python should support
full Unicode regardless of build.  When it comes to fixing specific
differences between builds, I hear that these differences are not
important because no one is using supplementary characters.

This example is less exotic than say str.center() or str.swapcase()
not because it involves less exotic characters - all non-BMP
characters are exotic by definition - but because it involves the core
definition of the Python language.

I should stop using e-mail to reply to bug reports!  The mangled example was

>>> 𐌀 = 5
  File "<stdin>", line 1
    𐌀 = 5
       ^
SyntaxError: invalid character in identifier

Le mercredi 29 décembre 2010 à 19:26 +0000, Alexander Belopolsky a
écrit :
> Would it look as exotic if presented like this?
> 
>   File "<stdin>", line 1
>     𐌀 = 5
>        ^
> SyntaxError: invalid character in identifier
> (works on a wide build)

Use non-ASCII identifiers is exotic. Use non-BMP identifiers is
crazy :-) Seriously, it can wait 3.3.

On Wed, Dec 29, 2010 at 3:36 PM, STINNER Victor <report@bugs.python.org> wrote:
..
> Use non-ASCII identifiers is exotic. Use non-BMP identifiers is
> crazy :-)

Hmm, we clearly disagree on what crosses the boundary of the mental
norm.   IMHO, it is crazy to require users to care which plane their
characters come from or whether their programs will be run on a wide
or a narrow build.  I see nothing wrong with a desire to use
characters from say "Mathematical Alphanumeric Symbols" block if that
makes some Python expressions look more like the mathematical formulas
that they represent.  However, it is not about any particular usage,
but about the language definition.  I don't remember even a suggestion
during PEP 3131 discussion that non-BMP characters should be excluded
from identifiers wholesale.

In any case, can someone remind me what was the use case that
motivated chr(i) returning a two-character string for i > 0xFFFF?  I
think we should either stop pretending that narrow builds can handle
non-BMP characters and disallow them in Python strings or we should
try to fix the bugs associated with them.

> Seriously, it can wait 3.3.

What exactly can wait until 3.3?  The presented patch introduces no
user visible changes.  It is only a stepping stone to restoring some
sanity in a way supplementary characters are treated by narrow builds.
 At the moment, it is a mine field: you can easily produce surrogate
pairs from string literals and codecs, but when you start using them,
you have 50% chance that things will blow up, 40% chance of getting
wrong result and maybe 10% chance that it will work.

>> Seriously, it can wait 3.3.
> 
> What exactly can wait until 3.3?  The presented patch introduces no
> user visible changes.  It is only a stepping stone to restoring some
> sanity in a way supplementary characters are treated by narrow builds.
>  At the moment, it is a mine field: you can easily produce surrogate
> pairs from string literals and codecs, but when you start using them,
> you have 50% chance that things will blow up, 40% chance of getting
> wrong result and maybe 10% chance that it will work.

I think the proposal is that fixing this minefield can wait until
Python 3.3 (or even 3.4, or later).

I plan to propose a complete redesign of the representation of Unicode
strings, which may well make this entire set of changes obsolete.

As for language definition: I think the definition is quite clear
and unambiguous. It may be that Python 3.2 doesn't fully implement it.

IOW: relax.

On Wed, Dec 29, 2010 at 8:02 PM, Martin v. Löwis <report@bugs.python.org> wrote:
..
>
> I plan to propose a complete redesign of the representation of Unicode
> strings, which may well make this entire set of changes obsolete.
>

Are you serious?  This sounds like a py4k idea.  Can you give us a
hint on what the new representation will be?  Meanwhile, what it your
recommendation for application developers?  Should they attempt to fix
the code that assumes len(chr(i)) == 1?  Should text processing
applications designed to run on a narrow build simply reject non-BMP
text? Should application writers avoid using str.isxyz() methods?

> As for language definition: I think the definition is quite clear
> and unambiguous. It may be that Python 3.2 doesn't fully implement it.
>

Given that until recently (r87433) the PEP and the reference manual
disagreed on the definition, I have to ask what definition you refer
to.  What Python 3.2 (or rather 3.1) implements, however is important
because it has been declared to be *the* definition of the Python
language regardless of what PEPs docs have to say.

> IOW: relax.

This is the easy part. :-)

On Wed, Dec 29, 2010 at 9:38 PM, Alexander Belopolsky
<report@bugs.python.org> wrote:
..
> Given that until recently (r87433) the PEP and the reference manual
> disagreed on the definition,

Actually, it looks like PEP 3131 and the Language Reference [1] still
disagree.  The latter says:

identifier  ::=  id_start id_continue*

which should probably be

identifier  ::=  xid_start xid_continue*

instead.
[1] http://docs.python.org/py3k/reference/lexical_analysis.html#identifiers

> Are you serious?  This sounds like a py4k idea.  Can you give us a
> hint on what the new representation will be?

I'm thinking about an approach of a variable representation:
one, two, or four bytes, depending on the widest character that
appears in the string. I think it can be arranged to make this mostly
backwards-compatible with existing APIs, so it doesn't need to wait
for py4k, IMO. OTOH, I'm not sure I'll make it for 3.3.

> Meanwhile, what it your
> recommendation for application developers?  Should they attempt to fix
> the code that assumes len(chr(i)) == 1?  Should text processing
> applications designed to run on a narrow build simply reject non-BMP
> text? Should application writers avoid using str.isxyz() methods?

Given that this is vaporware: proceed as if that idea didn't exist.

> Actually, it looks like PEP 3131 and the Language Reference [1] still
> disagree.  The latter says:
> 
> identifier  ::=  id_start id_continue*
> 
> which should probably be
> 
> identifier  ::=  xid_start xid_continue*
> 
> instead.

Interesting. XID_* is being used in the PEP since r57023, whereas the
documentation was added in r57824. In any case, this is now fixed in
r87575/r87576.

> I think the proposal is that fixing this minefield can wait until
> Python 3.3 (or even 3.4, or later).

That is what I was thinking.  (Alex: You might not know that Martin
was the main proponent of non-ASCII identifiers, so this assessment
should have some weight.)

> I'm thinking about an approach of a variable representation:
> one, two, or four bytes, depending on the widest character that
> appears in the string. I think it can be arranged to make this mostly
> backwards-compatible with existing APIs, so it doesn't need to wait
> for py4k, IMO. OTOH, I'm not sure I'll make it for 3.3.

That is an interesting idea.  I would be interested in helping out
when you'll implement it.

See also #12751.

A PEP 393 draft implementation is available at https://bitbucket.org/t0rsten/pep-393/ (branch pep-393); if this gets into 3.3, this issue will be outdated: there won't be "narrow" builds of Python anymore (nor will there be "wide" builds).

That's a really good news.
Some Unicode issues can still be fixed on 2.7 and 3.2 though.
FWIW I was planning to look at this and #9200 in the following days and see if I can fix them.

Martin v. Löwis wrote:
> 
> A PEP 393 draft implementation is available at https://bitbucket.org/t0rsten/pep-393/ (branch pep-393); if this gets into 3.3, this issue will be outdated: there won't be "narrow" builds of Python anymore (nor will there be "wide" builds).

Even if PEP 393 should go into Py4k one day (I don't believe that
such major changes can be done in a minor release), we will still have
to deal with surrogates in codecs, which is where these macros will get
used, so I don't see how PEP 393 relates to the idea of adding helper
macros to simplify the code.

I think the 4 macros:
 #define _Py_UNICODE_ISSURROGATE
 #define _Py_UNICODE_ISHIGHSURROGATE
 #define _Py_UNICODE_ISLOWSURROGATE
 #define _Py_UNICODE_JOIN_SURROGATES
are quite straightforward and can avoid using the trailing _.

Since I would like to see #9200 fixed on 3.2 (and possibly 2.7 too), would it be ok to:
 1) commit the patch with the trailing _ for all the macros on 3.2(/2.7);
 2) commit the patch with the trailing _ only for the _NEXT macros in 3.3;
 3) fix #9200 on all these branches using the new macros (with or without _);
 4) remove the trailing _ from the _NEXT macros in 3.4 if it turns out to work well;


> we will still have to deal with surrogates in codecs,
> which is where these macros will get used

They will also be used in many str methods and afaiu PEP 393 should address that.  I'm not sure it addresses codecs and builtin functions like chr() and ord() too.

> I think the 4 macros:
>  #define _Py_UNICODE_ISSURROGATE
>  #define _Py_UNICODE_ISHIGHSURROGATE
>  #define _Py_UNICODE_ISLOWSURROGATE
>  #define _Py_UNICODE_JOIN_SURROGATES
> are quite straightforward and can avoid using the trailing _.

I don't want to bikeshed, but can we have proper consistent word
separation?
_Py_UNICODE_IS_HIGH_SURROGATE, not _Py_UNICODE_ISHIGHSURROGATE
(etc.)

> > we will still have to deal with surrogates in codecs,
> > which is where these macros will get used
> 
> They will also be used in many str methods and afaiu PEP 393 should
> address that.  I'm not sure it addresses codecs and builtin functions
> like chr() and ord() too.

AFAIU, PEP 393 avoids producing surrogate pairs in the canonical
internal representation (that's one of its selling points). Only the
UTF-16 codecs would need to deal with surrogate pairs, in the encoded
form.

All the other macros[0] follow the same convention, e.g. Py_UNICODE_ISLOWER and Py_UNICODE_TOLOWER.  I agree that keeping the words separate makes them more readable though.
[0]: Include/unicodeobject.h:328

>Ezio Melotti <ezio.melotti@gmail.com> added the comment:

>I think the 4 macros:
> #define _Py_UNICODE_ISSURROGATE
> #define _Py_UNICODE_ISHIGHSURROGATE
> #define _Py_UNICODE_ISLOWSURROGATE
> #define _Py_UNICODE_JOIN_SURROGATES
>are quite straightforward and can avoid using the trailing _.

For what it's worth, I've seen Unicode documentation that talks about
that prefers the terms "lead surrogate" and "trail surrogate" as being
clearer than the terms "high surrgoate" and "low   surrogate".

For example, from the Unicode BOM FAQ at http://unicode.org/faq/utf_bom.html

    Q: What are surrogates?

    A: Surrogates are code points from two special ranges of Unicode values, reserved for use as the leading, and
       trailing values of paired code units in UTF-16. Leading, also called high, surrogates are from D800₁₆ to DBFF₁₆,
       and trailing, or low, surrogates are from DC00₁₆ to DFFF₁₆. They are called surrogates, since they do not
       represent characters directly, but only as a pair.

BTW, considering recent discussions, you might want to read:

    Q: Are there any 16-bit values that are invalid?

    A: The two values FFFE₁₆ and FFFF₁₆ as well as the 32 values from FDD0₁₆ to FDEF₁₆ represent noncharacters. They are
       invalid in interchange, but may be freely used internal to an implementation. Unpaired surrogates are invalid as
       well, i.e. any value in the range D800₁₆ to DBFF₁₆ not followed by a value in the range DC00₁₆ to DFFF₁₆, or any
       value in the range DC00₁₆ to DFFF₁₆ not preceded by a value in the range D800₁₆ to DBFF₁₆. [AF]

and also the answer to:

    Q: Are there any paired surrogates that are invalid?

whose answer I here omit for brevity, as it is a table.

I suspect that you guys are now increasingly sold on the answer to the next FAQ right after that one, now. :)

    Q: Because supplementary characters are uncommon, does that mean I can ignore them?

    A: Just because supplementary characters (expressed with surrogate pairs in UTF-16) are uncommon does 
       not mean that they should be neglected. They include:

        * emoji symbols and emoticons, for interoperating with Japanese mobile phones
        * uncommon (but not unused) CJK characters, important for personal and place names
        * variation selectors for ideographic variation sequences
        * important symbols for mathematics
        * numerous minority scripts and historic scripts, important for some user communities

Another example of using "lead" and "trail" surrogates is in the first
sentence from http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UTF16.html

    * Naming: For clarity, High and Low surrogates are called Lead and Trail in the API, which gives a better sense of
      their ordering in a string. offset16 and offset32 are used to distinguish offsets to UTF-16 boundaries vs offsets
      to UTF-32 boundaries. int char32 is used to contain UTF-32 characters, as opposed to char16, which is a UTF-16
      code unit.
    * Roundtripping Offsets: You can always roundtrip from a UTF-32 offset to a UTF-16 offset and back. Because of the
      difference in structure, you can roundtrip from a UTF-16 offset to a UTF-32 offset and back if and only if
      bounds(string, offset16) != TRAIL.
    * Exceptions: The error checking will throw an exception if indices are out of bounds. Other than than that, all
      methods will behave reasonably, even if unmatched surrogates or out-of-bounds UTF-32 values are present.
      UCharacter.isLegal() can be used to check for validity if desired.
    * Unmatched Surrogates: If the string contains unmatched surrogates, then these are counted as one UTF-32 value.
      This matches their iteration behavior, which is vital. It also matches common display practice as missing glyphs
      (see the Unicode Standard Section 5.4, 5.5).
    * Optimization: The method implementations may need optimization if the compiler doesn't fold static final methods.
      Since surrogate pairs will form an exceeding small percentage of all the text in the world, the singleton case
      should always be optimized for.

You can also see this reflected in the utf.h file from the ICU project as part of their C API in ICU4C:

    #define     U_SENTINEL   (-1)
            This value is intended for sentinel values for APIs that (take or) return single code points (UChar32). 
    #define     U_IS_UNICODE_NONCHAR(c)
            Is this code point a Unicode noncharacter? 
    #define     U_IS_UNICODE_CHAR(c)
            Is c a Unicode code point value (0..U+10ffff) that can be assigned a character? 
    #define     U_IS_BMP(c)   ((uint32_t)(c)<=0xffff)
            Is this code point a BMP code point (U+0000..U+ffff)? 
    #define     U_IS_SUPPLEMENTARY(c)   ((uint32_t)((c)-0x10000)<=0xfffff)
            Is this code point a supplementary code point (U+10000..U+10ffff)? 
    #define     U_IS_LEAD(c)   (((c)&0xfffffc00)==0xd800)
            Is this code point a lead surrogate (U+d800..U+dbff)? 
    #define     U_IS_TRAIL(c)   (((c)&0xfffffc00)==0xdc00)
            Is this code point a trail surrogate (U+dc00..U+dfff)? 
    #define     U_IS_SURROGATE(c)   (((c)&0xfffff800)==0xd800)
            Is this code point a surrogate (U+d800..U+dfff)? 
    #define     U_IS_SURROGATE_LEAD(c)   (((c)&0x400)==0)
            Assuming c is a surrogate code point (U_IS_SURROGATE(c)), is it a lead surrogate? 
    #define     U_IS_SURROGATE_TRAIL(c)   (((c)&0x400)!=0)
            Assuming c is a surrogate code point (U_IS_SURROGATE(c)), is it a trail surrogate?

Another one is:

    http://www.opensource.apple.com/source/WebCore/WebCore-1C25/icu/unicode/utf16.h

which contains:

    #define U16_IS_SINGLE(c) !U_IS_SURROGATE(c)
    #define U16_IS_LEAD(c) (((c)&0xfffffc00)==0xd800)
    #define U16_IS_TRAIL(c) (((c)&0xfffffc00)==0xdc00)
    #define U16_IS_SURROGATE(c) U_IS_SURROGATE(c)
    #define U16_IS_SURROGATE_LEAD(c) (((c)&0x400)==0)
    #define U16_SURROGATE_OFFSET ((0xd800<<10UL)+0xdc00-0x10000)
    #define U16_GET_SUPPLEMENTARY(lead, trail) \
    #define U16_LEAD(supplementary) (UChar)(((supplementary)>>10)+0xd7c0)
    #define U16_TRAIL(supplementary) (UChar)(((supplementary)&0x3ff)|0xdc00)
    #define U16_LENGTH(c) ((uint32_t)(c)<=0xffff ? 1 : 2)

In fact, you might want to read over that file, as it has embedded documentation
for these, and has other macros for being careful about surrogates.  For example,
here's one in full:

    /**
     * Get a code point from a string at a random-access offset,
     * without changing the offset.
     * "Unsafe" macro, assumes well-formed UTF-16.
     *
     * The offset may point to either the lead or trail surrogate unit
     * for a supplementary code point, in which case the macro will read
     * the adjacent matching surrogate as well.
     * The result is undefined if the offset points to a single, unpaired surrogate.
     * Iteration through a string is more efficient with U16_NEXT_UNSAFE or U16_NEXT.
     *
     * @param s const UChar * string
     * @param i string offset
     * @param c output UChar32 variable
     * @see U16_GET
     * @stable ICU 2.4
     */
    #define U16_GET_UNSAFE(s, i, c) { \
	(c)=(s)[i]; \
	if(U16_IS_SURROGATE(c)) { \
	    if(U16_IS_SURROGATE_LEAD(c)) { \
		(c)=U16_GET_SUPPLEMENTARY((c), (s)[(i)+1]); \
	    } else { \
		(c)=U16_GET_SUPPLEMENTARY((s)[(i)-1], (c)); \
	    } \
	} \
    }

So keeping your preamble bits, I might have considered doing it
this way if it were me doing it:

    #define _Py_UNICODE_IS_SURROGATE
    #define _Py_UNICODE_IS_LEAD_SURROGATE
    #define _Py_UNICODE_IS_TRAIL_SURROGATE
    #define _Py_UNICODE_JOIN_SURROGATES

But I also come from a culture that uses more underscores than you guys tend 
to, as shown in some of the macro names shown below from utf8.h file.  I find
that most projects use more underscores in uppercase names than Python does. :)

--tom

#define UTF_START_MARK(len) (((len) >  7) ? 0xFF : (0xFE << (7-(len))))
#define UTF_START_MASK(len) (((len) >= 7) ? 0x00 : (0x1F >> ((len)-2)))
#define UTF_CONTINUATION_MARK           0x80
#define UTF_ACCUMULATION_SHIFT          6
#define UTF_CONTINUATION_MASK           ((U8)0x3f)
#define UNISKIP(uv) ( (uv) < 0x80           ? 1 : \
#define UNISKIP(uv) ( (uv) < 0x80           ? 1 : \
#define NATIVE_IS_INVARIANT(c)          UNI_IS_INVARIANT(NATIVE8_TO_UNI(c))
#define IN_BYTES (CopHINTS_get(PL_curcop) & HINT_BYTES)
#define UNICODE_SURROGATE_FIRST         0xD800
#define UNICODE_SURROGATE_LAST          0xDFFF
#define UNICODE_REPLACEMENT             0xFFFD
#define UNICODE_BYTE_ORDER_MARK         0xFEFF
#define PERL_UNICODE_MAX        0x10FFFF
#define UNICODE_WARN_SURROGATE     0x0001       /* UTF-16 surrogates */
#define UNICODE_WARN_NONCHAR       0x0002       /* Non-char code points */
#define UNICODE_WARN_SUPER         0x0004       /* Above 0x10FFFF */
#define UNICODE_WARN_FE_FF         0x0008       
#define UNICODE_DISALLOW_SURROGATE 0x0010
#define UNICODE_DISALLOW_NONCHAR   0x0020
#define UNICODE_DISALLOW_SUPER     0x0040
#define UNICODE_DISALLOW_FE_FF     0x0080
#define UNICODE_WARN_ILLEGAL_INTERCHANGE \
#define UNICODE_DISALLOW_ILLEGAL_INTERCHANGE \
#define UNICODE_ALLOW_SURROGATE 0
#define UNICODE_ALLOW_SUPER     0
#define UNICODE_ALLOW_ANY       0
#define UNICODE_IS_SURROGATE(c)         ((c) >= UNICODE_SURROGATE_FIRST && \
#define UNICODE_IS_REPLACEMENT(c)       ((c) == UNICODE_REPLACEMENT)
#define UNICODE_IS_BYTE_ORDER_MARK(c)   ((c) == UNICODE_BYTE_ORDER_MARK)
#define UNICODE_IS_NONCHAR(c)           ((c >= 0xFDD0 && c <= 0xFDEF) \
#define UNICODE_IS_SUPER(c)             ((c) > PERL_UNICODE_MAX)
#define UNICODE_IS_FE_FF(c)             ((c) > 0x7FFFFFFF)
#define UNICODE_GREEK_CAPITAL_LETTER_SIGMA      0x03A3
#define UNICODE_GREEK_SMALL_LETTER_FINAL_SIGMA  0x03C2
#define UNICODE_GREEK_SMALL_LETTER_SIGMA        0x03C3
#define GREEK_SMALL_LETTER_MU                   0x03BC
#define GREEK_CAPITAL_LETTER_MU 0x039C  /* Upper and title case of MICRON */
#define LATIN_CAPITAL_LETTER_Y_WITH_DIAERESIS 0x0178    /* Also is title case */
#define LATIN_CAPITAL_LETTER_SHARP_S    0x1E9E
#define UNI_DISPLAY_ISPRINT     0x0001
#define UNI_DISPLAY_BACKSLASH   0x0002
#define UNI_DISPLAY_QQ          (UNI_DISPLAY_ISPRINT|UNI_DISPLAY_BACKSLASH)
#define UNI_DISPLAY_REGEX       (UNI_DISPLAY_ISPRINT|UNI_DISPLAY_BACKSLASH)
#define LATIN_SMALL_LETTER_SHARP_S      0x00DF
#define LATIN_SMALL_LETTER_Y_WITH_DIAERESIS 0x00FF
#define MICRO_SIGN 0x00B5
#define LATIN_CAPITAL_LETTER_A_WITH_RING_ABOVE 0x00C5
#define LATIN_SMALL_LETTER_A_WITH_RING_ABOVE 0x00E5
#define ANYOF_FOLD_SHARP_S(node, input, end)    \
#define SHARP_S_SKIP 2

PS: Those won't always make sense for lack of continuation lines and enclosing ifdefs.

I now see there are lots of good things in the BOM FAQ that have come up
lately regarding surrogates and other illegal characters, and about what
can go in data streams.  

I quote a few of these from http://unicode.org/faq/utf_bom.html below:

    Q: How do I convert an unpaired UTF-16 surrogate to UTF-8? 

    A: A different issue arises if an unpaired surrogate is encountered when converting ill-formed UTF-16 data. 
       By represented such an *unpaired* surrogate on its own as a 3-byte sequence, the resulting UTF-8 data stream
       would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires
       that encoding form conversion always results in valid data stream. Therefore a converter *must* treat this
       as an error.

    Q: How do I convert an unpaired UTF-16 surrogate to UTF-32? 

    A: If an unpaired surrogate is encountered when converting ill-formed UTF-16 data, any conformant converter must
       treat this as an error. By representing such an unpaired surrogate on its own, the resulting UTF-32 data stream
       would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that
       encoding form conversion always results in valid data stream.

    Q: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, then can I still assume the remaining
       UTF-8 bytes are in big-endian order?

    A: Yes, UTF-8 can contain a BOM. However, it makes no difference as to the endianness of the byte stream. UTF-8
       always has the same byte order. An initial BOM is only used as a signature — an indication that an otherwise
       unmarked text file is in UTF-8. Note that some recipients of UTF-8 encoded data do not expect a BOM. Where UTF-8
       is used transparently in 8-bit environments, the use of a BOM will interfere with any protocol or file format
       that expects specific ASCII characters at the beginning, such as the use of "#!" of at the beginning of Unix
       shell scripts.

    Q: What should I do with U+FEFF in the middle of a file?

    A: In the absence of a protocol supporting its use as a BOM and when not at the beginning of a text stream, U+FEFF
       should normally not occur. For backwards compatibility it should be treated as ZERO WIDTH NON-BREAKING SPACE
       (ZWNBSP), and is then part of the content of the file or string. The use of U+2060 WORD JOINER is strongly
       preferred over ZWNBSP for expressing word joining semantics since it cannot be confused with a BOM. When
       designing a markup language or data protocol, the use of U+FEFF can be restricted to that of Byte Order Mark. In
       that case, any U+FEFF occurring in the middle of a file can be treated as an unsupported character.

    Q: How do I tag data that does not interpret U+FEFF as a BOM?

    A: Use the tag UTF-16BE to indicate big-endian UTF-16 text, and UTF-16LE to indicate little-endian UTF-16 text. 
       If you do use a BOM, tag the text as simply UTF-16. 

    Q: Why wouldn’t I always use a protocol that requires a BOM?

    A: Where the data has an associated type, such as a field in a database, a BOM is unnecessary. In particular, 
       if a text data stream is marked as UTF-16BE, UTF-16LE, UTF-32BE or UTF-32LE, a BOM is neither necessary *nor
       permitted*. Any U+FEFF would be interpreted as a ZWNBSP.  Do not tag every string in a database or set of fields
       with a BOM, since it wastes space and complicates string concatenation. Moreover, it also means two data fields
       may have precisely the same content, but not be binary-equal (where one is prefaced by a BOM).

Somewhat frustratingly, I am now almost more confused than ever by the last two sentences here:

    Q: What is a UTF?

    A: A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate
       code points) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format” for
       UTF; the two terms are merely synonyms for the same concept.

       Each UTF is reversible, thus every UTF supports *lossless round tripping*: mapping from any Unicode coded
       character sequence S to a sequence of bytes and back will produce S again. To ensure round tripping, a UTF
       mapping *must also* map all code points that are not valid Unicode characters to unique byte sequences. These
       invalid code points are the 66 *noncharacters* (including FFFE and FFFF), as well as unpaired surrogates.

My confusion is about the invalid code points. The first two FAQs I cite at the top are quite clear that it is illegal
to have unpaired surrogates in a UTF stream.  I don’t understand therefore what it saying about “must also” mapping all
code points that aren’t valid Unicode characters to “unique byte sequences” to ensure roundtripping.  At first reading,
I’d almost say those appear to contradict each other.  I must just be being boneheaded though.  It’s very early morning
yet, and maybe it will become clearer upon a fifth or sixth reading.  Maybe it has to with replacement characters?  No, 
that can’t be right.  Muddle muddle.  Sigh.

Important material is also found in http://www.unicode.org/faq/basic_q.html:

    Q: Are surrogate characters the same as supplementary characters?

    A: This question shows a common confusion. It is very important to distinguish surrogate code points (in the range
       U+D800..U+DFFF) from supplementary code points (in the completely different range, U+10000..U+10FFFF). Surrogate
       code points are reserved for use, in pairs, in representing supplementary code points in UTF-16.

       There are supplementary characters (i.e. encoded characters represented with a single supplementary code point), but
       there are not and will never be surrogate characters (i.e. encoded characters represented with a single surrogate
       code point).

    Q: What is the difference between UCS-2 and UTF-16?

    A: UCS-2 is obsolete terminology which refers to a Unicode implementation up to Unicode 1.1, before surrogate code
       points and UTF-16 were added to Version 2.0 of the standard. This term should now be avoided.

       UCS-2 does not define a distinct data format, because UTF-16 and UCS-2 are identical for purposes of data exchange.
       Both are 16-bit, and have exactly the same code unit representation.

       Sometimes in the past an implementation has been labeled "UCS-2" to indicate that it does not support supplementary
       characters and doesn't interpret pairs of surrogate code points as characters. Such an implementation would not
       handle processing of character properties, code point boundaries, collation, etc. for supplementary characters.

And in reference to UTF-16 being slower by code point than by code unit:

    Q: How about using UTF-32 interfaces in my APIs?

    A: Except in some environments that store text as UTF-32 in memory, most Unicode APIs are using UTF-16. With UTF-16
       APIs  the low level indexing is at the storage or code unit level, with higher-level mechanisms for graphemes or
       words specifying their boundaries in terms of the code units. This provides efficiency at the low levels, and the
       required functionality at the high levels.

        If its [sic] ever necessary to locate the nᵗʰ character, indexing by character can be implemented as a high
        level operation. However, while converting from such a UTF-16 code unit index to a character index or vice versa
        is fairly straightforward, it does involve a scan through the 16-bit units up to the index point. In a test run,
        for example, accessing UTF-16 storage as characters, instead of code units resulted in a 10× degradation. While
        there are some interesting optimizations that can be performed, it will always be slower on average. Therefore
        locating other boundaries, such as grapheme, word, line or sentence boundaries proceeds directly from the code
        unit index, not indirectly via an intermediate character code index.

I am somewhat amused by this summary:

    Q: What does Unicode conformance require?

    A: Chapter 3, Conformance discusses this in detail. Here's a very informal version: 

        * Unicode characters don't fit in 8 bits; deal with it.
        * 2 [sic] Byte order is only an issue in I/O.
        * If you don't know, assume big-endian.
        * Loose surrogates have no meaning.
        * Neither do U+FFFE and U+FFFF.
        * Leave the unassigned codepoints alone.
        * It's OK to be ignorant about a character, but not plain wrong.
        * Subsets are strictly up to you.
        * Canonical equivalence matters.
        * Don't garble what you don't understand.
        * Process UTF-* by the book.
        * Ignore illegal encodings.
        * Right-to-left scripts have to go by bidi rules. 

And don’t know what I think about this, except that there sure a lot of 
screw‐ups out there if it is truly as easy as they would would have you believe:

    Given that any industrial-strength text and internationalization support API has to be able to handle sequences of
    characters, it makes little difference whether the string is internally represented by a sequence of [...] code
    units, or by a sequence of code-points [...]. Both UTF-16 and UTF-8 are designed to make working with substrings
    easy, by the fact that the sequence of code units for a given code point is unique.

Take this all with a grain of salt, since I found various typos in these FAQs
and occasionally also language that seems to reflect an older nomenclature than
is now seen in the current published Unicode Standard, meaning 6.0.0.  Probably
best then to take only general directives from their FAQs and leave language‐
lawyering to the formal printed Standard, insofar as that is possible — which
sometimes it is not, because they do make mistakes from time to time, and even
less frequently, correct these.  :)

--tom

Antoine Pitrou <report@bugs.python.org> wrote
   on Tue, 16 Aug 2011 09:18:46 -0000: 

>> I think the 4 macros:
>>  #define _Py_UNICODE_ISSURROGATE
>>  #define _Py_UNICODE_ISHIGHSURROGATE
>>  #define _Py_UNICODE_ISLOWSURROGATE
>>  #define _Py_UNICODE_JOIN_SURROGATES
>> are quite straightforward and can avoid using the trailing _.

> I don't want to bikeshed, but can we have proper consistent word separation?
> _Py_UNICODE_IS_HIGH_SURROGATE, not _Py_UNICODE_ISHIGHSURROGATE
> (etc.)

Oh good, I thought it was only me whohadtroublereadingthose. :)

--tom

Ezio Melotti <report@bugs.python.org> wrote
   on Tue, 16 Aug 2011 09:23:50 -0000: 

> All the other macros[0] follow the same convention, e.g. Py_UNICODE_ISLOWER
> and Py_UNICODE_TOLOWER.  I agree that keeping the words separate makes them
> more readable though.

> 	[0]: Include/unicodeobject.h:328

I am guessing that that is not quite why those don't have underscores
in them.  I bet it is actually something else.  Watch:

    % unigrep '^\s*#\s*define\s+Py_[\p{Lu}_]+\b' unicodeobject.h
    #define Py_UNICODEOBJECT_H
    #define Py_USING_UNICODE
    #define Py_UNICODE_WIDE
    #define Py_UNICODE_ISSPACE(ch) \
    #define Py_UNICODE_ISLOWER(ch) _PyUnicode_IsLowercase(ch)
    #define Py_UNICODE_ISUPPER(ch) _PyUnicode_IsUppercase(ch)
    #define Py_UNICODE_ISTITLE(ch) _PyUnicode_IsTitlecase(ch)
    #define Py_UNICODE_ISLINEBREAK(ch) _PyUnicode_IsLinebreak(ch)
    #define Py_UNICODE_TOLOWER(ch) _PyUnicode_ToLowercase(ch)
    #define Py_UNICODE_TOUPPER(ch) _PyUnicode_ToUppercase(ch)
    #define Py_UNICODE_TOTITLE(ch) _PyUnicode_ToTitlecase(ch)
    #define Py_UNICODE_ISDECIMAL(ch) _PyUnicode_IsDecimalDigit(ch)
    #define Py_UNICODE_ISDIGIT(ch) _PyUnicode_IsDigit(ch)
    #define Py_UNICODE_ISNUMERIC(ch) _PyUnicode_IsNumeric(ch)
    #define Py_UNICODE_ISPRINTABLE(ch) _PyUnicode_IsPrintable(ch)
    #define Py_UNICODE_TODECIMAL(ch) _PyUnicode_ToDecimalDigit(ch)
    #define Py_UNICODE_TODIGIT(ch) _PyUnicode_ToDigit(ch)
    #define Py_UNICODE_TONUMERIC(ch) _PyUnicode_ToNumeric(ch)
    #define Py_UNICODE_ISALPHA(ch) _PyUnicode_IsAlpha(ch)
    #define Py_UNICODE_ISALNUM(ch) \
    #define Py_UNICODE_COPY(target, source, length)                         \
    #define Py_UNICODE_FILL(target, value, length) \
    #define Py_UNICODE_MATCH(string, offset, substring) \
    #define Py_UNICODE_REPLACEMENT_CHARACTER ((Py_UNICODE) 0xFFFD)

It looks like what is actually happening there is that you started out
with names of the normal ctype(3) macroish thingies:

     isalpha isupper islower isdigit isxdigit isalnum isspace ispunct
     isprint isgraph iscntrl isblank isascii  toupper isblank isascii
     toupper tolower toascii

and wanted to preserve those, which would lead to Py_UNICODE_TOLOWER and
Py_UNICODE_TOUPPER, since there are no functions in the original C versions
those seem to mirror.  Then when you wanted more of that ilk, you sensibly
kept to the same naming convention.

I eyeball few exceptions to that style here:

    % perl -nle '/^\s*#\s*define\s+(Py_[\p{Lu}_]+)\b/ and print $1' Include/*.h | sort -dfu | fmt -150
    Py_ABSTRACTOBJECT_H Py_ALIGNED Py_ALLOW_RECURSION Py_ARITHMETIC_RIGHT_SHIFT Py_ASDL_H Py_AST_H Py_ATOMIC_H Py_BEGIN_ALLOW_THREADS Py_BITSET_H
    Py_BLOCK_THREADS Py_BLTINMODULE_H Py_BOOLOBJECT_H Py_BYTEARRAYOBJECT_H Py_BYTES_CTYPE_H Py_BYTESOBJECT_H Py_CAPSULE_H Py_CELLOBJECT_H Py_CEVAL_H
    Py_CHARMASK Py_CLASSOBJECT_H Py_CLEANUP_SUPPORTED Py_CLEAR Py_CODECREGISTRY_H Py_CODE_H Py_COMPILE_H Py_COMPLEXOBJECT_H Py_CURSES_H Py_DECREF
    Py_DEPRECATED Py_DESCROBJECT_H Py_DICTOBJECT_H Py_DTSF_ALT Py_DTSF_SIGN Py_DTST_FINITE Py_DTST_INFINITE Py_DTST_NAN Py_END_ALLOW_RECURSION
    Py_END_ALLOW_THREADS Py_ENUMOBJECT_H Py_EQ Py_ERRCODE_H Py_ERRORS_H Py_EVAL_H Py_FILEOBJECT_H Py_FILEUTILS_H Py_FLOATOBJECT_H Py_FORCE_DOUBLE
    Py_FORCE_EXPANSION Py_FORMAT_PARSETUPLE Py_FRAMEOBJECT_H Py_FUNCOBJECT_H Py_GCC_ATTRIBUTE Py_GE Py_GENOBJECT_H Py_GETENV Py_GRAMMAR_H Py_GT
    Py_HUGE_VAL Py_IMPORT_H Py_INCREF Py_INTRCHECK_H Py_INVALID_SIZE Py_ISALNUM Py_ISALPHA Py_ISDIGIT Py_IS_FINITE Py_IS_INFINITY Py_ISLOWER Py_IS_NAN
    Py_ISSPACE Py_ISUPPER Py_ISXDIGIT Py_ITEROBJECT_H Py_LE Py_LISTOBJECT_H Py_LL Py_LOCAL Py_LOCAL_INLINE Py_LONGINTREPR_H Py_LONGOBJECT_H Py_LT
    Py_MARSHAL_H Py_MARSHAL_VERSION Py_MATH_E Py_MATH_PI Py_MEMCPY Py_MEMORYOBJECT_H Py_METAGRAMMAR_H Py_METHODOBJECT_H Py_MODSUPPORT_H Py_MODULEOBJECT_H
    Py_NAN Py_NE Py_NODE_H Py_OBJECT_H Py_OBJIMPL_H Py_OPCODE_H Py_OSDEFS_H Py_OVERFLOWED Py_PARSETOK_H Py_PGEN_H Py_PGENHEADERS_H Py_PRINT_RAW
    Py_PYARENA_H Py_PYDEBUG_H Py_PYFPE_H Py_PYGETOPT_H Py_PYMATH_H Py_PYMEM_H Py_PYPORT_H Py_PYSTATE_H Py_PYTHON_H Py_PYTHONRUN_H Py_PYTHREAD_H
    Py_PYTIME_H Py_RANGEOBJECT_H Py_REFCNT Py_REF_DEBUG Py_RETURN_FALSE Py_RETURN_INF Py_RETURN_NAN Py_RETURN_NONE Py_RETURN_TRUE Py_SAFE_DOWNCAST
    Py_SET_ERANGE_IF_OVERFLOW Py_SET_ERRNO_ON_MATH_ERROR Py_SETOBJECT_H Py_SIZE Py_SLICEOBJECT_H Py_STRCMP_H Py_STRTOD_H Py_STRUCTMEMBER_H Py_STRUCTSEQ_H
    Py_SYMTABLE_H Py_SYSMODULE_H Py_TOKEN_H Py_TOLOWER Py_TOUPPER Py_TPFLAGS_BASE_EXC_SUBCLASS Py_TPFLAGS_BASETYPE Py_TPFLAGS_BYTES_SUBCLASS
    Py_TPFLAGS_DEFAULT Py_TPFLAGS_DICT_SUBCLASS Py_TPFLAGS_HAVE_GC Py_TPFLAGS_HAVE_STACKLESS_EXTENSION Py_TPFLAGS_HAVE_VERSION_TAG Py_TPFLAGS_HEAPTYPE
    Py_TPFLAGS_INT_SUBCLASS Py_TPFLAGS_IS_ABSTRACT Py_TPFLAGS_LIST_SUBCLASS Py_TPFLAGS_LONG_SUBCLASS Py_TPFLAGS_READY Py_TPFLAGS_READYING
    Py_TPFLAGS_TUPLE_SUBCLASS Py_TPFLAGS_TYPE_SUBCLASS Py_TPFLAGS_UNICODE_SUBCLASS Py_TPFLAGS_VALID_VERSION_TAG Py_TRACEBACK_H Py_TRACE_REFS
    Py_TRASHCAN_SAFE_BEGIN Py_TRASHCAN_SAFE_END Py_TUPLEOBJECT_H Py_TYPE Py_UCNHASH_H Py_ULL Py_UNBLOCK_THREADS Py_UNICODE_COPY Py_UNICODE_FILL
    Py_UNICODE_ISALNUM Py_UNICODE_ISALPHA Py_UNICODE_ISDECIMAL Py_UNICODE_ISDIGIT Py_UNICODE_ISLINEBREAK Py_UNICODE_ISLOWER Py_UNICODE_ISNUMERIC
    Py_UNICODE_ISPRINTABLE Py_UNICODE_ISSPACE Py_UNICODE_ISTITLE Py_UNICODE_ISUPPER Py_UNICODE_MATCH Py_UNICODEOBJECT_H Py_UNICODE_REPLACEMENT_CHARACTER
    Py_UNICODE_TODECIMAL Py_UNICODE_TODIGIT Py_UNICODE_TOLOWER Py_UNICODE_TONUMERIC Py_UNICODE_TOTITLE Py_UNICODE_TOUPPER Py_UNICODE_WIDE Py_USING_UNICODE
    Py_VA_COPY Py_VISIT Py_WARNINGS_H Py_WEAKREFOBJECT_H Py_XDECREF Py_XINCREF

See what I mean?    Most of them that remain tend to be things that one could
construe as compound words, like "RANGEOBJECT" or "CODEREGISTRY",  though some
people might find a few a bit on the longish side to read unaided by
underscores, like "BYTEARRAYOBJECT".

'Nuff bikeshedding. :)

--tom

Tom Christiansen wrote:
> So keeping your preamble bits, I might have considered doing it
> this way if it were me doing it:
> 
>     #define _Py_UNICODE_IS_SURROGATE
>     #define _Py_UNICODE_IS_LEAD_SURROGATE
>     #define _Py_UNICODE_IS_TRAIL_SURROGATE
>     #define _Py_UNICODE_JOIN_SURROGATES
> 
> But I also come from a culture that uses more underscores than you guys tend 
> to, as shown in some of the macro names shown below from utf8.h file.  I find
> that most projects use more underscores in uppercase names than Python does. :)

The reasoning behind e.g. "ISSURROGATE" is that those names originate
from and are consistent with the already existing ISLOWER/ISUPPER/ISTITLE
macros which in return stem from the C APIs of the same names
(see unicodeobject.h for reference).

Regarding low/high vs. lead/trail: The Unicode database uses the
terms low/high and we do in Python as well, so let's stick with
those.

What I don't understand is why those macros should be declared
private to Python (with the leading underscore). They are quite
useful for extensions implementing codecs or other transformations
as well.

BTW: I think the other issues mentioned in the discussion are more
important to get right, than the names of those macros.

Marc-Andre Lemburg <report@bugs.python.org> wrote
   on Tue, 16 Aug 2011 12:11:22 -0000: 

> The reasoning behind e.g. "ISSURROGATE" is that those names originate
> from and are consistent with the already existing ISLOWER/ISUPPER/ISTITLE
> macros which in return stem from the C APIs of the same names
> (see unicodeobject.h for reference).

I eventually figured that part out in the larger context.  
Makes sense looked at that way.

> Regarding low/high vs. lead/trail: The Unicode database uses the terms
> low/high and we do in Python as well, so let's stick with those.

Yes, those are their block assignments,  Block=High_Surrogates and Block=Low_Surrogates.
I just thought I should mention that in the time since those were invented (which cannot
be changed), after using them in real code for some years, their lingo seems to have 
evolved away from those initial names and toward lead/trail as less confusing.

> What I don't understand is why those macros should be declared
> private to Python (with the leading underscore). They are quite
> useful for extensions implementing codecs or other transformations
> as well.

I was wondering about that myself.  Beyond there being a lot fewer of those
private macros in the Python *.h files, they also seem to be of rather different
character than the iswhatever() macros:

    $ perl -nle '/^\s*#\s*define\s+(_Py_[\p{Lu}_]+)\b/ and print $1' *.h | sort -dfu | fmt -160
    _Py_ANNOTATE_BARRIER_DESTROY _Py_ANNOTATE_BARRIER_INIT _Py_ANNOTATE_BARRIER_WAIT_AFTER _Py_ANNOTATE_BARRIER_WAIT_BEFORE _Py_ANNOTATE_BENIGN_RACE
    _Py_ANNOTATE_BENIGN_RACE_SIZED _Py_ANNOTATE_BENIGN_RACE_STATIC _Py_ANNOTATE_CONDVAR_LOCK_WAIT _Py_ANNOTATE_CONDVAR_SIGNAL _Py_ANNOTATE_CONDVAR_SIGNAL_ALL
    _Py_ANNOTATE_CONDVAR_WAIT _Py_ANNOTATE_ENABLE_RACE_DETECTION _Py_ANNOTATE_EXPECT_RACE _Py_ANNOTATE_FLUSH_STATE _Py_ANNOTATE_HAPPENS_AFTER
    _Py_ANNOTATE_HAPPENS_BEFORE _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END _Py_ANNOTATE_IGNORE_READS_BEGIN
    _Py_ANNOTATE_IGNORE_READS_END _Py_ANNOTATE_IGNORE_SYNC_BEGIN _Py_ANNOTATE_IGNORE_SYNC_END _Py_ANNOTATE_IGNORE_WRITES_BEGIN _Py_ANNOTATE_IGNORE_WRITES_END
    _Py_ANNOTATE_MUTEX_IS_USED_AS_CONDVAR _Py_ANNOTATE_NEW_MEMORY _Py_ANNOTATE_NO_OP _Py_ANNOTATE_PCQ_CREATE _Py_ANNOTATE_PCQ_DESTROY _Py_ANNOTATE_PCQ_GET
    _Py_ANNOTATE_PCQ_PUT _Py_ANNOTATE_PUBLISH_MEMORY_RANGE _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX _Py_ANNOTATE_RWLOCK_ACQUIRED _Py_ANNOTATE_RWLOCK_CREATE
    _Py_ANNOTATE_RWLOCK_DESTROY _Py_ANNOTATE_RWLOCK_RELEASED _Py_ANNOTATE_SWAP_MEMORY_RANGE _Py_ANNOTATE_THREAD_NAME _Py_ANNOTATE_TRACE_MEMORY
    _Py_ANNOTATE_UNPROTECTED_READ _Py_ANNOTATE_UNPUBLISH_MEMORY_RANGE _Py_AS_GC _Py_CHECK_REFCNT _Py_COUNT_ALLOCS_COMMA _Py_DEC_REFTOTAL _Py_DEC_TPFREES
    _Py_INC_REFTOTAL _Py_INC_TPALLOCS _Py_INC_TPFREES _Py_PARSE_PID _Py_REF_DEBUG_COMMA _Py_SET_EDOM_FOR_NAN

> BTW: I think the other issues mentioned in the discussion are more
> important to get right, than the names of those macros.

Yup.  Just paint it red. :)

--tom

I'm reposting my patch from #12751. I think that it's simpler than belopolsky's patch: it doesn't add public macros in unicodeobject.h and don't add the complex Py_UNICODE_NEXT() macro. My patch only adds private macros in unicodeobject.c to factorize the code.

I don't want to add public macros because with the stable API and with the PEP 393, we are trying to hide the Py_UNICODE type and PyUnicodeObject internals. In belopolsky's patch, only Py_UNICODE_NEXT() is used outside unicodeobject.c.

Copy/paste of the initial message of my issue #12751 (msg142108):
---------------
A lot of code is duplicated in unicodeobject.c to manipulate ("encode/decode") surrogates. Each function has from one to three different implementations. The new decode_ucs4() function adds a new implementation. Attached patch replaces this code by macros.

I think that only the implementations of IS_HIGH_SURROGATE and IS_LOW_SURROGATE are important for speed. ((ch & 0xFFFFFC00UL) == 0xD800) (from decode_ucs4) is *a little bit* faster than (0xD800 <= ch && ch <= 0xDBFF) on my CPU (Atom Z520 @ 1.3 GHz): running test_unicode 4 times takes ~54 sec instead of ~57 sec (-3%).

These 3 macros have to be checked, I wrote the first one:

#define IS_SURROGATE(ch) (((ch) & 0xFFFFF800UL) == 0xD800)
#define IS_HIGH_SURROGATE(ch) (((ch) & 0xFFFFFC00UL) == 0xD800)
#define IS_LOW_SURROGATE(ch) (((ch) & 0xFFFFFC00UL) == 0xDC00)

I added cast to Py_UCS4 in COMBINE_SURROGATES to avoid integer overflow if Py_UNICODE is 16 bits (narrow build). It's maybe useless.

#define COMBINE_SURROGATES(ch1, ch2) \
 (((((Py_UCS4)(ch1) & 0x3FF) << 10) | ((Py_UCS4)(ch2) & 0x3FF)) + 0x10000)

HIGH_SURROGATE and LOW_SURROGATE require that their ordinal argument has been preproceed to fit in [0; 0xFFFF]. I added this requirement in the comment of these macros. It would be better to have only one macro to do the two operations, but because "*p++" (dereference and increment) is usually used, I prefer to avoid one unique macro (I don't like passing *p++ in a macro using its argument more than once).

Or we may add a third macro using HIGH_SURROGATE and LOW_SURROGATE.

I rewrote the main loop of PyUnicode_EncodeUTF16() to avoid an useless test on ch2 on narrow build.

I also added a IS_NONBMP macro just because I prefer macro over hardcoded constants.
---------------

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> I'm reposting my patch from #12751. I think that it's simpler than belopolsky's patch: it doesn't add public macros in unicodeobject.h and don't add the complex Py_UNICODE_NEXT() macro. My patch only adds private macros in unicodeobject.c to factorize the code.
> 
> I don't want to add public macros because with the stable API and with the PEP 393, we are trying to hide the Py_UNICODE type and PyUnicodeObject internals. In belopolsky's patch, only Py_UNICODE_NEXT() is used outside unicodeobject.c.

PEP 393 is an optional feature for extension writers. If they don't
need PEP 393 style stable ABIs and want to use the macros, they
should be able to. I'm therefore -1 on making them private.

Regarding separating adding the various surrogate macros and
the next-macros: I don't see a problem with adding both in
Python 3.3.

Marc-Andre Lemburg wrote:
> 
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
> 
> STINNER Victor wrote:
>>
>> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
>>
>> I'm reposting my patch from #12751. I think that it's simpler than belopolsky's patch: it doesn't add public macros in unicodeobject.h and don't add the complex Py_UNICODE_NEXT() macro. My patch only adds private macros in unicodeobject.c to factorize the code.
>>
>> I don't want to add public macros because with the stable API and with the PEP 393, we are trying to hide the Py_UNICODE type and PyUnicodeObject internals. In belopolsky's patch, only Py_UNICODE_NEXT() is used outside unicodeobject.c.
> 
> PEP 393 is an optional feature for extension writers. If they don't
> need PEP 393 style stable ABIs and want to use the macros, they
> should be able to. I'm therefore -1 on making them private.

Sorry, I mean PEP 384, not PEP 393. Whether PEP 393 will turn out
to be a workable solution has yet to be seen, but that's a
different subject. In any case, Py_UNICODE and access macros
for PyUnicodeObject are in wide-spread use, so trying to hide
them won't work until we reach Py4k.

> Regarding separating adding the various surrogate macros and
> the next-macros: I don't see a problem with adding both in
> Python 3.3.

(oops, msg142225 was for issue #12326)

The code review links point to something weird.  Victor, can you upload your patch for review?

My first impression is that your patch does not accomplish much beyond replacing some literal expressions with macros.  What I wanted to achieve with this issue was to enable writing code without #ifdef Py_UNICODE_WIDE branches.  In your patch these branches seem to still be there and in fact it appears that new code is longer than the old one (I am not sure why, but I see more '+' than '-'s in your patch.)

> The code review links point to something weird.

That's because I posted a patch for another issue. It's the patch set 5, not the patch set 6 :-)

Direct link:
http://bugs.python.org/review/10542/patch/3174/9874

> My first impression is that your patch does not accomplish much beyond
> replacing some literal expressions with macros.

Yes, and it avoids the duplication of some code patterns, as explained in my message. I would like to avoid constants in the code. Some macros are *a little bit* faster than the current code.

> What I wanted to achieve with this issue was to enable writing code
> without #ifdef Py_UNICODE_WIDE branches.

Yes, and I think that it's better to split this issue in two steps:

 1- add macros for the surrogates (test, join, ...)
 2- Py_UNICODE_NEXT()

> In your patch these branches seem to still be there
> and in fact it appears that new code is longer than the old one

Yes, the code adds more lines than it removes. Is it a problem? My goal is to have more readable code (easier to maintain).

As I said in msg142175 I think the Py_UNICODE_IS{HIGH|LOW|}SURROGATE and Py_UNICODE_JOIN_SURROGATES can be committed without trailing _ in 3.3 and with trailing _ in 2.7/3.2.  They should go in unicodeobject.h and be public in 3.3+.

Regarding the name, it would be fine with me to use PyUNICODE_IS_HIGH_SURROGATE.  Other IS* macros don't use spaces, but JOIN_SURROGATES and other proposed macros (e.g. PUT_NEXT/WRITE_NEXT) do.  Also these macros are not related to any existing API like e.g. isalpha.  I think HIGH/LOW are fine, we can mention lead/trail in the doc.

Regarding the implementation, we could use Victor's one if it's faster and it has no other side effects.

Regarding the other macros:
 * _Py_UNICODE_NEXT and _Py_UNICODE_PUT_NEXT are useful, so once we have agreed about the name they can go in.  They can be private in all the 3 branches and made public in 3.4 if they work well;
 * IS_NONBMP doesn't simplify much the code but makes it more readable.  ICU has U_IS_BMP, but in most of the cases we want to check for non-BMP, so if we add this macro it might be ok to check for non-BMP;
 * I'm not sure HIGH_SURROGATE/LOW_SURROGATE are useful with _Py_UNICODE_NEXT.  If they are they should get a better name because the current one is not clear about what they do.


Unless someone disagrees I'll prepare a patch with PyUNICODE_IS_{HIGH_|LOW_|}SURROGATE and Py_UNICODE_JOIN_SURROGATES for unicodeobject.h, using them where necessary, using with Victor implementation and commit it (after a review).

We can think about the rest later.

Le 17/08/2011 07:04, Ezio Melotti a écrit :
> As I said in msg142175 I think the Py_UNICODE_IS{HIGH|LOW|}SURROGATE and Py_UNICODE_JOIN_SURROGATES can be committed without trailing _ in 3.3 and with trailing _ in 2.7/3.2.  They should go in unicodeobject.h

For Python 2.7 and 3.2, I would prefer to not touch a public header, and 
so add the macros in unicodeobject.c.

> and be public in 3.3+.

If you want to make my HIGH_SURROGATE and LOW_SURROGATE macros public, 
they will use to substract 0x10000 themself (whereas my macros require 
the ordinal to be preproceed).

>   * _Py_UNICODE_NEXT and _Py_UNICODE_PUT_NEXT are useful, so once we have agreed about the name they can go in.  They can be private in all the 3 branches and made public in 3.4 if they work well;

Note: I don't think that _Py_UNICODE*NEXT should go into Python 2.7 or 3.2.

>   * IS_NONBMP doesn't simplify much the code but makes it more readable.  ICU has U_IS_BMP, but in most of the cases we want to check for non-BMP, so if we add this macro it might be ok to check for non-BMP;

If you want to make it public, it's better to call it PyUNICODE_IS_BMP() 
(check if the argument is in U+0000-U+FFFF).

>   * I'm not sure HIGH_SURROGATE/LOW_SURROGATE are useful with _Py_UNICODE_NEXT.  If they are they should get a better name because the current one is not clear about what they do.

They are still useful for UTF-16 encoders (to UTF-16-LE/BE and 16-bit 
wchar_t*). We can keep HIGH_SURROGATE and LOW_SURROGATE private in 
unicodeobject.c.

> Unless someone disagrees I'll prepare a patch with PyUNICODE_IS_{HIGH_|LOW_|}SURROGATE and Py_UNICODE_JOIN_SURROGATES for unicodeobject.h, using them where necessary, using with Victor implementation and commit it (after a review).

Cool. I suppose that you mean PyUNICODE_JOIN_SURROGATES (not 
Py_UNICODE_JOIN_SURROGATES). I used the verb "combine", taken from a 
comment in unicodeobject.c. "combine" is maybe better than "join"?

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Le 17/08/2011 07:04, Ezio Melotti a écrit :
>> As I said in msg142175 I think the Py_UNICODE_IS{HIGH|LOW|}SURROGATE and Py_UNICODE_JOIN_SURROGATES can be committed without trailing _ in 3.3 and with trailing _ in 2.7/3.2.  They should go in unicodeobject.h

Ezio used two different naming schemes in his email. Please always
use Py_UNICODE_... or _Py_UNICODE (not PyUNICODE_ or _PyUNICODE_).

> For Python 2.7 and 3.2, I would prefer to not touch a public header, and 
> so add the macros in unicodeobject.c.

Why would you want to touch Python 2.7 at all ?

>> and be public in 3.3+.
> 
> If you want to make my HIGH_SURROGATE and LOW_SURROGATE macros public, 
> they will use to substract 0x10000 themself (whereas my macros require 
> the ordinal to be preproceed).

This can be done by having two definitions of the macros: one set for
UCS2 builds and one for UCS4.

>>   * _Py_UNICODE_NEXT and _Py_UNICODE_PUT_NEXT are useful, so once we have agreed about the name they can go in.  They can be private in all the 3 branches and made public in 3.4 if they work well;
> 
> Note: I don't think that _Py_UNICODE*NEXT should go into Python 2.7 or 3.2.

Certainly not into Python 2.7. Adding macros in patch level releases is
also not such a good idea.

>>   * IS_NONBMP doesn't simplify much the code but makes it more readable.  ICU has U_IS_BMP, but in most of the cases we want to check for non-BMP, so if we add this macro it might be ok to check for non-BMP;
> 
> If you want to make it public, it's better to call it PyUNICODE_IS_BMP() 
> (check if the argument is in U+0000-U+FFFF).

Py_UNICODE_IS_BMP() please.

>>   * I'm not sure HIGH_SURROGATE/LOW_SURROGATE are useful with _Py_UNICODE_NEXT.  If they are they should get a better name because the current one is not clear about what they do.
> 
> They are still useful for UTF-16 encoders (to UTF-16-LE/BE and 16-bit 
> wchar_t*). We can keep HIGH_SURROGATE and LOW_SURROGATE private in 
> unicodeobject.c.
>
>> Unless someone disagrees I'll prepare a patch with PyUNICODE_IS_{HIGH_|LOW_|}SURROGATE and Py_UNICODE_JOIN_SURROGATES for unicodeobject.h, using them where necessary, using with Victor implementation and commit it (after a review).
> 
> Cool. I suppose that you mean PyUNICODE_JOIN_SURROGATES (not 
> Py_UNICODE_JOIN_SURROGATES). I used the verb "combine", taken from a 
> comment in unicodeobject.c. "combine" is maybe better than "join"?

No, Py_UNICODE_... please !

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________
2011-10-04: PyCon DE 2011, Leipzig, Germany                48 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

> For Python 2.7 and 3.2, I would prefer to not touch a public header, 
> and so add the macros in unicodeobject.c.

Is there some reason for this?  I think it's better if we have them in the same place rather than renaming and moving them in another file between 3.2 and 3.3.

> If you want to make my HIGH_SURROGATE and LOW_SURROGATE macros 
> public, they will use to substract 0x10000 themself (whereas my 
> macros require the ordinal to be preproceed).

If they turn out to be useful and we find a clearer name we can even make them public in 3.3, but we'll have to see about that.

> Note: I don't think that _Py_UNICODE*NEXT should go into
> Python 2.7 or 3.2.

If they don't it won't be possible to fix #9200 in those branches (unless we decide that the bug shouldn't be fixed there, but I would rather fix it).

> If you want to make it public, it's better to call it 
> PyUNICODE_IS_BMP() (check if the argument is in U+0000-U+FFFF).

Yes, public APIs will follow the naming conventions.  Not sure if it's better to check if it's a BMP char, or if it's not.

> They are still useful for UTF-16 encoders (to UTF-16-LE/BE and 16-bit 
> wchar_t*). We can keep HIGH_SURROGATE and LOW_SURROGATE private in 
> unicodeobject.c.

What are the naming convention for private macros in the same .c file where they are used?  Shouldn't they get at least a trailing _?

> Unless someone disagrees I'll prepare a patch with PyUNICODE_IS_{HIGH_|LOW_|}SURROGATE and Py_UNICODE_JOIN_SURROGATES for unicodeobject.h, using them where necessary, using with Victor implementation and commit it (after a review).

> Cool. I suppose that you mean PyUNICODE_JOIN_SURROGATES (not 
> Py_UNICODE_JOIN_SURROGATES).

All the other macros use PyUNICODE_*.

> I used the verb "combine", taken from a  comment in unicodeobject.c. 
> "combine" is maybe better than "join"?

I like join, it's clear enough and shorter.

Ah yes, the correct prefix for functions working on Py_UNICODE 
characters/strings is "Py_UNICODE", not "PyUNICODE", sorry.

>> For Python 2.7 and 3.2, I would prefer to not touch a public header,
>> and so add the macros in unicodeobject.c.
>
> Is there some reason for this?

We don't add new features to stable releases.

>> If you want to make my HIGH_SURROGATE and LOW_SURROGATE macros
>> public, they will use to substract 0x10000 themself (whereas my
>> macros require the ordinal to be preproceed).
>
> If they turn out to be useful and we find a clearer name we can even make them public in 3.3, but we'll have to see about that.

I don't think that they are useful outside unicodeobject.c.

>> Note: I don't think that _Py_UNICODE*NEXT should go into
>> Python 2.7 or 3.2.
>
> If they don't it won't be possible to fix #9200 in those branches

I don't think that #9200 is a bug, but more a feature request.

> Not sure if it's better to check if it's a BMP char, or if it's not.

I prefer a shorter name and avoiding double negation: 
!Py_UNICODE_IS_NON_BMP(ch).

> What are the naming convention for private macros in the same .c file where they are used?

Hopefully, there is no convention for private macros :-)

>  Shouldn't they get at least a trailing _?

Nope.

> Ezio used two different naming schemes in his email. Please always
> use Py_UNICODE_... or _Py_UNICODE (not PyUNICODE_ or _PyUNICODE_).

Indeed, that was a typo + copy/paste.  I meant to say Py_UNICODE_* and _Py_UNICODE_*.  Sorry about the confusion.

> Why would you want to touch Python 2.7 at all ?
> [...]
> Certainly not into Python 2.7. Adding macros in patch level releases
> is also not such a good idea.

Because it has the bug and we can fix it (the macros will be private so that we don't add any feature).
Also what about 3.2?  Are you saying that we should fix the bug in 3.2/3.3 only and leave 2.x alone or that you don't want the bug to be fixed in all the bug-fix releases (i.e. 2.7/3.2)?
My idea is to fix the bug in 2.7/3.2/3.3 using the macros, but only make them public in 3.3 so that new features are exposed only in 3.3.

Ezio Melotti wrote:
> 
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
> 
>> Ezio used two different naming schemes in his email. Please always
>> use Py_UNICODE_... or _Py_UNICODE (not PyUNICODE_ or _PyUNICODE_).
> 
> Indeed, that was a typo + copy/paste.  I meant to say Py_UNICODE_* and _Py_UNICODE_*.  Sorry about the confusion.

Good :-)

>> Why would you want to touch Python 2.7 at all ?
>> [...]
>> Certainly not into Python 2.7. Adding macros in patch level releases
>> is also not such a good idea.
> 
> Because it has the bug and we can fix it (the macros will be private so that we don't add any feature).
> Also what about 3.2?  Are you saying that we should fix the bug in 3.2/3.3 only and leave 2.x alone or that you don't want the bug to be fixed in all the bug-fix releases (i.e. 2.7/3.2)?
> My idea is to fix the bug in 2.7/3.2/3.3 using the macros, but only make them public in 3.3 so that new features are exposed only in 3.3.

For bug fixes, you can put the macros straight into unicodeobject.c,
but please leave unicodeobject.h untouched - otherwise people will
mess around with these macros (even if they are private) and users
will start to wonder about linker errors if they use old patch
level releases of Python 2.7/3.2.

Also note that some of these macros change the behavior of Python
- that's good if it fixes a bug (obviously :-)), but bad if it changes
areas that are correctly implemented and then suddenly expose
new behavior.

> For bug fixes, you can put the macros straight into unicodeobject.c,
> but please leave unicodeobject.h untouched - otherwise people will
> mess around with these macros (even if they are private) and users
> will start to wonder about linker errors if they use old patch
> level releases of Python 2.7/3.2.

OK, so in 2.7/3.2 I'll put them in unicodeobject.c, and in 3.3 I'll move them in unicodeobject.c.

Regarding the name, other macros in unicodeobject.c don't have any prefix, so we can do the same (e.g. IS_SURROGATE) for 2.7/3.2 if that's fine.

> Also note that some of these macros change the behavior of Python
> - that's good if it fixes a bug (obviously :-)), but bad if it
> changes areas that are correctly implemented and then suddenly expose
> new behavior.

After this we can fix #9200 and make narrow builds behave correctly (i.e. like wide ones) with non-BMP chars (at least in some places).

Ezio Melotti wrote:
> 
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
> 
>> For bug fixes, you can put the macros straight into unicodeobject.c,
>> but please leave unicodeobject.h untouched - otherwise people will
>> mess around with these macros (even if they are private) and users
>> will start to wonder about linker errors if they use old patch
>> level releases of Python 2.7/3.2.
> 
> OK, so in 2.7/3.2 I'll put them in unicodeobject.c, and in 3.3 I'll move them in unicodeobject.c.
> 
> Regarding the name, other macros in unicodeobject.c don't have any prefix, so we can do the same (e.g. IS_SURROGATE) for 2.7/3.2 if that's fine.

Sure.

>> Also note that some of these macros change the behavior of Python
>> - that's good if it fixes a bug (obviously :-)), but bad if it
>> changes areas that are correctly implemented and then suddenly expose
>> new behavior.
> 
> After this we can fix #9200 and make narrow builds behave correctly (i.e. like wide ones) with non-BMP chars (at least in some places).

Ok.

On 8/17/2011 6:30 AM, Ezio Melotti wrote:
> OK, so in 2.7/3.2 I'll put them in unicodeobject.c, and in 3.3 I'll move them in unicodeobject.c.

I believe the second file should be unicodeobject.h, correct?

Correct.

> Also what about 3.2?  Are you saying that we should fix the bug in
> 3.2/3.3 only and leave 2.x alone or that you don't want the bug to be
> fixed in all the bug-fix releases (i.e. 2.7/3.2)?

Notice that the macros themselves don't fix any bugs. As for the bugs
you apparently want to fix using these macros: they should be considered
on a case-by-case basis. Some of your planned bug fixes may introduce
incompatibilities that rule out fixing them.

> OK, so in 2.7/3.2 I'll put them in unicodeobject.c

It looks like #9200 only needs Py_UNICODE_NEXT, which can be implemented without the other Py_UNICODE_*SURROGATE* macros.

I attached a patch to fix the str.is* methods on #9200 that also includes the macro.

Since they are not public there, I don't see a reason to do 2 separate commits on 2.7/3.2 (one for the feature and one for the fix).

The attached patch adds the following 4 public macros to unicodeobjects.h:
  Py_UNICODE_IS_SURROGATE(ch)
  Py_UNICODE_IS_HIGH_SURROGATE(ch)
  Py_UNICODE_IS_LOW_SURROGATE(ch)
  Py_UNICODE_JOIN_SURROGATES(high, low)
and documents them.

Since _Py_UNICODE_NEXT is still private, I'll commit it later as part as #9200.

Ezio Melotti wrote:
> 
> Ezio Melotti <ezio.melotti@gmail.com> added the comment:
> 
> The attached patch adds the following 4 public macros to unicodeobjects.h:
>   Py_UNICODE_IS_SURROGATE(ch)
>   Py_UNICODE_IS_HIGH_SURROGATE(ch)
>   Py_UNICODE_IS_LOW_SURROGATE(ch)
>   Py_UNICODE_JOIN_SURROGATES(high, low)
> and documents them.
> 
> Since _Py_UNICODE_NEXT is still private, I'll commit it later as part as #9200.

Looks good.

New changeset 77171f993bf2 by Ezio Melotti in branch 'default':
#10542: Add 4 macros to work with surrogates: Py_UNICODE_IS_SURROGATE, Py_UNICODE_IS_HIGH_SURROGATE, Py_UNICODE_IS_LOW_SURROGATE, Py_UNICODE_JOIN_SURROGATES.
http://hg.python.org/cpython/rev/77171f993bf2

The PEP 393 has been accepted and merge into Python 3.3. Python 3.3 doesn't need the Py_UNICODE_NEXT macro anymore. But my macros (unicode_macros.patch) are still useful.

Py_UNICODE_NEXT has been removed from 3.3 but it's still available and used in 2.7/3.2 (even if it's private).  In order to fix #10521 on 2.7/3.2 the _Py_UNICODE_PUT_NEXT macro attached to this patch is required.

Closing now.