The Unicode Technical Committee posted Public Review
Issue #29, describing a bug in the documentation of NFC
and NFKC in the text of UAX #15 Unicode Normalization
Forms. I have examined unicodedata.c in the Python
implementation (2.3.4) and it appears the
implementation of normalization in Python 2.3.4 may
have the bug therein described. Please see the
description of the bug and the textual fix that is
being made to UAX #15, at the URL:
http://www.unicode.org/review/pr-29.html
The bug is in the definition of rule D2, affecting the
characters "blocked" during re-composition.

You may contact me by e-mail, or fill out the
Unicode.org error reporting form if you have any
questions or concerns.

Since Python uses Unicode internally, it may also be
wise to have someone from the Python development
community on the Unicode Consortium's notification list
to receive immediate notifications of public review
issues, bugs, and other announcements affecting
implementation of the standard.

Logged In: YES 
user_id=38388

Thanks for submitting a bug report. The problem does indeed
occur in the Python normalization code:

>>> unicodedata.normalize('NFC', u'\u0B47\u0300\u0B3E')
u'\u0b4b\u0300'

I think the following line in unicodedata.c needs to be changed:

          if (comb1 && comb == comb1) {
              /* Character is blocked. */
              i1++;
              continue;
          }

to

          if (comb && (comb1 == 0 || comb == comb1)) {
              /* Character is blocked. */
              i1++;
              continue;
          }

Martin, what do you think ?

Logged In: YES 
user_id=1146994

Thanks all for quick reply. My initial thoughts regarding a
fix were as below. The relevant piece of code seems to be in
function "nfc_nfkc()" in the file unicodedata.c

>           if (comb1 && comb == comb1) { 
>               /* Character is blocked. */ 
>               i1++; 
>               continue; 
>           } 

That should possibly be changed to: 

>           if (comb1 && (comb <= comb1)) { 
>               /* Character is blocked. */ 
>               i1++; 
>               continue; 
>           } 

because the new spec says "either B is a starter or it has
the same or higher combining class as C".

Logged In: YES 
user_id=21627

Is it true that the most recent interpretation of this PR
suggests that the correction should only apply to Unicode
4.1? If so, I think Python should abstain from adopting the
change right now, and should defer that to the point when
the Unicode 4.1 database is incorporated.

Logged In: YES 
user_id=1146994

Yes. The "current" version of UAX #15 is an annex to Unicode
4.1, which will be coming out very soon. No previous
versions of Unicode have been changed. Previous versions of
UAX #15 apply to previous versions of the standard. The UTC
plans to issue a "corrigendum" for this problem, and the
corrigendum is something that *can* be applied to
implementations of earlier versions of Unicode. In that
case, one would cite the implementation of "Unicode Version
X with Corrigendum Y" as shown on the "Enumerated Versions"
page of the Unicode web site. To follow corrigenda, you may
want to keep tabs on the "Updates and Errata" page on the
Unicode web site. This is likely to be Corrigendum #5. You
could fix the bug when you update Python to Unicode 4.1, or
fix it when the corrigendum comes out. Of course, I would
recommend fixing bugs sooner rather than later, but your
release plans may be such that one path is easier. If it's
going to be a long time before you update to 4.1, you may
want to fix the bug and cite the corrigendum when it comes
out. If you plan to update to 4.1 soon after it comes out,
perhaps fixing the bug with that update is fine.

Logged In: YES 
user_id=21627

When this is fixed, the @Part3 data in the normalization
tests need to be considered as well.

Python 2.6 and probably also 2.5 contains still the line if (comb1 &&
comb == comb1) {...}

The code is the same as described by MAL and we're now on Unicode DB  5.1.

Here is a patch fixing Unicode issue "PR29", I used the testcases 
given in http://www.unicode.org/review/pr-29.html

Commited: r78646 (trunk), r78647 (py3k), r78648 (3.1).

Leave the issue open to remember me that I have to backport to 2.6 (after the 2.6.5 release).

> Commited: r78646 (trunk)

Backport done: r79201 (2.6).