Python has no standard support for the Unicode Collation Library as explained in UTS #10.  This is request that UCA library be added to the standard Python distribution.

Collation underlies virtually everything we do with text, not just sorting but any sort of comparison. Furthermore, the UCA is tailorable for locales in a portable way that does not require dodgy vendor support. It is a very important step in making Python suitable for full Unicode text processing.

Sounds like a fair feature request for Python 3.3, as long as the intention is that users must import some module from the standard library and use functions defined in that module.  The operations and methods defined for str instances (e.g. ==, <, etc.) should not change their behavior.

Is there an existing 3rd party library that we could adopt (even if it isn't perfect yet)?

> Sounds like a fair feature request for Python 3.3, as long as the
> intention is that users must import some module from the standard
> library and use functions defined in that module.  The operations and
> methods defined for str instances (e.g. ==, <, etc.) should not change
> their behavior.

> Is there an existing 3rd party library that we could adopt (even if it isn't perfect yet)?

I *think* you could use ICU's.  

I'm pretty sure the Parrot people use ICU libraries.

--tom

I know I sound like NIH, but I'm always reluctant to add a big 3rd party lib like ICU to the permanent dependencies of all future Python distros.  If people want to use ICU they already can.  OTOH I don't have a better idea. :-(

I would like to be involved in the design of the API for a UCA module and its routines for loading Unicode Collation Element Tables (not making the mistake of using global state like the locale module does).

Raymond Hettinger <raymond.hettinger@gmail.com> added the comment:

> I would like to be involved in the design of the API for a UCA module
> and its routines for loading Unicode Collation Element Tables (not
> making the mistake of using global state like the locale module does).

Is this the problem where a locale is global to a process (or thread)?

The way I'm used to using the UCA module in Perl, that's never a problem,
because it's completely object-oriented.  There's no global state.  You 
instantiate a collator object with all the state it needs, like

    collation_level
    upper_before_lower
    backwards_levels
    normalization
    override_CJK
    override_Hangul
    katakana_before_hiragana
    variable
    locale
    preprocess

And then you use that object for all your collation needs, including
not just sorting but also string comparison and even searches.

For example, you could instantiate a first collator object with its level
set to one, meaning just compare base alphanumerics not diacritics or case
or nonletters, and a second with the defaults so that it uses all four
levels or a different normalization.  I have on occasion had more than one
collator object around at once each with its own locale, like if I want to
compare different locales' comparisons.

--tom

I should probably mention the importance in the design of a UCA module of
being able to specify which UCA version number you want it to behave like
in case you plan to override some of the DUCET entries.  That way if you
run under a later UCA with different DUCET weights, your own tailorings will
still make sense.  If you don't do this, your collation tailorings can break 
in a new release of the UCA.

--tom

Guido van Rossum <report@bugs.python.org> wrote
   on Fri, 26 Aug 2011 21:55:03 -0000: 

> I know I sound like NIH, but I'm always reluctant to add a big 3rd
> party lib like ICU to the permanent dependencies of all future Python
> distros.  If people want to use ICU they already can.  OTOH I don't
> have a better idea. :-(

I know exactly what you mean.  I would not want to push that on anyone,
being dependent on a gigantic 3rd-party module.  I just tried to answer
the question.  The only two full UCA implementations I know of are ICU's
and Perl's, which does not use ICU (since we're UTF-8, etc).

I just wish Python had Unicode collation, is all.

--tom

PS: (I haven't had good luck the ICU bindings in 3.2.)

Remember, sorting standard Persian(Farsi) characters does not work properly with current implementation of Python 3.x
As the result, python is probably unable to sort properly in some other languages.

Here is correct order of alphabet in Persian:

 "آ",    "ا",    "ب",    "پ",    "ت",    "ث",    "ج",    "چ",    "ح",    "خ",    "د",    "ذ",    "ص",    "ض",    "ط",    "ظ",    "ع",    "غ",    "ف",    "ق",    "ک",    "گ",    "ك",    "ل",    "م",    "ن",    "و",    "ه",    "ی",    "ي",

After sorting using sorted():

آ, ا, ب, ت, ث, ج, ح, خ, د, ذ, ص, ض, ط, ظ, ع, غ, ف, ق, ك, ل, م, ن, ه, و, ي, پ, چ, ک, گ, ی,

Isn’t this done by the system? It feels like barking at the wrong tree.

No, this is not an OS dependent issue. Python does not use Unicode collation(uses utf-8) for sorting.