A recent issue with one of my programs has shown that
locale.getdefaultlocale() does not handle correctly a corner case. The
issue URL is this one:

http://bitbucket.org/rg3/youtube-dl/issue/7/

Essentially, some users have LANG set to something like
es_CA.UTF-8@valencia. In that case, locale.getdefaultlocale() returns,
as the encoding, the string "utf_8_valencia", which cannot be used as an
argument to the string encode() function. The obvious correct encoding
in this case is UTF-8.

I have traced the problem and it seems that it could be fixed by the
attached patch. It checks if the encoding, at that point, contains the
'@' symbol and, in that case, removes everything starting at that point,
leaving only "UTF-8".

I am not sure if this patch or a similar one should be applied to other
Python versions. My system has Python 2.5.2 and that's what I have patched.

Explanation as to why I put the code there:

* The simple case, es_CA.UTF-8 goes through that point too and enters
the "if".
* I wanted to remove what goes after the '@' symbol at that point, so it
either needed to be removed before the call to the normalizing function
or inside the normalization.
* As this is not what I would consider a normalization, I put the code
before the function call.

Thanks for your hard work. I hope my patch is valid.

Regards.

I just realized that the "if" I introduced is not really needed.
"encoding = encoding.split('@')[0]" works whether the '@' symbol is
present or not.

I wasn't able to reproduce this by just setting my LC_ALL environment
variable to es_CA.UTF-8@valencia and calling getdefaultlocale.  Can you
provide more complete steps to reproduce?

You are right. The issue is not reproduced with es_CA.UTF-8@valencia but
with ca_ES.UTF-8@valencia. The fact that the first case works makes me
think maybe there's another way to solve the problem. Can you check that?

Further investigation:

The guy who had this issue may be from Valencia, Spain. According to the
manpage for setlocale(3) in my system, the form is usually
language[_territory][.codeset][@modifier]. So, in this case, it would
make sense for the language to be "ca" (Catalan) and territory "ES" (Spain).

My patch may be fine after all. Because, if at that point the @modifier
is still present (I have seen code that removes it before that point),
you'd still want to remove it and keep only the "codeset", which is the
interesting part.

OK, it turns out that this is one of a class of known bugs of long
standing (see issue554676 and issue1080864, for example).  The
recommended solution is to not use locale.getdefaultlocale, but to use
locale.getperferredencoding.  I have confirmed that that works for the
case of ca_ES.UTF-8@valencia in python2.5.

There is at least a doc bug here, since no mention of this
fragility/recommendation is made in the getdefaultlocale documentation.

Using getpreferredencoding seems to be the correct solution to your
problem.  However, the locale.py module contains a number of examples of
modifiers in the locale_alias table.  Presumably this case could be
added, but it is not clear to me what the policy is on that at this
time, so I'm adding Martin to the nosy list looking for some guidance.

Excellent. Thanks for the tip. I'll now proceed to modify my code to use
getpreferredencoding. Still, I think getdefaultlocale should work
because it could be used in other situations, I suppose.

Bumping this as part of a bug scrub at EuroPython. Is this still an issue? Should we fix in docs or in code?

I don't know if the behavior is considered a bug or just undocumented, but under Python 2.7.3 it's still the same. locale.getpreferredencoding() does return UTF-8, but the second element in the tuple locale.getdefaultlocale() is "utf_8_valencia", which is not a valid encoding despite the documentation saying it's supposed to be an encoding name.

From my terminal:

$ python -V
Python 2.7.3

$ LANG=ca_ES.UTF-8@valencia python -c 'import locale; print locale.getpreferredencoding()'
UTF-8

$ LANG=ca_ES.UTF-8@valencia python -c 'import locale; print locale.getdefaultlocale()'
('ca_ES', 'utf_8_valencia')

$ LANG=ca_ES.UTF-8 python -c 'import locale; print locale.getpreferredencoding()'
UTF-8

$ LANG=ca_ES.UTF-8 python -c 'import locale; print locale.getdefaultlocale()'
('ca_ES', 'UTF-8')

The patch is not work for "ca_ES@valencia" locale.

And there are issues for such locales: "ks_in@devanagari", "ks_IN@devanagari.UTF-8", "sd", "sd_IN@devanagari.UTF-8" ("ks_in@devanagari" in locale_alias maps to "ks_IN@devanagari.UTF-8" and "sd" to "sd_IN@devanagari.UTF-8").

Here is yet some inconsistency:

$ LANG=uk_ua.microsoftcp1251 ./python -c "import locale; print(locale.getdefaultlocale())"
('uk_UA', 'CP1251')
$ LANG=uk_ua.microsoft-cp1251 ./python -c "import locale; print(locale.getdefaultlocale())"
('uk_UA', 'microsoft_cp1251')

$ ./python -c "import locale; print(locale.normalize('ka_ge.georgianacademy'))"
ka_GE.GEORGIAN-ACADEMY
$ ./python -c "import locale; print(locale.normalize('ka_GE.GEORGIAN-ACADEMY'))"
ka_GE.georgian_academy

Here is a complex patch for more careful locale parsing.

A related issue (a case which isn't taken into account by Serhiy's patch) is http://bugs.python.org/issue18378

Patch updated. Added tests. The locale_alias mapping updated to be self-consistency (i.e. for every name in locale_alias.values() normalize(name) == name).

It would be great if this could get a review by MAL, since it looks like a non-trivial change.

Also, you have some (commented out) debug prints in there.

On 13.09.2013 15:30, Serhiy Storchaka wrote:
> 
> Serhiy Storchaka added the comment:
> 
> Patch updated. Added tests. The locale_alias mapping updated to be self-consistency (i.e. for every name in locale_alias.values() normalize(name) == name).

Could you elaborate on the alias changes ?

Were those coming from an updated X11 local.alias file ?

If so, I'd suggest to create two patches: one with the alias
updates (which can then also be backported) and one with the
new normalization code (which is a new feature and probably
cannot be backported).

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

> Also, you have some (commented out) debug prints in there.

These debug prints were in old code.

Ah, I see.  I only scanned the patch quickly, obviously.

> Could you elaborate on the alias changes ?
> Were those coming from an updated X11 local.alias file ?

No, they are not from X11 local.alias file. They are a result of the test_locale_alias self-test, I have fixed all failures.

This test can't be backported without rest of changes, because they fix other error, for example processing encodings with hyphen. Without them test_locale_alias will fail even with updated locale_alias. I.e. we can backport either changes to locale_alias without tests, or changes to locale_alias with all changes to parser and tests, or changes to parser and all tests except test_locale_alias.

Current code doesn't work with locales with modifiers and locales with hyphenated encodings.

Here is a patch without changes to locale_alias.

On 13.09.2013 16:34, Serhiy Storchaka wrote:
> 
> Serhiy Storchaka added the comment:
> 
>> Could you elaborate on the alias changes ?
>> Were those coming from an updated X11 local.alias file ?
> 
> No, they are not from X11 local.alias file. They are a result of the test_locale_alias self-test, I have fixed all failures.
> 
> This test can't be backported without rest of changes, because they fix other error, for example processing encodings with hyphen. Without them test_locale_alias will fail even with updated locale_alias. I.e. we can backport either changes to locale_alias without tests, or changes to locale_alias with all changes to parser and tests, or changes to parser and all tests except test_locale_alias.
> 
> Current code doesn't work with locales with modifiers and locales with hyphenated encodings.

Then I don't understand changes such as:

-    'chinese-s':                            'zh_CN.eucCN',
+    'chinese-s':                            'zh_CN.gb2312',

or

-    'sp':                                   'sr_CS.ISO8859-5',
-    'sp_yu':                                'sr_CS.ISO8859-5',
+    'sp':                                   'sr_RS.ISO8859-5',
+    'sp_yu':                                'sr_RS.ISO8859-5',

The .test_locale_alias() checks that the normalize()
function returns the the alias given in the alias table.

If you have to make changes to the alias table that cause the
encoding to or locale to change, something is wrong with
normalize() function.

Note that we are using the X11 locale.alias file as basis for
the mapping, so any such changes need to be found there as
well.

The Tools/i18n/makelocalealias.py script can be used to create
an updated listing.

Please remember that the output of the alias table
is a C runtime locale string. Those do not necessarily
use the same encodings as we do in Python.

Perhaps we should open a separate ticket for the update of the
alias table. I just ran the script on my older dev system
and it returned this list of changes compared to what's in
Python 2.7:

#    added 'ar_in'
#    added 'as_in'
#    added 'be_bg'
#    added 'bo_in'
#    added 'en_dk'
#    added 'hne_in'
#    added 'ks_in'
#    added 'mai_in'
#    added 'ml_in'
#    added 'ne_np'
#    added 'or_in'
#    added 'pa_pk'
#    added 'sd_in'
#    added 'sd_in@devanagari'
#    added 'te_in'
#    updated 'univ' -> 'en_US.utf' to 'en_US.UTF-8'
#    added 'ur_in'
#    added 'zh_sg'

> Then I don't understand changes such as:
>
> -    'chinese-s':                            'zh_CN.eucCN',
> +    'chinese-s':                            'zh_CN.gb2312',
>
> or
>
> -    'sp':                                   'sr_CS.ISO8859-5',
> -    'sp_yu':                                'sr_CS.ISO8859-5',
> +    'sp':                                   'sr_RS.ISO8859-5',
> +    'sp_yu':                                'sr_RS.ISO8859-5',
>
> The .test_locale_alias() checks that the normalize()
> function returns the the alias given in the alias table.
>

It also test normalize(locale_alias[localname]) == locale_alias[localname]
== normalize(localname). I.e. that applying normalize() twice doesn't
change a result.

chinese-s is mapped to zh_CN.eucCN, but eucCN is mapped to gb2312. sp is
mapped to sr_CS.ISO8859-5, but sr_CS is mapped to sr_RS.UTF-8 and then
.ISO8859-5 replaces UTF-8. Of course we can recursive call normalize(), but
it will be more practical just update the mapping.

Ping. There are two duplicate issues opened last month.

Serhiy, in your patch you seem to have special treatment for
the devanagari modifier:

+                # Devanagari modifier placed before encoding.
+                return code, modifier.split('.')[1]

Probably because of 

       'ks_in@devanagari':                     'ks_IN@devanagari.UTF-8',
       'sd':                                   'sd_IN@devanagari.UTF-8',

in the locale_alias dictionary.

But I think these two lines are just wrong, this mistake
is inherited from the locale.alias from X.org where the
python locale_alias comes from.

glibc:

mfabian@ari:~
$ locale -a | grep ^sd
sd_IN
sd_IN.utf8
sd_IN.utf8@devanagari
sd_IN@devanagari
mfabian@ari:~
$ locale -a | grep ^ks
ks_IN
ks_IN.utf8
ks_IN.utf8@devanagari
ks_IN@devanagari
mfabian@ari:~
$ 

The encoding should always be *before* the modifier.

The /usr/share/X11/locale/locale.alias file in Ubuntu 12.04 LTS contains ks_IN@devanagari.UTF-8 and sd_IN@devanagari.UTF-8 entities. While the encoding is expected to be before the modifier, if there are systems with ks_IN@devanagari.UTF-8 or sd_IN@devanagari.UTF-8 locales we should support these weird case.

Serhiy> The /usr/share/X11/locale/locale.alias file in Ubuntu 12.04 LTS
Serhiy> contains ks_IN@devanagari.UTF-8 and sd_IN@devanagari.UTF-8
Serhiy> entities.

Yes, I know, that’s why I wrote that the Python code inherited this mistake
from X.org.

Serhiy> While the encoding is expected to be before the modifier, if
Serhiy> there are systems with ks_IN@devanagari.UTF-8 or
Serhiy> sd_IN@devanagari.UTF-8 locales we should support these weird case.

There are no such systems really, in X.org this is just a mistake.
glibc doesn’t write it like this and it is agains the specification
here:

http://pubs.opengroup.org/onlinepubs/007908799/xbd/envvar.html#tag_002

     [language[_territory][.codeset][@modifier]]

In glibc, sd_IN@devanagari.UTF-8 is an invalid locale name,
only sd_IN.UTF-8@devanagari is valid:

mfabian@ari:~
$ LC_ALL=sd_IN.UTF-8@devanagari locale charmap
UTF-8
mfabian@ari:~
$ LC_ALL=sd_IN@devanagari.UTF-8 locale charmap
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
ANSI_X3.4-1968
mfabian@ari:~
$ 

So I think this should be fixed in X.org.

>> Then I don't understand changes such as:
>>
>> -    'chinese-s':                            'zh_CN.eucCN',
>> +    'chinese-s':                            'zh_CN.gb2312',
>>
>> or
>>
>> -    'sp':                                   'sr_CS.ISO8859-5',
>> -    'sp_yu':                                'sr_CS.ISO8859-5',
>> +    'sp':                                   'sr_RS.ISO8859-5',
>> +    'sp_yu':                                'sr_RS.ISO8859-5',
>>
>> The .test_locale_alias() checks that the normalize()
>> function returns the the alias given in the alias table.

As mentioned earlier, the purpose of the alias table is to map
*normalized* local names to the C runtime string, which in some
cases use different encoding names that we use in Python.

> It also test normalize(locale_alias[localname]) == locale_alias[localname]
> == normalize(localname). I.e. that applying normalize() twice doesn't
> change a result.

That's not intended. The normalize() function is supposed to
prepare the locale for the lookup. It's not supposed to be applied
to the looked up value.

About the devangari special case:
This has been in the X11 file for ages and still is ...
http://cgit.freedesktop.org/xorg/lib/libX11/tree/nls/locale.alias.pre

> That's not intended. The normalize() function is supposed to
prepare the locale for the lookup. It's not supposed to be applied
to the looked up value.

Last patch doesn't contain this part of tests.

> There are no such systems really, in X.org this is just a mistake.
glibc doesn’t write it like this and it is agains the specification
here:

While normalize can return sd_IN@devanagari.UTF-8, _parse_localename() should be able correctly parse it. Removing sd_IN@devanagari.UTF-8 from alias table is another issue.

On 11.11.2013 20:21, Serhiy Storchaka wrote:
> 
>> That's not intended. The normalize() function is supposed to
>> prepare the locale for the lookup. It's not supposed to be applied
>> to the looked up value.
> 
> Last patch doesn't contain this part of tests.

Thanks.

Serhiy> While normalize can return sd_IN@devanagari.UTF-8, _parse_localename()
Serhiy> should be able correctly parse it.

But if normalize returns sd_IN@devanagari.UTF-8, isn’t that quite
useless because it is a locale name which does not actually work
in glibc?

Serhiy> Removing sd_IN@devanagari.UTF-8 from alias table is another issue.

Yes. I think it should be fixed in the alias table as well.

Marc-Andre, do you have comments or objections?

On 18.12.2013 22:57, Serhiy Storchaka wrote:
> 
> Marc-Andre, do you have comments or objections?

Your last patch looks fine, but I don't have time to test it.

Regarding the broken *devanagari* entries in the alias table:
I think we should remove or correct those.

The purpose of normalize() is to return a valid libc locale
identifier and if the values in the alias table are clearly
wrong and don't work with libc, there's little point in keeping
them, even if the X11 file still lists them with the wrong
notation.

If we can fix them so that they do work with libc, let's do that.
If we can't let's remove them. In both cases, please add a comment
mentioning the case and why things were changed/removed.

Hope that helps. Thanks.

New changeset 3d805bee06e2 by Serhiy Storchaka in branch '2.7':
Issue #5815: Fixed support for locales with modifiers.  Fixed support for
http://hg.python.org/cpython/rev/3d805bee06e2

New changeset 28883e89f335 by Serhiy Storchaka in branch '3.3':
Issue #5815: Fixed support for locales with modifiers.  Fixed support for
http://hg.python.org/cpython/rev/28883e89f335

New changeset b50971bccfc3 by Serhiy Storchaka in branch 'default':
Issue #5815: Fixed support for locales with modifiers.  Fixed support for
http://hg.python.org/cpython/rev/b50971bccfc3

Committed without devanagari special case and tests.

For devanagari modifier opened new issue20027.

Buildbot failure:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%203.3/builds/1314/steps/test/logs/stdio

======================================================================
ERROR: test_locale_alias (test.test_locale.NormalizeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildslave/3.3.murray-gentoo-wide/build/Lib/test/test_locale.py", line 374, in test_locale_alias
    with self.subTest(locale=(localename, alias)):
AttributeError: 'NormalizeTest' object has no attribute 'subTest'

New changeset e0675408f4af by Serhiy Storchaka in branch '2.7':
Don't use sebTest() in tests for issue #5815.
http://hg.python.org/cpython/rev/e0675408f4af

New changeset ed16f6695638 by Serhiy Storchaka in branch '3.3':
Don't use sebTest() in tests for issue #5815.
http://hg.python.org/cpython/rev/ed16f6695638

Oh, thanks Victor.