The locale module uses locale alias table derived from X11 locale.alias file for mapping bare locale names without encodings to locale names with encodings. However sometimes glibc default encoding for a locale differs from that used in X11 locale.alias.

Here is full differences table:

             GLibc                 X11 locale.alias

az_az az_AZ.UTF-8 az_AZ.ISO8859-9E
ca_ad ca_AD.ISO8859-15 ca_AD.ISO8859-1
ca_fr ca_FR.ISO8859-15 ca_FR.ISO8859-1
ca_it ca_IT.ISO8859-15 ca_IT.ISO8859-1
cy_gb cy_GB.ISO8859-14 cy_GB.ISO8859-1
en_in en_IN.UTF-8 en_IN.ISO8859-1
et_ee et_EE.ISO8859-1 et_EE.ISO8859-15
fi_fi fi_FI.ISO8859-1 fi_FI.ISO8859-15
gd_gb gd_GB.ISO8859-15 gd_GB.ISO8859-1
hi_in hi_IN.UTF-8 hi_IN.ISCII-DEV
iu_ca iu_CA.UTF-8 iu_CA.NUNACOM-8
iw_il iw_IL.ISO8859-8 he_IL.ISO8859-8
ka_ge ka_GE.GEORGIAN_PS ka_GE.GEORGIAN-ACADEMY
lo_la lo_LA.UTF-8 lo_LA.MULELAO-1
mi_nz mi_NZ.ISO8859-13 mi_NZ.ISO8859-1
nr_za nr_ZA.UTF-8 nr_ZA.ISO8859-1
nso_za nso_ZA.UTF-8 nso_ZA.ISO8859-15
ru_ru ru_RU.ISO8859-5 ru_RU.UTF-8
rw_rw rw_RW.UTF-8 rw_RW.ISO8859-1
sq_al sq_AL.ISO8859-1 sq_AL.ISO8859-2
ss_za ss_ZA.UTF-8 ss_ZA.ISO8859-1
ta_in ta_IN.UTF-8 ta_IN.TSCII-0
tg_tj tg_TJ.KOI8_T tg_TJ.KOI8-C
th_th th_TH.TIS_620 th_TH.ISO8859-11
tn_za tn_ZA.UTF-8 tn_ZA.ISO8859-15
ts_za ts_ZA.UTF-8 ts_ZA.ISO8859-1
tt_ru tt_RU.UTF-8 tt_RU.TATAR-CYR
ur_pk ur_PK.UTF-8 ur_PK.CP1256
uz_uz uz_UZ.ISO8859-1 uz_UZ.UTF-8
uz_uz@cyrillic uz_UZ.UTF-8@cyrillic uz_UZ.UTF-8
vi_vn vi_VN.UTF-8 vi_VN.TCVN
zh_cn zh_CN.GB2312 zh_CN.gb2312
zh_tw zh_TW.BIG5 zh_TW.big5
zh_tw.euctw zh_TW.EUC_TW zh_TW.eucTW

For example with the en_IN encoding:

>>> import locale, _locale
>>> _locale.setlocale(locale.LC_CTYPE)
'en_IN'
>>> locale.getlocale()
('en_IN', 'ISO8859-1')
>>> locale.nl_langinfo(locale.CODESET)
'UTF-8'
>>> locale.setlocale(locale.LC_CTYPE, locale.getlocale())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/locale.py", line 592, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Needed a test for few common locales (en_IN, ru_RU) and maybe for unusual locales (uz_uz, uz_uz@cyrillic).

I would prefer to have a separate issue that updates the aliases table to glibc 2.24.

I agree that it's reasonable to have glibc's aliases override
the X.org ones, but this patch makes some pretty significant changes to Python's default assumptions with respect to default encodings for several locales.

While some changes obviously make sense (e.g. 'ca_AD.ISO8859-1' to 'ca_AD.ISO8859-15'), others are less clear (e.g. 'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14' or 'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T' or several of the moves from ISO encodings to UTF-8). Is there some reference for why glibc chose different values than X.org for these ?

I also don't understand why some "xx.utf-8" locale mappings were removed - I don't think we should remove those, unless they are no lot needed due to some other logic implying these mappings.

Since these are major changes, we need an appropriate warning in the NEWS file (and the "What's New" document), an update of the top comment (under "### Database") to mention that the glibc database takes precedence and where to find it,

'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14'

Looks as just fixing an error. The default West-European ISO8859-1 is changed to Celtic cy_GB.ISO8859-14. This looks better option for Welsh.

'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T'

KOI8-C is not supported by Python, but KOI8-T is supported. I don't know what KOI8-C means, there are several rarely used incompatible encodings with this name.

I also don't understand why some "xx.utf-8" locale mappings were removed - I don't think we should remove those, unless they are no lot needed due to some other logic implying these mappings.

The aliases table is a table of exceptions. Removed entries no longer are exceptional.

On 07.03.2017 18:23, Serhiy Storchaka wrote:

Serhiy Storchaka added the comment:

> 'cy_GB.ISO8859-1' to 'cy_GB.ISO8859-14'

Looks as just fixing an error. The default West-European ISO8859-1 is changed to Celtic cy_GB.ISO8859-14. This looks better option for Welsh.

> 'tg_TJ.KOI8-C' to 'tg_TJ.KOI8-T'

KOI8-C is not supported by Python, but KOI8-T is supported. I don't know what KOI8-C means, there are several rarely used incompatible encodings with this name.

While all this may make sense, I'm missing some more reasoning
behind the differences between X.org and glibc.

This change also looks strange:

-    'ka_ge':                                'ka_GE.GEORGIAN-ACADEMY',
+    'ka_ge':                                'ka_GE.GEORGIAN_PS',
     'ka_ge.georgianacademy':                'ka_GE.GEORGIAN-ACADEMY',
     'ka_ge.georgianps':                     'ka_GE.GEORGIAN-PS',
     'ka_ge.georgianrs':                     'ka_GE.GEORGIAN-ACADEMY',

Why is GEORGIAN_PS written with an underscore whereas the other
mappings use dashes ?

Or this one:

• 'fi_fi': 'fi_FI.ISO8859-15',
  + 'fi_fi': 'fi_FI.ISO8859-1',

Why would a locale switch away from an encoding having
the Euro sign to one without it ?

Or why is this latin variant removed:

• 'nan_tw@latin': 'nan_TW.UTF-8@latin',

Why should Russians switch back to ISO ?

• 'ru_ru': 'ru_RU.UTF-8',
  + 'ru_ru': 'ru_RU.ISO8859-5',

or from ISO to KOI ?

• 'russian': 'ru_RU.ISO8859-5',
  + 'russian': 'ru_RU.KOI8-R',

The more I look at these changes, the more I believe we
should not simply take everything we find in the files
for granted. They obviously both have bugs.

> I also don't understand why some "xx.utf-8" locale mappings were removed - I don't think we should remove those, unless they are no longer needed due to some other logic implying these mappings.

The aliases table is a table of exceptions. Removed entries no longer are exceptional.

It's not a table of exceptions, it's a table mapping commonly
used locale settings to ones which the lib C understands :-)

But regardless, I checked the code and it is already
smart enough to convert lib C incompatible spellings such
as "utf8" to "UTF-8", so these entries can indeed be
removed, but only if the locale is otherwise listed.

In some cases, it's probably better to drop the ".utf8"
to have more generic mappings, e.g.

+ 'bhb_in.utf8': 'bhb_IN.UTF-8',

or

 'de_li.utf8':                           'de_LI.UTF-8',

though I'd expect that mapping to be:

 'de_li':                           'de_LI.ISO8859-1',

as for all other "de" entries.

Why is the X11 locale alias map used at all? It seems like it can only create confusion with libc.

Not all platforms use glibc 2.24 as libc.

Ideally most of entries should even not exist. We should ask libc for the default encoding if it is not included in the locale name. The aliases table should be used only for mapping commonly used but unsupported by libc locales to supported by libc locales.

On 08.03.2017 08:20, Serhiy Storchaka wrote:

Serhiy Storchaka added the comment:

Not all platforms use glibc 2.24 as libc.

True. Many don't even use glibc.

Ideally most of entries should even not exist. We should ask libc for the default encoding if it is not included in the locale name. The aliases table should be used only for mapping commonly used but unsupported by libc locales to supported by libc locales.

I think you have a wrong understanding of what this alias table
is used for: we need it to determine the lib C compatible locale
name without using lib C APIs such as setlocale(), since these are
not thread safe and have side-effects for the whole process.

The alias table is there to avoid having to go to the lib C
to ask it indirectly for more details. Unfortunately, there are
no cross-platform lib C APIs which would allow querying these
details without also changing the local settings of the process.

I know that Python still plays the usual "save current locale,
run setlocale(), revert to previous locale" trick in a couple
of places and this works if Python is the only thread running,
but it doesn't when embedded into other applications.

Regarding the patch: we cannot simply use the output from the
script to set new values. The changes have to be manually
reviewed as well.

E.g. this entry in the table is clearly a typo:

'en_zw.utf8':                           'en_ZS.UTF-8',

(it should read en_ZW.UTF-8)

This entry appears wrong as well:

'eo':                                   'eo_XX.ISO8859-3',

(XX is not a valid country ISO code)

How should we go about this ? Mark all the problems in the PR ?

The problem is that that table can get incorrect result for non-Linux platforms (or for Linux with old glibc).

On 08.03.2017 07:27, Benjamin Peterson wrote:

Why is the X11 locale alias map used at all? It seems like it can only create confusion with libc.

Because it was the only such maintained mapping available at the
time. It's also used for the X.org system, which has a rather strong
focus on user interfaces where locale matter a lot, unlike
the lib C :-)

On 08.03.2017 10:37, Serhiy Storchaka wrote:

The problem is that that table can get incorrect result for non-Linux platforms (or for Linux with old glibc).

Sure, it's a best effort approach.

Also note that on today's systems you often don't have the full set of
locales available anymore - instead these have to either be installed
separately or generated on the target system.

Our locale database works on all these system, regardless of
what's installed or not.

Why was the PR merged while we were still discussing it ?

"eo_XX" is just something that appears in the X11 locale.alias file. My change doesn't add that; it was already there. (for Esperanto, which I suppose explains the "XX")

Most of the changes you identify the glibc aliases taking precedence over the X11 ones. e.g., glibc has "fi_FI ISO-8859-1" while the X11 locale list has "fi_FI.ISO8859-15". That seems correct to me as far as the intent of this change is concerned.

How do you propose to pick and choose what we use from the X11 locale alias list?

On 09.03.2017 08:15, Benjamin Peterson wrote:

"eo_XX" is just something that appears in the X11 locale.alias file. My change doesn't add that; it was already there. (for Esperanto, which I suppose explains the "XX")

Yes, I know. That was an example of a bug in the X.org list.

Most of the changes you identify the glibc aliases taking precedence over the X11 ones. e.g., glibc has "fi_FI ISO-8859-1" while the X11 locale list has "fi_FI.ISO8859-15". That seems correct to me as far as the intent of this change is concerned.

No, it's not correct. ISO-8859-1 is the older version of Latin-1
without the Euro sign. ISO8859-15 adds it.

How do you propose to pick and choose what we use from the X11 locale alias list?

We have to go through the list one by one to check whether
the mapping update makes sense and is correct.

This will be difficult in a few cases where the glibc mapping
switches to UTF-8 from an ISO encoding. We'll have to find
evidence that this change does indeed make sense.

My take on this is that the X.org folks know better than the
glibc folks, since the former have to deal with end users that
rely on the locale settings a lot more than applications
using glibc for getting an initial locale setting right.

Also note that you are parsing the SUPPORTED file from
glibc (in slightly processed form):

https://github.com/bminor/glibc/blob/master/localedata/SUPPORTED

This file does not provide a locale alias mapping as
the routine in makelocalealias.py suggests. Instead it's
a list of locales to install by default:

https://github.com/bminor/glibc/blob/73dfd088936b9237599e4ab737c7ae2ea7d710e1/localedata/Makefile

In glibc you can define both the locale and the encoding separately
when creating a locale using localedef and the file simply provides
the default parameters to pass to this tool.

As such, I don't see how you can derive a default alias
meaning from the file.

It's simply an indication of what glibc would have installed
in case it were installed from source, but that's hardly ever
the case. On today's systems only a bare subset of locales
is installed and more added as necessary, so you rarely have
all the locales defined in SUPPORTED installed on a system.

So the file doesn't even provide a hint at what could
be installed on the system ("locale -a" gives you that list).

Here's the history:

https://github.com/bminor/glibc/commits/master/localedata/SUPPORTED

It's merely a list of additions and removals from the
default set. Nothing more. It does provide a list of
known and supported locales, but no usable or authoritative
encoding information (locales are defined using Unicode, so
the encoding is a parameter and not predefined).

Overall, I believe the file is pretty useless to use as
basis for an alias table providing encoding information.
It may provide some ideas for corrections, but should not
override the X.org one by default.

On the other hand, you have the local.alias master file:

https://cgit.freedesktop.org/xorg/lib/libX11/tree/nls/locale.alias.pre

together with the history of why changes were made and when.
This is an authoritative resource and people are making changes
against it from the user perspective.

I'd suggest to make the override optional in makelocalealias.py
via a command line switch and to use this for manually adding
or fixing X.org entries.

If you absolutely want to parse the glibc file per default as
well, please only let it add new entries, not override existing
ones. As we've seen in the patch, those overrides need to be
carefully reviewed.

Why is the X11 locale alias map used at all? It seems like it can only create confusion with libc.

Originally only the X11 locale alias map was used. The support of the glibc locale alias map was added 2.5 years ago (bpo-20079).

The SUPPORTED file from glibc is used for determining the default encoding for locales that don't include it explicitly. For example en_IN uses UTF-8 rather than ISO8859-1.

On 09.03.2017 11:47, Serhiy Storchaka wrote:

The SUPPORTED file from glibc is used for determining the default encoding for locales that don't include it explicitly. For example en_IN uses UTF-8 rather than ISO8859-1.

No, the glibc locales don't say anything about default encodings
used in a locale:

http://manpages.ubuntu.com/manpages/wily/en/man5/locale.5.html

These encodings are just used for determining the default
set of locale.encoding variants to install on the system,
nothing more:

https://github.com/bminor/glibc/blob/73dfd088936b9237599e4ab737c7ae2ea7d710e1/localedata/Makefile#L204

glibc does have a locale.alias file:

https://github.com/bminor/glibc/blob/73dfd088936b9237599e4ab737c7ae2ea7d710e1/intl/locale.alias

which uses the X.org format, but this is completely out of
date and declared obsolete.

Serhiy: If you believe that there's anything authoritative about
the glibc SUPPORTED file in terms of defining the commonly
used encoding in a locale, please provide references. These
should also clarify why the glibc encoding is the correct one
compared to the X.org mapping.

It doesn't help, trying to interpret things into such build
files. We need a database that is being actively maintained
and has a track record of representing what people actually
use in their locales. The only one I know is the X.org one.

The original issue is bpo-29571. The locale module returned encoding ISO8859-1 for locale en_IN (as in the X11 locale alias map), but glibc uses UTF-8 (as in glibc SUPPORT file).

Do you believe this program should work?

import locale, os
for l in open("/usr/share/i18n/SUPPORTED"):
    alias, encoding = l.strip().split()
    locale.setlocale(locale.LC_ALL, alias)
    try:
        enc = locale.getlocale()[1]
    except ValueError:
        continue # not in table
    normalized = enc.replace("ISO", "ISO-"). \
                     replace("_", "-"). \
                     replace("euc", "EUC-"). \
                     replace("big5", "big5-").upper()
    assert normalized == locale.nl_langinfo(locale.CODESET)

After my change it does—the encoding returned from getlocale() is the one actually being used by glibc. It fails dramatically on earlier versions of Python (for example on the en_IN example from bpo-29571.) I don't understand why Python needs to editorialize whatever choices libc or the system administrator has made.

Is getlocale() expected to return something different from the underlying C locale?

In fact, why have this table at all instead of using nl_langinfo to return the encoding for the current locale?

On 10.03.2017 08:37, Benjamin Peterson wrote:

Do you believe this program should work?

import locale, os
for l in open("/usr/share/i18n/SUPPORTED"):
alias, encoding = l.strip().split()
locale.setlocale(locale.LC_ALL, alias)
try:
enc = locale.getlocale()[1]
except ValueError:
continue # not in table
normalized = enc.replace("ISO", "ISO-"). \
replace("_", "-"). \
replace("euc", "EUC-"). \
replace("big5", "big5-").upper()
assert normalized == locale.nl_langinfo(locale.CODESET)

After my change it does—the encoding returned from getlocale() is the one actually being used by glibc. It fails dramatically on earlier versions of Python (for example on the en_IN example from bpo-29571.) I don't understand why Python needs to editorialize whatever choices libc or the system administrator has made.

Your program essentially tests what alias is configured
on your particular system. It will fail on older systems
(with a different or no version of SUPPORTED), it will fail on
systems that do not have all locales installed, it will
fail on systems that use the X.org aliases table as basis
rather than some list of supported locales of glibc, or
custom alias tables.

What we want in Python is a consistent mapping of aliases to locales
across all (Unix based) Python installations, just like what we
have for encoding aliases and those mappings should be taken
from a support alias database, not a list of default installations
on some glibc version.

Also note that a lot of these discussions are really academic,
since locales should always be specified with encoding.

While Unix gravitates to UTF-8 for all system related things,
users still use other encodings a lot for their daily operations,
as you can see in the X.org aliases file.

This is why defaulting to UTF-8 for locales (as e.g.
is done for many locales in the glibc default installs) is not
a good idea. Locales affect user work products. What's fine for
command line interfacing or piping, is not necessarily for
fine for e.g. documents created by users.

So to answer your question: No, I don't believe that SUPPORTED
has any authority for our purposes and thus don't think that
the program can be considered a valid test case.

The SUPPORTED file can server as extra resource for fixing bugs
in the table, but nothing more.

Is getlocale() expected to return something different from the underlying C locale?

getlocale() will return whatever is currently configured via
setlocale().

Of course, it can return something different from what some glibc
SUPPORTED lists as default installation encoding, if you don't provide
the encoding when using setlocale(), but it will always default
to the same locale and encoding on all platforms where you
run Python.

In fact, why have this table at all instead of using nl_langinfo to return the encoding for the current locale?

The table is meant to normalize locale names and enrich
them with default encodings from a well known database of
such aliases, where necessary. As mentioned above the locale setting
should ideally include the encoding as well, so that any such
guesses are not necessary.

Regarding nl_langinfo():

nl_langinfo() will only work if you have called
setlocale() already, since a process always starts up in
the C locale without this call.

If you don't have a problem with calling setlocale() for
testing the default locale settings (e.g. Python is not
embedded, you don't have other threads running, no
APIs which use locale information called yet, setlocale()
was already called to setup the locale, etc.),
you can use the approach taken by getpreferredencoding(),
which is to temporarily set the locale to the default.

Going forward, I think that the following changes make
sense:

• from ISO8859-1 to ISO8859-15 (the -15 version adds
  the Euro sign)

• casing changes e.g. 'zh_CN.gb2312' to 'zh_CN.GB2312'

• fixes which undo removal of modifiers such as
  'uz_uz@cyrillic' -> 'uz_UZ.UTF-8' to 'uz_UZ.UTF-8@cyrillic'

As for the other changes: please undo them and also
revert the unconditional use of glibc mappings overriding
the X.org ones, as mentioned earlier in the thread.

We can readd some of the modifications later on if there's
evidence that they actually do make sense.

Thanks,

Marc-Andre Lemburg
eGenix.com

I'm feeling there is something wrong with the current locale design. See issues bpo-504219, bpo-10466, bpo-20088, bpo-25191, bpo-29571.

I'm still confused about what getlocale() is supposed to do. Why do we attempt to return an encoding anyway if the underlying setlocale call doesn't return one? Is getlocale() not supposed to a simple wrapper over the C locale? If not, how is one supposed to get the encoding associated with the C locale?

The old alias table code meant that the encoding returned from getlocale() could be related to or completely unrelated to the actual C locale. Misunderstanding this results in issues like bpo-29571.

The main purpose of the alias table is to support normalization and this is used for getdefaultencoding() which was created to be able to determine the default encoding based on what X.org uses as default without doing temporary setlocale() tricks.

Now, normalization also happens when passing a locale value to the underlying setlocale(), mainly to avoid many common bugs due to setlocale() being extremely picky about the locale value. A side effect of this is that normalization will also kick in to add the encoding in case no encoding is given in the parameter.

Note that no normalization is necessary to simply set the configured default locale configured on the system. In such a case, you'd run setlocale('LC_ALL') and get what's configured.

If you run the lib C setlocale() with a locale without encoding, the encoding used by the system entirely on what's configured on the system. The SUPPORTED file only gives a hint at what glibc think it should install per default, but any admin or distributor could change these settings simply by running localedef with some other encoding (charmap in locale speak).

I suppose that we could resolve some of the confusion by adding a parameter to disable this normalization in setlocale().

New changeset df82808 by Benjamin Peterson in branch 'master':
bpo-20087: Revert "make the glibc alias table take precedence over the X11 one (#422)" (#713)
df82808

New changeset 02371e0 by Benjamin Peterson in branch 'master':
make the glibc alias table take precedence over the X11 one (#422)
02371e0

Hi all,

The locale in the latest Ubuntu 18.04 contains en_IL as valid locale, but Python cannot resolve this.
This makes test failure in pandas.
pandas-dev/pandas#20957

en_IL has significant impact because this is English locale and now supported in the latest Ubuntu. Is there any plan to add only en_IL?

(Note that I've already created the PR. ( #6707 ))

(pandas-dev) [pandas] locale -a
C
C.UTF-8
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
ja_JP.utf8
POSIX

Benjamin's patch did two things: 1) made the glibc alias table taking precedence over the X11 one; 2) updated the alias mapping with new glibc. The first part is controversial, but updating the alias mapping with new glibc is made regularly. PR 6708 updates it with glibc 2.27. This adds 39 new aliases and fixes bpo-32781 and bpo-33432.

New changeset cedc9b7 by Serhiy Storchaka in branch 'master':
bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (ПР-6708)
cedc9b7

New changeset 6049bda by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7':
[3.7] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708) (GH-6713)
6049bda

New changeset b1c70d0 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708) (GH-6714)
b1c70d0

New changeset a55ac80 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-20087: Update locale alias mapping with glibc 2.27 supported locales. (GH-6708). (GH-6717)
a55ac80

Thanks, Serhiy.

I believe we can close this old issue.

The discussion was certainly a useful one. I guess we should stop updating the alias table automatically and instead add new aliases or change existing ones based on more research and using the X11 files as well as glibc and other resources to help.