This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Some locales are unsupported
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asmodai, eric.araujo, lemburg, loewis, pitrou, ubuntujenkins
Priority: normal Keywords:

Created on 2010-04-11 21:20 by ubuntujenkins, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (20)
msg102891 - (view) Author: Luke Jennings (ubuntujenkins) Date: 2010-04-11 21:20
In the locale module there are some locales that are not supported these the ones that I am aware of are nl_AW, sr_RS sr_ME. This information was due to a project that captures screenshots in different languages and we have to retrieve the language code. Related to the origin of the bug https://bugs.edge.launchpad.net/quickshot/+bug/554861 .

If any more information is required please let me know.
msg102892 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-04-11 21:38
Hello

Not a local expert here, but since this module relies on the underlying libc locale support. Do other programs work correctly with this locale?

Apart from that, your program needs to catch and handle exceptions anyway.

Martin, I’m making you nosy, since you’re listed as locale area expect in the maintainers file. Hope it’s okay to do so.

Regards
msg102893 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-04-11 21:43
Sorry for sending this too fast. The beginning should read:

Not a locale expert here, but since this module relies on the underlying libc locale support, the bug is not Python-specific.
msg102894 - (view) Author: Luke Jennings (ubuntujenkins) Date: 2010-04-11 21:47
Other programs do work with the local. 

I am working on dealing with exceptions, I am rather new to programing and thought it would also be good to try and get this fixed in the original module.
msg102895 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-04-11 21:49
What I’m saying is that the original module is not a bug. I’ll let 
Martin confirm or infirm :)

Regards
msg102896 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 21:52
This doesn't look like a bug. You must install a locale before it gets available to Python.
Furthermore, according to the original traceback ( http://launchpadlibrarian.net/43012745/quickshot-latin.log ), the locale module isn't even involved.
msg102897 - (view) Author: Luke Jennings (ubuntujenkins) Date: 2010-04-11 21:58
Sorry for the confusion but that bug report has two very similar problems to it if you look at https://bugs.edge.launchpad.net/quickshot/+bug/554861/comments/7
and https://bugs.edge.launchpad.net/quickshot/+bug/554861/comments/6 .

I think that is calling the locale module. I apologize if i have put this down to the wrong thing I am new to python.
msg102900 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 22:02
Actually, my diagnosis was wrong. The locale module contains a list of locale aliases and some of the Serbian ones are missing. A recent locale.alias file from X11 is more complete.
msg102901 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-11 22:13
getdefaultlocale is inherently unmaintainable, and shouldn't be used by applications. I wish it was removed from Python (but unfortunately, too many people got tricked into believing that it does something useful).

That said, if anybody feels like updating the tables from the various data sources, please go ahead.
msg102902 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-04-11 22:15
Can we tell about getdefaultlocale’s uselessness in the docs?
msg102903 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-11 22:23
> Can we tell about getdefaultlocale’s uselessness in the docs?

I haven't quite understood what people want to use that function for. If
we knew the typical use cases, we could make recommendations what they
should use instead.

One use case is to find out the encoding of the locale, in order to
encode files. People should use locale.getpreferredencoding instead, or
trust that the io module will find out on its own.

However, these cases seem to be different.
msg102904 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 22:27
> However, these cases seem to be different.

I think it can be used to find out the current language setting of the
user, and trigger a third-party translation system (other than e.g.
gettext).
msg102905 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 22:27
When the table is updated in trunk, can it be backported to 2.6?
It does change a couple of mappings (example here is from sr_CS.* to sr_RS.*).
msg102909 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-11 23:04
> When the table is updated in trunk, can it be backported to 2.6?

With the changes to the encodings for some of the locales (e.g. 'ru'), I
would advise against such a backport.

This also demonstrates one fundamental flaw of the approach: even if X11
decides that 'ru' should be using "UTF-8", some local installations will
certainly differ. Likewise when they decided that "ru" uses iso-8859-5.
msg102910 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 23:08
Ok, the fix (updated tables) has been committed in trunk (r79977) and py3k (r79978). I suppose Ubuntu can backport the patch if they want their Python to be in sync with their own locales.
(even better, distributions should include the regeneration of locale aliases from their own locale.alias in their build scripts)
msg102923 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-12 08:19
Martin v. Löwis wrote:
> 
> Martin v. Löwis <martin@v.loewis.de> added the comment:
> 
> getdefaultlocale is inherently unmaintainable, and shouldn't be used by applications. I wish it was removed from Python (but unfortunately, too many people got tricked into believing that it does something useful).

I disagree on that.

getdefaultlocale() provides a way to access the default locale
(and encoding) on a platform without requiring a call to
setlocale(LC_ALL, "") which would affect the currently running
application and is not thread-safe (just like most other locale
C APIs in the C lib).

getpreferredencoding() just provides encoding information. It doesn't
provide locale information, so I don't see how that could be used
as replacement for getdefaultlocale().

The table needed for normalizing the various locale strings used in
the LANG environment variable (and others) needs to be updated every
now and then, but that's natural for a resource that changes based
on what people use locales for in real life.

> That said, if anybody feels like updating the tables from the various data sources, please go ahead.

I last updated the table in 2008. Will do that again this week.
msg102926 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-12 08:55
M.-A. Lemburg wrote:
>> That said, if anybody feels like updating the tables from the various data sources, please go ahead.
> 
> I last updated the table in 2008. Will do that again this week.

I saw that Antoine already did this for me: thanks, Antoine !

Could you tell me which X11 version you used ? I've modified the
generation script to also output the added aliases and would like
to list them as well.
msg102942 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-12 12:48
> Could you tell me which X11 version you used ? I've modified the
> generation script to also output the added aliases and would like
> to list them as well.

It's the file supplied on my Mandriva system, from
libx11-1.2.2-2mdv2010.0.src.rpm
msg102944 - (view) Author: Jeroen Ruigrok van der Werven (asmodai) * (Python committer) Date: 2010-04-12 13:05
That nl_AW is an interesting one. Aruba is part of the Kingdom of the Netherlands, so in essence it should follow nl_NL. From what I can see that locale seems to be limited to (certain) Linux systems. I'll raise the issue with the CLDR guys as well, since I wonder if this warrants its own locale or not (some monetary and number formatting).

But I think the outcome will be to add a nl_AW as well in similar vein as to what bo_CN does.
msg102969 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-12 20:09
> getdefaultlocale() provides a way to access the default locale
> (and encoding) on a platform without requiring a call to
> setlocale(LC_ALL, "")

That's what it's meant to do, but this is not what it actually does.
In fact, there is no way of determining the locale's encoding without
the help o fthe C library. As for "default locale": this is a flawed
notion in itself. If I use one locale for messages, and the other for
date representation - which one should be the "default" locale?
Also, "default" in what respect?

> getpreferredencoding() just provides encoding information. It doesn't
> provide locale information, so I don't see how that could be used
> as replacement for getdefaultlocale().

Many applications use getdefaultlocale()[1], i.e. ignore the locale part
of the result. They can easily be rewritten to use getpreferredencoding.

> The table needed for normalizing the various locale strings used in
> the LANG environment variable (and others) needs to be updated every
> now and then, but that's natural for a resource that changes based
> on what people use locales for in real life.

However, it also changes based on local administrator preferences -
something that the implementation does not take into account at all.
It may also depend on OS vendor preferences, which is also not
considered in the implementation.

IOW, it returns bogus results.
History
Date User Action Args
2022-04-11 14:56:59adminsetgithub: 52621
2010-04-12 20:09:24loewissetmessages: + msg102969
2010-04-12 13:05:28asmodaisetnosy: + asmodai
messages: + msg102944
2010-04-12 12:48:34pitrousetmessages: + msg102942
2010-04-12 08:55:27lemburgsetmessages: + msg102926
2010-04-12 08:19:31lemburgsetmessages: + msg102923
2010-04-11 23:08:36pitrousetstatus: open -> closed
resolution: fixed
messages: + msg102910

versions: - Python 2.6, Python 3.1
2010-04-11 23:04:23loewissetmessages: + msg102909
2010-04-11 22:27:57pitrousetmessages: + msg102905
2010-04-11 22:27:28pitrousetmessages: + msg102904
2010-04-11 22:23:50loewissetmessages: + msg102903
2010-04-11 22:15:29eric.araujosetmessages: + msg102902
2010-04-11 22:13:07loewissetmessages: + msg102901
2010-04-11 22:02:14pitrousetpriority: normal
versions: + Python 3.1, Python 2.7, Python 3.2
nosy: + lemburg

messages: + msg102900

resolution: not a bug -> (no value)
2010-04-11 21:58:06ubuntujenkinssetstatus: pending -> open

messages: + msg102897
2010-04-11 21:52:25pitrousetstatus: open -> pending

nosy: + pitrou
messages: + msg102896

resolution: not a bug
2010-04-11 21:49:09eric.araujosetmessages: + msg102895
2010-04-11 21:47:52ubuntujenkinssetmessages: + msg102894
2010-04-11 21:43:11eric.araujosetmessages: + msg102893
2010-04-11 21:38:36eric.araujosetnosy: + loewis, eric.araujo
messages: + msg102892
components: + Library (Lib), - Extension Modules
2010-04-11 21:20:23ubuntujenkinscreate