getlocale fails with an exception when the string returned by _setlocale is already an RFC 1766 language tag.

Example:

Python 3.10.0a4 (tags/v3.10.0a4:445f7f5, Jan  4 2021, 19:55:53) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'en-US')
'en-US'
>>> locale.getlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\flonidan\env\Python310\lib\locale.py", line 593, in getlocale
    return _parse_localename(localename)
  File "C:\flonidan\env\Python310\lib\locale.py", line 501, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: en-US

Expected result:
  ('en-US', None)

See https://github.com/wxWidgets/Phoenix/issues/1637 for an example of the ensuing problems.  wx.Locale calls C setlocale directly, but, as far as I can see, correctly, using dashes in the language code which is consistent with not only RFC 1766 but also the examples in Microsoft's documentation (https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-160).  CPython seems to assume underscored names such as 'en_US' instead, as shown by getdefaultlocale inserting an underscore.

I discovered that this can happen with underscores as well:

Python 3.8.7 (tags/v3.8.7:6503f05, Dec 21 2020, 17:59:51) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'en_DE')
'en_DE'
>>> locale.getlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\flonidan\env\Python38-64\lib\locale.py", line 591, in getlocale
    return _parse_localename(localename)
  File "C:\flonidan\env\Python38-64\lib\locale.py", line 499, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: en_DE

locale.setlocale does validate input - if you write nonsense in the second argument then you get an exception, "locale.Error: unsupported locale setting".  So I'm guessing the en_DE locale is actually being set here, and the problem is solely about getlocale making unfounded assumptions about the format.

Same thing happens in 3.10.0a4.

On 17.02.2021 10:55, Anders Munch wrote:
>>>> import locale
>>>> locale.setlocale(locale.LC_ALL, 'en_DE')
> 'en_DE'
>>>> locale.getlocale()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "C:\flonidan\env\Python38-64\lib\locale.py", line 591, in getlocale
>     return _parse_localename(localename)
>   File "C:\flonidan\env\Python38-64\lib\locale.py", line 499, in _parse_localename
>     raise ValueError('unknown locale: %s' % localename)
> ValueError: unknown locale: en_DE

The locale module does not know this encoding, so cannot
guess the encoding. Since getlocale() returns the language
code and encoding, this fails.

If you add the encoding, you should be fine:

>>> locale.setlocale(locale.LC_ALL, 'en_DE.UTF-8')
'en_DE.UTF-8'
>>> locale.getlocale()
('en_DE', 'UTF-8')

getlocale is documented to return None for the encoding if no encoding can be determined.  There's no need to guess.

I can't change the locale.setlocale call, because where I'm actually having the problem, I'm not even calling locale.setlocale: wxWidgets is calling C setlocale, that's where it comes from.

wxWidgets aside, it doesn't seem right that getlocale fails when setlocale succeeded.

On 17.02.2021 12:36, Anders Munch wrote:
> getlocale is documented to return None for the encoding if no encoding can be determined.  There's no need to guess.

Well, not quite... the documentation says that None can be returned,
not that it will return None in such cases.

What happens is that the value returned by the C lib's setlocale()
is normalized and parsed. If it doesn't include an encoding,
the table locale_alias is used to provide an encoding. If this
table does not include the locale, you get the ValueError.

We could enhance this to return None for the encoding instead
of raising an exception, but would this really help ?

Alternatively, we could add "en_DE" to the alias table and set
a default encoding to use. I assume this would have to be
ISO8859-15. OTOH, Windows supports UTF-8 for all locales in
more recent Windows versions, so perhaps we should use UTF-8
instead.

BTW: What is wxWidgets doing with the returned values ?

Resources:
- setlocale() in Visual C++ on Windows:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-160
- locale name formats supported by Windows CRT:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/locale-names-languages-and-country-region-strings?view=msvc-160
- Language locale tags known by Windows:

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c
- UTF8 support in Windows CRT:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale?view=msvc-160#utf-8-support

> BTW: What is wxWidgets doing with the returned values ?

wxWidgets doesn't call getlocale, it's a C++ library (wrapped by wxPython) that uses C setlocale.

What does use getlocale is time.strptime and datetime.datetime.strptime, so when getlocale fails, strptime fails.

> We could enhance this to return None for the encoding instead
> of raising an exception, but would this really help ?

Very much so.

Frankly, I don't get the impression that the current locale preferred encoding is used for *anything*.  Other than possibly having a role in implementing getpreferredencoding.

> Alternatively, we could add "en_DE" to the alias table and set
> a default encoding to use. 

Where would you get a complete list of all the new aliases that would need be to be added?

As the "en_DE" example shows, you'd need all combinations of languages and regions.  That's going to be a very long list.

> What does use getlocale is time.strptime and datetime.datetime.strptime

calendar.LocaleTextCalendar also uses getlocale() and getdefaultlocale(). The result from getdefaultlocale() cannot be set via setlocale() in Windows, which also breaks resetlocale().

    >>> locale.getdefaultlocale()
    ('en_GB', 'cp1252')
    >>> locale._build_localename(locale.getdefaultlocale())
    'en_GB.cp1252'

This is doubly invalid. It's a BCP-47 locale name with an encoding that's not UTF-8. As of Windows 10 v1803, UTF-8 is supported in ucrt locales, and for BCP-47 locale names, only UTF-8 is allowed to be set explicitly. (If not set to UTF-8 explicitly, then ucrt implicitly uses the given locale's legacy ANSI code page.) Even if something other than UTF-8 were allowed, ucrt will not accept a code page with a "cp" prefix.

Default LocaleTextCalendar case:

    >>> c = calendar.LocaleTextCalendar()
    >>> try: c.formatweekday(1, 10)
    ... except Exception as e: print(e)
    ...
    unsupported locale setting

It works as long as setting the locale succeeds and it can restore the original locale. For example, setting "es" (Spanish):

    >>> c = calendar.LocaleTextCalendar(locale='es')
    >>> locale.setlocale(locale.LC_TIME, 'C')
    'C'
    >>> c.formatweekday(1, 10)
    '  martes  '

Now try with the current locale as a BCP-47 locale name.

    >>> locale.setlocale(locale.LC_TIME, 'en')
    'en'

The parsed getlocale() result is, like with the default locale, doubly invalid. It's a BCP-47 locale name with an encoding that's not UTF-8, and the 'ISO8859-1' codeset is meaningless to ucrt.

    >>> locale.getlocale(locale.LC_TIME)
    ('en_US', 'ISO8859-1')

So restoring the locale fails:

    >>> try: c.formatweekday(1, 10)
    ... except Exception as e: print(e)
    ...
    unsupported locale setting

and it's still set to "es":

    >>> locale.setlocale(locale.LC_TIME)
    'es'

Is this the same as the other issue where get_locale is normalising the result according to some particular glibc logic and isn't at all portable?

> Is this the same as the other issue where get_locale is normalising 
> the result according to some particular glibc logic and isn't at 
> all portable? 

If I understand Anders' immediate problem correctly, I think it can be addressed by using setlocale() to save and restore the current locale in the calendar and _strptime modules. This requires no changes to the locale module, let alone the complete rewrite that's required to make getlocale(), normalize(), _parse_localename(), and _build_localename() work reliably in Windows.

It's not just a Windows problem. For example, getlocale() doesn't return the POSIX locale modifier in a 3-tuple (language, encoding, modifier). So it can't be used to restore a locale for which the modifier is mandatory.

The following example in Linux uses Serbian, a language that's customarily written with both the Cyrillic and Latin alphabets (i.e. BCP 47 / RFC 5646 language tags "sr-Cyrl-RS" and "sr-Latn-RS"). The Latin-based Unix locale name uses a "latin" modifier.

Say the process is currently using the Latin-based locale, but I need the name of a weekday in Cyrillic:

    >>> locale.setlocale(locale.LC_TIME, 'sr_RS.UTF-8@latin')
    'sr_RS.UTF-8@latin'
    >>> c = calendar.LocaleTextCalendar(locale='sr_RS.UTF-8')
    >>> c.formatweekday(1, 10)
    '  уторак  '

LocaleTextCalendar() temporarily sets LC_TIME to the given locale and then restores the previous locale. But this is based on the getlocale() result, which omits the "latin" modifier. So now my current LC_TIME locale has changed to Cyrillic:

    >>> locale.setlocale(locale.LC_TIME)
    'sr_RS.UTF-8'

A related problem with modifiers affects getdefaultlocale(). For example, in Linux:

    $ LC_ALL=sr_RS.UTF-8@latin python -q
    >>> import locale, calendar

In this case, the LC_ALL environment variable specifies the Latin-based locale, but getdefaultlocale() omits this important detail:

    >>> locale.getdefaultlocale()
    ('sr_RS', 'UTF-8')

Based on the default locale set in the LC_ALL environment variable, the following is supposed to return the Latin name "utorak", not the Cyrillic name "уторак":

    >>> c = calendar.LocaleTextCalendar()
    >>> c.formatweekday(1, 10)
    '  уторак  '

If I make it call setlocale(LC_TIME, '') instead of getdefaultlocale(), I get the right result:

    >>> c = calendar.LocaleTextCalendar(locale='')
    >>> c.formatweekday(1, 10)
    '  utorak  '

Thus an empty string should be the default locale value in LocaleTextCalendar(), instead of getdefaultlocale().

On 18.02.2021 10:36, Eryk Sun wrote:
> It's not just a Windows problem. For example, getlocale() doesn't return the POSIX locale modifier in a 3-tuple (language, encoding, modifier). So it can't be used to restore a locale for which the modifier is mandatory.

The APIs were written at a time where locale modifiers simply did
not exist. Support could be added via a special locale tuple return
object, which looks like 2-tuple, but comes with extra attributes
to store the modifier, in order to maintain backwards compatibility
-- as is done in several other places as well (e.g. the codec
registry).

This would indeed require a larger rewrite of the logic.

To Steve's question: The locale aliasing logic is based
on the X11 locale aliasing system. It was later extended to
also add glibc aliases. This is portable across many Unix based
systems, but support for Windows is only partial, due to the
completely different approach Windows' CRT took to locales.

Even on most Unix systems, the locale information did not include
encoding information, which we needed to help applications
with Unicode I/O encoding support, so this had to be enriched and
because X11 was the defacto standard on Unix at the time, its locale
aliasing table was used for this.

At the time Fredrik Lundh added some Windows support via
locale codes.

On 17.02.2021 15:02, Anders Munch wrote:
>> BTW: What is wxWidgets doing with the returned values ?
> 
> wxWidgets doesn't call getlocale, it's a C++ library (wrapped by wxPython) that uses C setlocale.
> 
> What does use getlocale is time.strptime and datetime.datetime.strptime, so when getlocale fails, strptime fails.

Would they work with getlocale() returning None for the encoding ?

>> We could enhance this to return None for the encoding instead
>> of raising an exception, but would this really help ?
> 
> Very much so.
> 
> Frankly, I don't get the impression that the current locale preferred encoding is used for *anything*.  Other than possibly having a role in implementing getpreferredencoding.

The logic for getdefaultencoding() predates getpreferredencoding().
I had added it because calling setlocale() just to figure out the default
encoding and then resetting it to the original setting is dangerous
in a multi-threaded application such as a web server -- setlocale()
changes the locale for the entire process.

>> Alternatively, we could add "en_DE" to the alias table and set
>> a default encoding to use. 
> 
> Where would you get a complete list of all the new aliases that would need be to be added?

We could start with the list of supported locales in Windows:

https://docs.microsoft.com/en-us/cpp/c-runtime-library/locale-names-languages-and-country-region-strings?view=msvc-160

Since everything is moving toward UTF-8 as standard encoding,
I'd suggest use alias the plain locale codes to <locale>.UTF-8.

> The APIs were written at a time where locale modifiers 
> simply did mot exist. 

Technically, locale modifiers did exist circa 2000, but I suppose you mean that they were uncommon to the point of being unheard of at the time.

The modifier field was specified in the X/Open Portability Guide Issue 3 (XPG3) in 1989, and again in XPG4 in 1992 as "language[_territory][.codeset][@modifier]". I can't provide links to the specifications (they're not freely available), but here's a link to X/Open "Internationalisation Guide Version 2" (1993), which defines the modifier field in section 5.1.2 (pages 88-89):

https://pubs.opengroup.org/onlinepubs/009269599/toc.pdf

> Support could be added via a special locale tuple return
> object, which looks like 2-tuple, but comes with extra attributes
> to store the modifier

That's a good idea and worth implementing. But the _strptime and calendar modules have no need to call getlocale(LC_TIME). IMO, it adds fragility for no benefit. All they need to save is the result of setlocale(LC_TIME). 

Also, the default locale for calendar.LocaleTextCalendar has no need to use getdefaultlocale() instead of using an empty string, i.e. setlocale(LC_TIME, ""). The latter is simpler and more reliable.

---

> support for Windows is only partial, due to the
> completely different approach Windows' CRT took to locales.

Using the same implementation for POSIX and Windows is needlessly complicated, and difficult to reason about how it behaves in all cases.

I suggest implementing separate versions of normalize() and _parse_localename() for Windows, making use of direct queries via _winapi.GetLocaleInfoEx() (to be added). 

The mapping in encodings.aliases also needs comprehensive coverage for Windows code pages (e.g. cp20127 -> ascii, cp28591 -> latin_1, etc). A poor match should not be aliased, such as code page 20932 and euc_JP. (For all 3-byte sequences in standard euc-JP, code page 20932 encodes 2-byte sequences by dropping the lead byte and masking the third byte as ASCII.)

If the locale string doesn't include a codeset, then normalize() shouldn't do anything to obtain one. It's not necessary in Windows. If there's a codeset, normalize() should ensure it's "UTF-8", "OCP", "ACP", or a Windows code page in the right form, e.g. "ascii" -> "20127". ucrt supports "ACP" and "OCP" codesets for the locale's ANSI and OEM code pages. These must be in uppercase, e.g. "hindi.acp" -> "hindi.ACP". ucrt will set the latter as "Hindi_India.utf8" (it's a Unicode-only locale), which should parse as ("Hindi_India", "UTF-8").

If the locale without the codeset isn't a valid Windows BCP-47 locale name, as determined by the NLS API, then normalize() should only care about case-insensitive normalization of "C" and "POSIX" as "C", e.g. "c" -> "C". No other normalization is necessary. ucrt supports case-insensitive "language[_country[.codepage]]" and ".codepage" forms, where language and country are either the full English names, LOCALE_SENGLISHLANGUAGENAME and LOCALE_SENGLISHCOUNTRYNAME, or 3-letter abbreviations, LOCALE_SABBREVLANGNAME and LOCALE_SABBREVCTRYNAME, such as "enu_USA". It also supports locale aliases such as "american[.codeset]". If the result isn't "C" or a BCP-47 locale name, ucrt setlocale() always returns the "language_country.codepage" form with full English names.

A BCP-47 locale name such as "en" or "en_US" cannot be used with a codeset other than UTF-8. If no codeset is specified, ucrt implicitly uses the locale's ANSI code page. 

If a BCP-47 locale name is paired with a codeset that's neither the given locale's ANSI codepage nor UTF-8, then normalize it to the "language_country.codepage" form. For example, "fr_FR.latin-1" -> "French_France.28591". Parse the latter as ("French_France", "ISO-8859-1"). 

If a BCP-47 locale name is paired with the locale's ANSI code page, then normalize it without the code page, e.g. "sr_Latn_RS.cp1250" -> "sr_Latn_RS". Look up the locale's ANSI code page when parsing the latter, e.g. "sr_Latn_RS" -> ("sr_Latn_RS", "cp1250"). 

If a BCP-47 locale name is paired with UTF-8, then there isn't much to do other than normalize the locale name and encoding name, e.g. "en_us.utf8" -> "en_US.UTF-8".

On 18.02.2021 19:04, Eryk Sun wrote:
> 
> Eryk Sun <eryksun@gmail.com> added the comment:
> 
>> The APIs were written at a time where locale modifiers 
>> simply did mot exist. 
> 
> Technically, locale modifiers did exist circa 2000, but I suppose you mean that they were uncommon to the point of being unheard of at the time.
> 
> The modifier field was specified in the X/Open Portability Guide Issue 3 (XPG3) in 1989, and again in XPG4 in 1992 as "language[_territory][.codeset][@modifier]". I can't provide links to the specifications (they're not freely available), but here's a link to X/Open "Internationalisation Guide Version 2" (1993), which defines the modifier field in section 5.1.2 (pages 88-89):
> 
> https://pubs.opengroup.org/onlinepubs/009269599/toc.pdf

Thanks for the link.

The standard may have already been defined in POSIX, but I'm pretty
sure this was not a thing on the SuSE Linux system I was working off
at the time :-)

This is what glibc supported in 2000:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/SUPPORTED;h=5ca1beb8eacbe5d86322cb2d015ec260422813d8;hb=1914953f71bf9ace1981d838b3777e473e63e8d1

vs. today:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/SUPPORTED

The first time I ever saw @modifiers in the wild was when the
Euro symbol was introduced in preparation for Jan 1, 2002.

Note that the encoding alias table in Python 2.0 does include an
entry "utf8@ucs4", which originated from the X11 local.alias file.
Apparently those modifiers were already in use on Digital Unix
at the time:

https://github.com/freedesktop/xorg-libX11/blob/deae12c6b683898f5213992d561a59d4ea889cca/nls/locale.alias.pre
(this is from 2003; couldn't find earlier references)

It is interesting to see that the locale.alias.pre file actually
did include Windows locale definitions. On Linux, those don't get
compiled in, so I didn't get to see them :-)

Also interesting is that they aliased e.g. cp1252 to ISO-8859-1,
even though the two encodings are not the same.

>> What does use getlocale is time.strptime and datetime.datetime.strptime, so when getlocale fails, strptime fails.

> Would they work with getlocale() returning None for the encoding ?

Yes. All getlocale is used for in _strptime.py is comparing the value returned to the previous value returned.

I expect changing _strptime._getlang to return the unnormalized value locale._setlocale(locale.LC_TIME, None) would work as well and more robustly.

> All getlocale is used for in _strptime.py is comparing the value 
> returned to the previous value returned.

Which is why _strptime should be calling setlocale(LC_TIME), the same as the calendar module. That's not to say that I don't think getlocale() and normalize() need to be fixed. But returning None for the encoding when there's no codeset, while it works for a few cases, doesn't address many other cases.

For example, normalize() and getlocale() will often be wrong in many cases when an encoding is guessed, such as normalize('en_US') -> 'en_US.ISO8859-1' and normalize('ja_JP') -> 'ja_JP.eucJP'. The encoding is wrong, plus no encoding except UTF-8 is allowed in a BCP-47 locale, so setlocale() will fail. 

In all but four cases, classic ucrt "language_country.codepage" locales such as "Japanese_Japan.932" are parsed in a 'benignly' incorrect way (i.e. not as RFC 1766 language tags), which at least roundtrips with setlocale(). It simply splits out the codeset, e.g. ('Japanese_Japan', '932'). The four misbehaving cases are actually the ones for which getlocale() works as documented because the locale_alias mapping has an entry for them.

    * "French_France.1252" -> ('fr_FR', 'cp1252')
    * "German_Germany.1252" -> ('de_DE', 'cp1252')
    * "Portuguese_Brazil.1252" -> ('pt_BR', 'cp1252')
    * "Spanish_Spain.1252" -> ('es_ES', 'cp1252')

The problem is that the parsed tuples don't roundtrip because normalize() keeps the encoding, complete with the 'cp' prefix, and only UTF-8 is allowed in a BCP-47 locale. For example:

    >>> locale.setlocale(locale.LC_CTYPE, 'French_France.1252')
    'French_France.1252'
    >>> locale.getlocale()
    ('fr_FR', 'cp1252')
    >>> locale.normalize(locale._build_localename(locale.getlocale()))
    'fr_FR.cp1252'
    >>> try: locale.setlocale(locale.LC_CTYPE, locale.getlocale())
    ... except locale.Error as e: print(e)
    ...
    unsupported locale setting

I suppose normalize() could be special cased in Windows to look for a BCP-47 locale and omit the encoding if it's not UTF-8. I suppose _parse_localename() could be special cased to use None for the encoding if there's no codeset. But this just leaves me feeling unsettled and disappointed that we could be doing a better job of providing the documented behavior of getlocale() and normalize() by implementing them separately for Windows using the tools that the OS provides.

FYI, I've commented on this problem across a few issues, including bpo-20088 and bpo-23425 in early 2015, and then extensively in bpo-37945 in mid 2019. Plus my latest comments in msg387256 in this issue. The latter suggestions could be combined with something like the mapping that's generated by the code in msg235937 in bpo-23425, in order to parse a classic ucrt locale string such as "Japanese_Japan.932" properly as ("ja_JP", "cp932"), and then build and normalize it back as "Japanese_Japan.932".