Add support for glibc supported locales #64278

serhiy-storchaka · 2013-12-27T10:12:28Z

BPO	20079
Nosy	@malemburg, @loewis, @vstinner, @berkerpeksag, @serhiy-storchaka
Files	locale_glibc_supported.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2014-10-19.21:06:13.487>
created_at = <Date 2013-12-27.10:12:27.573>
labels = ['type-feature', 'library']
title = 'Add support for glibc supported locales'
updated_at = <Date 2014-10-19.21:06:13.486>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2014-10-19.21:06:13.486>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2014-10-19.21:06:13.487>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2013-12-27.10:12:27.573>
creator = 'serhiy.storchaka'
dependencies = []
files = ['33278']
hgrepos = []
issue_num = 20079
keywords = ['patch']
message_count = 18.0
messages = ['206987', '206988', '228058', '228061', '228067', '228112', '228117', '228125', '228178', '228184', '228186', '228187', '228188', '228189', '228190', '228192', '228194', '229242']
nosy_count = 7.0
nosy_names = ['lemburg', 'loewis', 'vstinner', 'Arfrever', 'python-dev', 'berker.peksag', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20079'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

serhiy-storchaka · 2013-12-27T10:12:23Z

Proposed patch adds to locale alias table the mappings for locales supported in recent glibc (v 2.18). It also modifies the makelocalealias.py script so that it parses the SUPPORTED file from glibc sources and supports command line options for source paths.

serhiy-storchaka · 2013-12-27T10:14:06Z

Totally added 100 new mappings.

malemburg · 2014-10-01T07:24:50Z

Thanks, Serhiy.

The patch looks good, except for one nit: the makelocalealias.py normaly also generates a list of changes and these are put at the top of the locale_alias dictionary.

Could you add that as well ?

serhiy-storchaka · 2014-10-01T07:52:03Z

The makelocalealias.py generates only a list of removes and updates, not additions.

I recommend first apply bpo-20076, it will eliminate most additions.

malemburg · 2014-10-01T10:11:47Z

On 01.10.2014 09:52, Serhiy Storchaka wrote:

The makelocalealias.py generates only a list of removes and updates, not additions.

Ah, ok.

I recommend first apply bpo-20076, it will eliminate most additions.

Agreed. Please apply both patches.

Thanks,

Marc-Andre Lemburg
eGenix.com

serhiy-storchaka · 2014-10-01T20:00:47Z

Oh, sorry, bpo-20076 doesn't eliminate additions in this issue, but vice verse this issue eliminates some additions of bpo-20076. So this patch should be applied first.

python-dev · 2014-10-01T20:51:15Z

New changeset 810542d07b4e by Serhiy Storchaka in branch 'default':
Issue bpo-20079: Added locales supported in glibc 2.18 to locale alias table.
https://hg.python.org/cpython/rev/810542d07b4e

New changeset 00e4190b308f by Serhiy Storchaka in branch '3.4':
Issue bpo-20079: Added locales supported in glibc 2.18 to locale alias table.
https://hg.python.org/cpython/rev/00e4190b308f

New changeset 3bc5edbbadae by Serhiy Storchaka in branch '2.7':
Issue bpo-20079: Added locales supported in glibc 2.18 to locale alias table.
https://hg.python.org/cpython/rev/3bc5edbbadae

serhiy-storchaka · 2014-10-01T21:17:44Z

Thank you for your review Marc-Andre.

See also related bpo-20087.

python-dev · 2014-10-02T07:53:00Z

New changeset 5502a82fb103 by Serhiy Storchaka in branch 'default':
Issue bpo-20079: makelocalealias.py now supports installed SUPPORTED file,
https://hg.python.org/cpython/rev/5502a82fb103

New changeset 4a19ce6c6e0c by Serhiy Storchaka in branch '2.7':
Issue bpo-20079: Updated alias mapping to glibc 2.19 SUPPORTED file.
https://hg.python.org/cpython/rev/4a19ce6c6e0c

New changeset e9aaefa3f2eb by Serhiy Storchaka in branch '3.4':
Issue bpo-20079: Updated alias mapping to glibc 2.19 SUPPORTED file.
https://hg.python.org/cpython/rev/e9aaefa3f2eb

New changeset 994a694ab71c by Serhiy Storchaka in branch 'default':
Issue bpo-20079: Updated alias mapping to glibc 2.19 SUPPORTED file.
https://hg.python.org/cpython/rev/994a694ab71c

vstinner · 2014-10-02T08:24:35Z

A lot of buildbot failed. Example:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/2692/steps/test/logs/stdio

======================================================================
FAIL: test_valencia_modifier (test.test_locale.NormalizeTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_locale.py", line 428, in test_valencia_modifier
    self.check('ca_ES@valencia', 'ca_ES.ISO8859-1@valencia')
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_locale.py", line 370, in check
    self.assertEqual(locale.normalize(localename), expected, msg=localename)
AssertionError: 'ca_ES.ISO8859-15@valencia' != 'ca_ES.ISO8859-1@valencia'
- ca_ES.ISO8859-15@valencia
?                -
+ ca_ES.ISO8859-1@valencia
 : ca_ES@valencia

malemburg · 2014-10-02T08:38:01Z

On 02.10.2014 10:24, STINNER Victor wrote:

STINNER Victor added the comment:

A lot of buildbot failed. Example:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/2692/steps/test/logs/stdio

======================================================================
FAIL: test_valencia_modifier (test.test_locale.NormalizeTest)
----------------------------------------------------------------------

> Traceback (most recent call last):
>   File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_locale.py", line 428, in test_valencia_modifier
>     self.check('ca_ES@valencia', 'ca_ES.ISO8859-1@valencia')
>   File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_locale.py", line 370, in check
>     self.assertEqual(locale.normalize(localename), expected, msg=localename)
> AssertionError: 'ca_ES.ISO8859-15@valencia' != 'ca_ES.ISO8859-1@valencia'
> - ca_ES.ISO8859-15@valencia
> ?                -
> + ca_ES.ISO8859-1@valencia
>  : ca_ES@valencia

I guess the test have to be adjusted as well :-)

What I don't understand is why the above case failed. That mapping
wasn't changed by the patch, AFAICT.

python-dev · 2014-10-02T08:38:52Z

New changeset 7ce459fc57b9 by Serhiy Storchaka in branch '2.7':
Issue bpo-20079: Fixed tests.
https://hg.python.org/cpython/rev/7ce459fc57b9

New changeset 8c5555dea138 by Serhiy Storchaka in branch '3.4':
Issue bpo-20079: Fixed tests.
https://hg.python.org/cpython/rev/8c5555dea138

New changeset 433048fd4206 by Serhiy Storchaka in branch 'default':
Issue bpo-20079: Fixed tests.
https://hg.python.org/cpython/rev/433048fd4206

malemburg · 2014-10-02T08:41:50Z

On 02.10.2014 10:38, Marc-Andre Lemburg wrote:

What I don't understand is why the above case failed. That mapping
wasn't changed by the patch, AFAICT.

Ah, the change is in the second patch round you applied, which
is not on the ticket as separate patch file.

Reviewing that, I think this change need to be documented
(in the comment above the table) and also researched a bit:

'kk_kz': 'kk_KZ.ptcp154',
+ 'kk_kz': 'kk_KZ.RK1048',

serhiy-storchaka · 2014-10-02T08:45:27Z

Oh, sorry. I should be more careful.

Test failed due to mismatch between glibc and X11 locale.alias (bpo-20087). In X11 locale.alias ca_ES is mapped to ca_ES.ISO8859-1, and in glibc 2.19 it is mapped to ca_ES.ISO8859-15. ca_ES@valencia exists only in glibc SUPPORTED file and was added in the last commit (due to mismatch with ca_ES).

serhiy-storchaka · 2014-10-02T08:49:41Z

Reviewing that, I think this change need to be documented
(in the comment above the table) and also researched a bit:

The 'kk_kz' was added just, in the first commit of this issue (which was based on glibc 2.18 data). No changes to previous Python versions, only addition.

serhiy-storchaka · 2014-10-02T09:13:04Z

Test failed due to mismatch between glibc and X11 locale.alias (bpo-20087).
In X11 locale.alias ca_ES is mapped to ca_ES.ISO8859-1, and in glibc 2.19
it is mapped to ca_ES.ISO8859-15. ca_ES@valencia exists only in glibc
SUPPORTED file and was added in the last commit (due to mismatch with
ca_ES).

Hmm, no, both aliases are from glibc SUPPORTED file:

ca_ES.UTF-8 UTF-8
ca_ES ISO-8859-1
ca_ES@euro ISO-8859-15
ca_ES.UTF-8@valencia UTF-8
ca_ES@valencia ISO-8859-15

malemburg · 2014-10-02T09:44:40Z

On 02.10.2014 11:13, Serhiy Storchaka wrote:

Serhiy Storchaka added the comment:

> Test failed due to mismatch between glibc and X11 locale.alias (bpo-20087).
> In X11 locale.alias ca_ES is mapped to ca_ES.ISO8859-1, and in glibc 2.19
> it is mapped to ca_ES.ISO8859-15. ca_ES@valencia exists only in glibc
> SUPPORTED file and was added in the last commit (due to mismatch with
> ca_ES).

Hmm, no, both aliases are from glibc SUPPORTED file:

ca_ES.UTF-8 UTF-8
ca_ES ISO-8859-1
ca_ES@euro ISO-8859-15
ca_ES.UTF-8@valencia UTF-8
ca_ES@valencia ISO-8859-15

See the comment in
https://sourceware.org/bugzilla/attachment.cgi?id=966
"The point of having
% a separate locale is only for PO translations, which have a lot of social
% support and are very appreciated by the Valencian-speaking community."

Since it's a new locale, they simply used the Latin-1 version with Euro
symbol to start with, which is a good move. Most of Europe is stuck
with having ISO-8859-1 as default, which does not include the Euro
symbol. Then again: most Unix installations use UTF-8 nowadays anyway.

serhiy-storchaka · 2014-10-13T12:32:44Z

I think there is nothing more to do here and the issue can be closed.

serhiy-storchaka added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Dec 27, 2013

serhiy-storchaka self-assigned this Oct 1, 2014

serhiy-storchaka closed this as completed Oct 1, 2014

vstinner reopened this Oct 2, 2014

serhiy-storchaka closed this as completed Oct 19, 2014

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for glibc supported locales #64278

Add support for glibc supported locales #64278

serhiy-storchaka commented Dec 27, 2013

serhiy-storchaka commented Dec 27, 2013

serhiy-storchaka commented Dec 27, 2013

malemburg commented Oct 1, 2014

serhiy-storchaka commented Oct 1, 2014

malemburg commented Oct 1, 2014

serhiy-storchaka commented Oct 1, 2014

python-dev mannequin commented Oct 1, 2014

serhiy-storchaka commented Oct 1, 2014

python-dev mannequin commented Oct 2, 2014

vstinner commented Oct 2, 2014

malemburg commented Oct 2, 2014

python-dev mannequin commented Oct 2, 2014

malemburg commented Oct 2, 2014

serhiy-storchaka commented Oct 2, 2014

serhiy-storchaka commented Oct 2, 2014

serhiy-storchaka commented Oct 2, 2014

malemburg commented Oct 2, 2014

serhiy-storchaka commented Oct 13, 2014

Add support for glibc supported locales #64278

Add support for glibc supported locales #64278

Comments

serhiy-storchaka commented Dec 27, 2013

serhiy-storchaka commented Dec 27, 2013

serhiy-storchaka commented Dec 27, 2013

malemburg commented Oct 1, 2014

serhiy-storchaka commented Oct 1, 2014

malemburg commented Oct 1, 2014

Thanks,

serhiy-storchaka commented Oct 1, 2014

python-dev mannequin commented Oct 1, 2014

serhiy-storchaka commented Oct 1, 2014

python-dev mannequin commented Oct 2, 2014

vstinner commented Oct 2, 2014

malemburg commented Oct 2, 2014

python-dev mannequin commented Oct 2, 2014

malemburg commented Oct 2, 2014

serhiy-storchaka commented Oct 2, 2014

serhiy-storchaka commented Oct 2, 2014

serhiy-storchaka commented Oct 2, 2014

malemburg commented Oct 2, 2014

serhiy-storchaka commented Oct 13, 2014