classification
Title: setlocale error message is confusing
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eric.araujo, ezio.melotti, haypo, lemburg, loewis, nailor, petri.lehtinen, python-dev, skrah, terry.reedy, vincent.chute
Priority: normal Keywords: easy, patch

Created on 2008-06-09 12:08 by vincent.chute, last changed 2011-11-05 08:28 by petri.lehtinen. This issue is now closed.

Files
File name Uploaded Description Edit
issue3067.patch nailor, 2011-10-18 09:44 review
issue3067_v2.patch nailor, 2011-10-18 16:07 review
issue3067_v3.patch nailor, 2011-10-18 20:04 review
Messages (27)
msg67861 - (view) Author: (vincent.chute) Date: 2008-06-09 12:08
import locale
locale.setlocale( locale.LC_ALL, u'ja_JP.utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/locale.py", line 475, in setlocale
    locale = normalize(_build_localename(locale))
  File "/usr/lib/python2.5/locale.py", line 383, in _build_localename
    language, encoding = localetuple
ValueError: too many values to unpack

The problem is line 473:
    if locale and type(locale) is not type(""):

Replacing this with
     if locale and not isinstance(locale, basestring):
fixes the problem.
msg67862 - (view) Author: (vincent.chute) Date: 2008-06-09 12:11
I have confirmed this exists on trunk

http://svn.python.org/view/python/trunk/Lib/locale.py?rev=63824&view=markup

(63824 is the latest)
where the line in question is now 475
msg82511 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-02-20 03:08
FWIW the type("") is gone in Py3, now it is:
"if locale and not isinstance(locale, _builtin_str):"
where "from builtins import str as _builtin_str"
(http://svn.python.org/view/python/branches/py3k/Lib/locale.py?view=markup)

However Py3.0 now raises the same error when the second arg is a byte
string:
>>> locale.setlocale(locale.LC_ALL, b'ja_JP.utf8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Programs\Python30\lib\locale.py", line 500, in setlocale
    locale = normalize(_build_localename(locale))
  File "C:\Programs\Python30\lib\locale.py", line 408, in _build_localename
    language, encoding = localetuple
ValueError: too many values to unpack

On Py3, locale.setlocale() should allow only unicode strings and reject
byte strings.
msg112650 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-03 18:53
The docs say that the locale arg should be None, tuple, or string, so I take that to mean that Unicode should be OK for 2.x, and that would help porting to 3.x. If bytes are rejected in 3.x, there should be TypeError raised, not ValueError, as is still the case in 3.1.2.
msg138619 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-06-18 23:28
After more thought and investigation, I have changed my opinions on this issue.


Allowing unicode string for locale in 2.7:
Since the module predates unicode strings (it is in 1.5) and since the locale string is passed to a C function, 'string' in the doc can just as well be taken to mean ascii byte string only, as the code requires. As far as I know, unicode is never needed. Allowing such could be considered a feature addition, which is not allowed for 2.7. So I would reject the OP's request (and have hence changed the title).

Expected failure cases could be added to test_locale.py.


Options for locale name:
As I remember, multiple assignments in 1.5, as in

def _build_localename(localetuple):
    language, encoding = localetuple

required a tuple on the right and was called 'tuple unpacking'.
Now, any iterable producing 2 items works; Rather than change to code to check that 'localetuple' really is a tuple (which could break code and the principle of duck-typing), I think the doc should be updated to

"If locale is specified, it may be a None, a string, or an iterable producing two strings, language code and encoding."

This is not a feature addition but a recognition of a new feature added versions ago. The parameter name in the private function should then be shortened to just 'locale'.

Test cases with non-tuples could be added if not present now.


Exception message:
The current message arises from setlocale assuming that a 'locale' that is neither None or a string is a valid iterable for a call to _build_localename. A strings of the wrong type produces too many items and hence the obscure message.

Python is known to be inconsistent in its usage of ValueError versus TypeError for builtin function args. Guido has said to leave inconsistencies rather than break code. So I retract the ValueError to TypeError suggestion.

The accompanying messages, however, can be improved. The lines above that fails could be wrapped with
    try:
        language, encoding = locale
    except ValueError:
        raise ValueError("Locale must be None, a string, or an iterable of two strings -- language code, encoding -- not {}".format(locale))

The scope of the wrapper could be extended to the entire function so that failure of
        return language + '.' + encoding
would also be caught. Failure would happen if locale produced 2 non-strings items, so that the double assignment 'worked', but the string concatenation failed.

A complication: the doc says
"exception locale.Error 
Exception raised when setlocale() fails.

locale.setlocale(category, locale=None) 
...If the modification of the locale fails, the exception Error is raised."

So it seems that either a) the wrapper above should raise Error instead, or the doc could more clearly say "Exception raised when the locale passed to setlocale is not recognized."
msg138715 - (view) Author: (vincent.chute) Date: 2011-06-20 13:58
"Since the module predates unicode strings (it is in 1.5) and since the locale string is passed to a C function, 'string' in the doc can just as well be taken to mean ascii byte string only, as the code requires."

My only comment is that generally it doesn't seem reasonable to me that developer should need to investigate the history and implementation of a function in order to understand the documentation correctly.
msg138718 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-06-20 14:09
> On Py3, locale.setlocale() should allow only unicode strings
> and reject byte strings.

I agree and it is the current behaviour (of Python 3.3). I don't see any use case of a byte strings in locale.setlocale() with Python 3.3, so I remove Python 3 from the versions of this issue.
msg138719 - (view) Author: Roundup Robot (python-dev) Date: 2011-06-20 14:14
New changeset d370d609d09b by Victor Stinner in branch '2.7':
Close #3067: locale.setlocale() accepts a Unicode locale.
http://hg.python.org/cpython/rev/d370d609d09b
msg138721 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-06-20 14:17
I fixed locale.setlocale() of Python 2.7 to accept Unicode string because it helps porting to Python 3...

But I think that the commit is just useless because we will have to wait until Python 2.7.3 is released, and if you want to support older Python versions, we will have to encode the locale explicitly to ASCII.

Anyway, you should move to Python 3 (3.2 or later if possible) if you want a better Unicode support.
msg138739 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-06-20 16:51
Victor, the issue for 3.x, which remains, is to improve the error message. I also suggested a doc change, though I would like Mark or Martin's comments before I would make it.

>But I think that the commit is just useless because we will have to wait until Python 2.7.3 is released, and if you want to support older Python versions, we will have to encode the locale explicitly to ASCII.

Exactly. 'Older versions' includes older versions of 2.7. This is why I suggested that making the change to 2.7 would be a feature addition, which is not permitted for the very reason you give. I think the commit should be reverted.

Certainly, when a another developer says "This patch should be rejected and not committed' after careful review, you should discuss, possibly on pydev, before committing.
msg138756 - (view) Author: Roundup Robot (python-dev) Date: 2011-06-20 20:07
New changeset e72a2a60316f by Victor Stinner in branch '2.7':
Revert d370d609d09b as requested by Terry Jan Reedy:
http://hg.python.org/cpython/rev/e72a2a60316f
msg145789 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2011-10-18 09:44
Added a patch that implements two things:

setlocale now raises locale.Error('Locale must be None, a string, or an iterable of two strings -- language code, encoding.'). I decided to remove the proposed .format(locale), as it wasa a bit confusing when passing a tuple containing invalid items.

I also added two tests, one for bytes and another for a tuple of two bytes.
msg145818 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-18 15:48
Thanks for the patch.  Exception messages are considered implementation details, so I would not test them.  Testing that an exception is raised is good enough IMO.
msg145827 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2011-10-18 16:07
I modified the patch not to contain the tests against exception messages
msg145852 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-10-18 18:49
I think the reported exception type is incorrect. Given that the error message is 'Locale must be None, a string, or an iterable of two strings -- language code, encoding.', it very much sounds like a TypeError is being reported here.

So I think all that's needed is that the ValueError is converted into a TypeError.

Also notice that the tuple unpacking may actually succeed:

py> locale.setlocale(locale.LC_ALL,u"en")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/locale.py", line 513, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
msg145855 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2011-10-18 19:08
Maybe we should return TypeError with the same message then? That would require some modification of documentation though, as it states: "If the modification of the locale fails, the exception Error is raised.".

I don't really understand the "locale unpacking may actually succeed". Isn't that what supposed to happen, to my knowledge "en" is not a valid locale and that's a totally different issue? If I'm wrong, please correct, I've just started wandering in to Python Core development :)
msg145859 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-10-18 19:30
> Maybe we should return TypeError with the same message then? That
> would require some modification of documentation though, as it
> states: "If the modification of the locale fails, the exception Error
> is raised.".

No, any operation can report TypeError and ValueError without explicit
mentioning in the documentation. Saying that the parameters should be
this and that implies that if they are different, you get a TypeError
or ValueError.

> I don't really understand the "locale unpacking may actually
> succeed". Isn't that what supposed to happen, to my knowledge "en" is
> not a valid locale and that's a totally different issue?

See my example again:

py> locale.setlocale(locale.LC_ALL,u"en")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/locale.py", line 513, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
py> locale.setlocale(locale.LC_ALL,u"eng")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/locale.py", line 512, in setlocale
    locale = normalize(_build_localename(locale))
  File "/usr/lib/python2.6/locale.py", line 420, in _build_localename
    language, encoding = localetuple
ValueError: too many values to unpack

So for u"eng" you get the ValueError. For u"en", you get past that
point, and then get a locale.Error. These are both Unicode strings,
but the outcome is quite different (and still would be different
under your patch).
msg145862 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2011-10-18 19:42
Thanks for clarification! I see the problem now. So if I get this correctly we should change the _build_localename to raise TypeError? If the given locale is in wrong format, we'll get TypeError, but if it's valid type but otherwise invalid locale (like 'en'), we'll get ValueError (or more specifically locale.Error).
msg145864 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-10-18 19:53
> Thanks for clarification! I see the problem now. So if I get this
> correctly we should change the _build_localename to raise TypeError?

Yes, that's what I'm proposing.

> If the given locale is in wrong format, we'll get TypeError, but if
> it's valid type but otherwise invalid locale (like 'en'), we'll get
> ValueError (or more specifically locale.Error).

Ideally, yes. Notice that it will be difficult to produce a TypeError
for u"en", unless you explicitly test for Unicode objects.
msg145866 - (view) Author: Jyrki Pulliainen (nailor) * Date: 2011-10-18 20:04
Uploaded a new patch that raises TypeError
msg147030 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-04 20:41
New changeset 931ae170e51c by Petri Lehtinen in branch '3.2':
Issue #3067: Fix the error raised by locale.setlocale()
http://hg.python.org/cpython/rev/931ae170e51c

New changeset d90d88380aca by Petri Lehtinen in branch 'default':
Issue #3067: Fix the error raised by locale.setlocale()
http://hg.python.org/cpython/rev/d90d88380aca
msg147032 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-04 20:43
Terry: Do you still think there's need for a doc update?
msg147060 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-11-05 00:19
Yes. I think in locale.rst (assuming that is the name)
'''
exception locale.Error 
Exception raised when setlocale() fails.

locale.setlocale(category, locale=None) 
If *locale* is specified, it may be a string, a tuple of the form (language code, encoding), or None. If it is a tuple, it is converted to a string using the locale aliasing engine.
'''
should be changed to
'''
exception locale.Error 
Exception raised when the locale passed to setlocale() is not recognized.

locale.setlocale(category, locale=None) 
If *locale* is specified, it may be a None, a string, or an iterable of two strings, language code and encoding. String pairs are converted to a single string using the locale aliasing engine.
'''
where language code and encoding are gray shaded as they are now.
msg147069 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-05 07:24
> If *locale* is specified, it may be a None, a string, or an iterable of two strings, language code and encoding. String pairs are converted to a single string using the locale aliasing engine.

What about the possible None value then? Do you think that mentions to
it be dropped?

I don't think so, because setlocale(locale.LC_ALL, None) is an
explicit way of saying "Return me the current value", especially
because the function's name is SETlocale, which doesn't make it
explicit.

If None is not dropped, the ", language code and encoding" should
maybe be in parentheses insteead: "to strings (language code and
encoding), or None..."
msg147070 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-11-05 07:30
Yes, parentheses would be better.
msg147075 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-05 08:25
New changeset 34c9465f5023 by Petri Lehtinen in branch '2.7':
Issue #3067: Enhance the documentation and docstring of locale.setlocale()
http://hg.python.org/cpython/rev/34c9465f5023

New changeset 98806dd03506 by Petri Lehtinen in branch '3.2':
Issue #3067: Enhance the documentation and docstring of locale.setlocale()
http://hg.python.org/cpython/rev/98806dd03506

New changeset 8a27920efffe by Petri Lehtinen in branch 'default':
Issue #3067: Enhance the documentation and docstring of locale.setlocale()
http://hg.python.org/cpython/rev/8a27920efffe
msg147076 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-05 08:28
I decided to restructure the documentation of setlocale() a bit and I think it's better now overall. It includes Terry's suggestions.

I think this issue can now be closed. Thanks for the report and patches!
History
Date User Action Args
2014-02-27 20:27:36ned.deilylinkissue20793 superseder
2011-11-05 08:28:25petri.lehtinensetstatus: open -> closed

messages: + msg147076
2011-11-05 08:25:14python-devsetmessages: + msg147075
2011-11-05 07:30:07terry.reedysetmessages: + msg147070
2011-11-05 07:24:03petri.lehtinensetmessages: + msg147069
2011-11-05 00:19:21terry.reedysetstatus: pending -> open

nosy: + docs@python
messages: + msg147060

assignee: docs@python
components: + Documentation, - Library (Lib), Unicode
2011-11-04 20:43:41petri.lehtinensetstatus: open -> pending
resolution: fixed
messages: + msg147032
2011-11-04 20:41:56python-devsetmessages: + msg147030
2011-11-03 10:44:57petri.lehtinensetnosy: + petri.lehtinen
2011-10-18 20:04:33nailorsetfiles: + issue3067_v3.patch

messages: + msg145866
2011-10-18 19:53:13loewissetmessages: + msg145864
2011-10-18 19:42:25nailorsetmessages: + msg145862
2011-10-18 19:30:37loewissetmessages: + msg145859
2011-10-18 19:08:11nailorsetmessages: + msg145855
2011-10-18 18:49:57loewissetmessages: + msg145852
2011-10-18 16:07:15nailorsetfiles: + issue3067_v2.patch

messages: + msg145827
2011-10-18 15:48:58eric.araujosetnosy: + eric.araujo
messages: + msg145818
2011-10-18 09:44:45nailorsetfiles: + issue3067.patch

nosy: + nailor
messages: + msg145789

keywords: + patch
2011-06-20 20:07:07python-devsetmessages: + msg138756
2011-06-20 16:51:34terry.reedysetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg138739

versions: + Python 3.2, Python 3.3
2011-06-20 14:17:47hayposetmessages: + msg138721
2011-06-20 14:14:55python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg138719

resolution: fixed
stage: test needed -> resolved
2011-06-20 14:09:30hayposetmessages: + msg138718
versions: - Python 3.2, Python 3.3
2011-06-20 13:58:46vincent.chutesetmessages: + msg138715
2011-06-19 22:28:11hayposetnosy: + haypo
2011-06-18 23:28:30terry.reedysetnosy: + lemburg, loewis
title: setlocale fails with unicode strings on Py2 and with byte strings on Py3 -> setlocale error message is confusing
messages: + msg138619

versions: + Python 3.3, - Python 3.1
2011-06-18 09:42:45skrahsetnosy: + skrah
2010-08-03 18:53:13terry.reedysettype: behavior
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6, Python 2.5, Python 3.0
keywords: + easy
nosy: + terry.reedy

messages: + msg112650
stage: test needed
2009-02-20 03:08:09ezio.melottisetnosy: + ezio.melotti
title: setlocale Tracebacks on unicode locale strings -> setlocale fails with unicode strings on Py2 and with byte strings on Py3
messages: + msg82511
components: + Unicode
versions: + Python 2.6, Python 3.0
2008-06-09 12:11:25vincent.chutesetmessages: + msg67862
2008-06-09 12:08:13vincent.chutecreate