classification
Title: Python fails silently on bad locale
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.0, Python 3.1
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: facundobatista, mark.dickinson, motteneder, nelis, stuartcw, vstinner, wangchun
Priority: critical Keywords: patch

Created on 2008-02-24 10:26 by motteneder, last changed 2008-12-11 18:13 by mark.dickinson. This issue is now closed.

Files
File name Uploaded Description Edit
full_make_output.txt facundobatista, 2008-02-26 12:19 Full make output when LANG=UTF-8
make.log motteneder, 2008-02-26 14:26 make output
make.log motteneder, 2008-02-26 14:31 make output
issue2173.patch mark.dickinson, 2008-12-08 16:48
Messages (11)
msg62892 - (view) Author: Michael Otteneder (motteneder) Date: 2008-02-24 10:26
Python seems to fail silently when LANG enviroment variable is set to a
bad value. In Mac OS X Leopard it is set too UTF-8 for instance which
does not seem to be valid. Python fails building the modules during make
and the generated python.exe is unable to output anything. Setting the
LANG variable to C or some other valid value fixes the problem. But I
suppose python should fail more verbose in case of a bad locale,or maybe
it should fallback to a standard locale.
msg63039 - (view) Author: Facundo Batista (facundobatista) * (Python committer) Date: 2008-02-26 12:19
Same here, Ubuntu on x86 32b.

Doing first a "make clean", ./configure runs ok.

But then, in the "make":

...
  File "/home/facundo/devel/reps/python/py3k/Lib/locale.py", line 479,
in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
make: *** [sharedmods] Error 1

Full make output is attached.

The problem seems to be in the setlocale module. I compiled everything
with LANG in es_AR.UTF-8, but then executing Python again with a bad locale:

facundo@virtub:~/devel/reps/python/py3k$ LANG=UTF-8 ./python 
Python 3.0a2+ (py3k:61084, Feb 26 2008, 10:12:19) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getpreferredencoding()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/facundo/devel/reps/python/py3k/Lib/locale.py", line 515,
in getpreferredencoding
    setlocale(LC_CTYPE, "")
  File "/home/facundo/devel/reps/python/py3k/Lib/locale.py", line 479,
in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
>>> 

Regards,
msg63042 - (view) Author: Michael Otteneder (motteneder) Date: 2008-02-26 14:26
Interestingly it fails with a different error message on Mac. The Ubuntu
one is at least helpful.
I get this during make(configure also runs fine) :
gcc  -u _PyMac_Error -o python.exe \
			Modules/python.o \
			libpython3.0.a -ldl      
make: *** [sharedmods] Error 1

The created python.exe also behaves very strange:
Tinuviel:py3k mc$ ./python.exe 
Python 3.0a2+ (py3k:61071, Feb 26 2008, 14:50:42) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("hello")
>>> exit()
>>> exit()
>>> dfklasjd
>>> 

Locales:
Tinuviel:py3k mc$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=


Solution: set LANG to "C" or some other valid value.
msg63044 - (view) Author: Michael Otteneder (motteneder) Date: 2008-02-26 14:31
On my system I get different results. The error message is everything
but useful.
My Output:
gcc  -u _PyMac_Error -o python.exe \
			Modules/python.o \
			libpython3.0.a -ldl      
make: *** [sharedmods] Error 1
Tinuviel:py3k mc$ ./python.exe 
Python 3.0a2+ (py3k:61071, Feb 26 2008, 14:50:42) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print("hello")
>>> exit()
>>> exit()
>>> dfklasjd
>>> 

Locales:
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=
msg75726 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-11-11 03:25
Can you retry with Python 3.0rc2?
msg75813 - (view) Author: Wang Chun (wangchun) Date: 2008-11-13 06:12
This issue remains unsolved in the latest python 3.0rc2+ subversion 
repository as of 2008-11-13.
msg77318 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-12-08 16:13
See issue 4585, which appears to be the same problem.
msg77321 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-12-08 16:23
I think I've traced the 'no output' problem back to the device_encoding 
function in Modules/posixmodule.c.  The problem occurs when this function 
is called to try to get the encoding for stdout.

On my machine, if I do:

LC_CTYPE="UTF-8" ./python.exe

then the nl_langinfo call in device_encoding returns an empty string.  If 
I do

LC_CTYPE="bogus" ./python.exe

then it returns "US-ASCII"

and if I do

LC_CTYPE="en_US.UTF-8" ./python.exe

then it returns "UTF-8".

In the first case (where the encoding is set to ""), any subsequent 
attempts to send anything to stdout result in a LookupError with the 
message "unknown encoding".  But of course, since this exception message 
is itself sent to stdout (or possibly stderr?) the same problem occurs 
again and we just don't see any output.

I don't yet know why this causes the module build to fail, or why exit() 
doesn't work.

I think we should try to get this fixed before 3.0.1.  Any help would be 
welcomed.
msg77325 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-12-08 16:48
Could someone please try the attached patch and see if it solves the 
problem.

I'm not sure whether Facundo's problem is the same issue or not.
msg77616 - (view) Author: Stuart Woodward (stuartcw) Date: 2008-12-11 15:28
I'm running on Mac OS X. I applied the patch issue2173.patch and it
solved the "make: *** [sharedmods] Error 1" problem. A cursory test of
/usr/local/bin/python3.0 was successful.
msg77623 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-12-11 18:09
Thanks, Stuart!

I've committed the fix in r67703 and r67704, so it should appear in 3.0.1.  
This should fix the silent build failure, and the zero-output interpreter.

The behaviour is still not ideal:  a bad LC_CTYPE setting causes 
sys.stdout and sys.stdin to have their encodings set to 'ascii'.  'UTF-8' 
would probably be a better guess, on OS X.  But this is a lesser problem.  
To fix it, we should try to understand why nl_langinfo is returning an 
empty string in the first place.  But maybe only Apple knows the answer to 
that one. :)
History
Date User Action Args
2008-12-11 18:13:31mark.dickinsonsetstatus: open -> closed
resolution: fixed
2008-12-11 18:09:35mark.dickinsonsetmessages: + msg77623
2008-12-11 15:28:43stuartcwsetnosy: + stuartcw
messages: + msg77616
2008-12-08 16:48:15mark.dickinsonsetfiles: + issue2173.patch
keywords: + patch
messages: + msg77325
versions: + Python 3.1
2008-12-08 16:44:10mark.dickinsonsetnosy: + nelis
2008-12-08 16:23:54mark.dickinsonsetpriority: normal -> critical
messages: + msg77321
2008-12-08 16:13:58mark.dickinsonsetnosy: + mark.dickinson
messages: + msg77318
2008-11-13 06:12:55wangchunsetnosy: + wangchun
messages: + msg75813
2008-11-11 03:25:28vstinnersetnosy: + vstinner
messages: + msg75726
2008-03-20 03:33:15jafosetpriority: normal
2008-02-26 19:08:09georg.brandlsettitle: Pyton fails silently on bad locale -> Python fails silently on bad locale
2008-02-26 14:31:43mottenedersetfiles: + make.log
messages: + msg63044
2008-02-26 14:26:51mottenedersetfiles: + make.log
messages: + msg63042
2008-02-26 12:19:07facundobatistasetfiles: + full_make_output.txt
nosy: + facundobatista
messages: + msg63039
2008-02-24 10:26:29mottenedercreate