This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: io.open() fails to open ascii file if LANG env not set
Type: behavior Stage: patch review
Components: IO Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, r.david.murray, ronaldoussoren, sentinel, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2015-10-09 22:38 by sentinel, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
io.openBugEx.py sentinel, 2015-10-09 22:38 simple case recreating the exception
Pull Requests
URL Status Linked Edit
PR 3712 merged serhiy.storchaka, 2017-09-23 21:34
PR 3714 merged python-dev, 2017-09-23 23:50
PR 3779 merged serhiy.storchaka, 2017-09-27 05:30
Messages (14)
msg252693 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-10 06:29
Can't reproduce on Linux.

$ python2.7 io.openBugEx.py
Local pref:  UTF-8
LANG env:  uk_UA.utf8
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
Local pref:  ANSI_X3.4-1968
LANG env has key:  False
ANSI_X3.4-1968
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL

What is your OS?
msg252861 - (view) Author: Lloyd Carothers (sentinel) * Date: 2015-10-12 15:29
Mac OSX 10.8.5
Python 2.7.10 installed via macports

$ python2.7 io.openBugEx.py
Local pref:  UTF-8
LANG env:  en_US.UTF-8
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
Local pref:
LANG env has key:  False

Traceback (most recent call last):
  File "io.openBugEx.py", line 21, in <module>
    print io.open(tmpfile).read()
LookupError: unknown encoding:
msg252865 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-12 15:48
I cannot reproduce this on 10.10.3:

rdmurray@intel-vm-buildbot:~>sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10.3
BuildVersion:   14D136

rdmurray@intel-vm-buildbot:~>LANG=en_us.UTF-8 python temp
Local pref:  UTF-8
LANG env:  en_us.UTF-8
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
Local pref:  UTF-8
LANG env has key:  False
UTF-8
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL

This is using the installed Apple python, but I get the same result using a recent local 2.7 build.
msg252868 - (view) Author: Lloyd Carothers (sentinel) * Date: 2015-10-12 16:16
The difference, and I believe the crux of the issue is that locale.getprefferedencoding() is still returning and encoding for you, mine returns and emtpy str.

Do you have any other envs set that getprefferedencoding my be using:
 'LANGUAGE', 'LC_ALL', 'LC_CTYPE', and 'LANG'
Perhaps if you wipe all of these?
msg252873 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-10-12 16:59
I can reproduce the problem using the MacPorts python2.7 but not the Apple-supplied Python 2.7 or the python.org one.  The difference seems to be that the MacPorts Python is built with the MacPorts-supplied GNU gettext and its libintl, whereas the others aren't (since Apple does not ship libintl).  I don't have time at the moment to dig into why the libintl version of setlocale retunes '' instead of 'US-ASCII'.  Perhaps locale.py or io could handle that case better?
msg252874 - (view) Author: Lloyd Carothers (sentinel) * Date: 2015-10-12 17:31
Should I submit elsewhere?
msg252876 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-10-12 17:59
Well, it might be a bug in libintl, but it would be a good idea to investigate further before reporting it upstream to them.  And we may want to deal with the possibility of a blank return regardless.
msg252878 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-12 18:12
Please don't change the type of the issue to "crash". This is different type of issues. See classification in https://docs.python.org/devguide/triaging.html#type.
msg252879 - (view) Author: Lloyd Carothers (sentinel) * Date: 2015-10-12 18:19
Type changed from cash to behavior.
msg302811 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-23 21:42
There are few issues here.

1. Missed "goto error" after setting an exception in TextIOWrapper constructor. This leads to raising wrong and less informative exception (and maybe worse in Python 3, but this is harder to reproduce).

2. getpreferredencoding() can return None in rare circumferences. But no code using it (except TextIOWrapper) expects this. What is better, fail or fallback to ASCII?
msg302821 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-23 23:50
New changeset d6238a76c655e0feb13478505220dc9049f1682f by Serhiy Storchaka in branch 'master':
bpo-25359: Add missed "goto error" after setting an exception. (#3712)
https://github.com/python/cpython/commit/d6238a76c655e0feb13478505220dc9049f1682f
msg302823 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-24 00:29
New changeset 7e32cee1a68a489c4f2e916ac7003004b51887de by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-25359: Add missed "goto error" after setting an exception. (GH-3712) (#3714)
https://github.com/python/cpython/commit/7e32cee1a68a489c4f2e916ac7003004b51887de
msg303095 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-27 05:44
New changeset 4954b8dc5305de72ce618522522a2910c3a34126 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-25359: Add missed "goto error" after setting an exception. (GH-3712) (#3779)
https://github.com/python/cpython/commit/4954b8dc5305de72ce618522522a2910c3a34126
msg303161 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-27 17:05
Possible related issues: issue6393 and issue30409.
History
Date User Action Args
2022-04-11 14:58:22adminsetgithub: 69546
2017-09-27 17:05:13serhiy.storchakasetmessages: + msg303161
2017-09-27 05:44:05serhiy.storchakasetmessages: + msg303095
2017-09-27 05:30:06serhiy.storchakasetpull_requests: + pull_request3766
2017-09-24 00:29:19serhiy.storchakasetmessages: + msg302823
2017-09-23 23:50:15python-devsetpull_requests: + pull_request3699
2017-09-23 23:50:01serhiy.storchakasetmessages: + msg302821
2017-09-23 21:42:07serhiy.storchakasetmessages: + msg302811
2017-09-23 21:34:28serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request3697
2015-10-12 18:19:05sentinelsetmessages: + msg252879
2015-10-12 18:12:56serhiy.storchakasettype: crash -> behavior
messages: + msg252878
2015-10-12 17:59:40r.david.murraysetmessages: + msg252876
2015-10-12 17:31:37sentinelsettype: behavior -> crash
messages: + msg252874
2015-10-12 16:59:23ned.deilysettype: crash -> behavior
messages: + msg252873
2015-10-12 16:16:19sentinelsettype: behavior -> crash
messages: + msg252868
components: - macOS
2015-10-12 15:48:52r.david.murraysettype: crash -> behavior
2015-10-12 15:48:43r.david.murraysetnosy: + r.david.murray, ronaldoussoren, ned.deily
messages: + msg252865
components: + macOS
2015-10-12 15:29:14sentinelsetstatus: pending -> open
type: behavior -> crash
messages: + msg252861
2015-10-10 06:29:36serhiy.storchakasetstatus: open -> pending

nosy: + serhiy.storchaka
messages: + msg252693

type: crash -> behavior
2015-10-09 22:38:34sentinelcreate