classification
Title: Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale
Type: behavior Stage:
Components: Unicode Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Sworddragon, a.badger, bkabrda, ezio.melotti, haypo, ishimoto, jwilk, larry, loewis, ncoghlan, pitrou, python-dev, r.david.murray, serhiy.storchaka, vadmium
Priority: normal Keywords: patch

Created on 2013-12-13 16:40 by haypo, last changed 2014-04-28 17:19 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
c_locale_surrogateescape.patch haypo, 2013-12-13 16:40 review
test_ls.py haypo, 2013-12-13 17:03
Messages (33)
msg206111 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-12-13 16:40
When LANG=C is used to get the english language (which is a mistake, LC_CTYPE=C should be used instead) or when Python is started with an empty environment (no environment variable), Python gets the POSIX locale (aka "C locale") for the LC_CTYPE (encoding) locale.

Standard streams use the locale encoding, which is usually ASCII with POSIX locale on most platforms (except on AIX: ISO 8859-1). In this case, data read from the OS (environment variables, command line arguments, filenames, etc.) may contain surrogate characters because of the internal usage of the surrogateescape error handler (see the PEP 383 for the rationale).

The problem is that standard output uses the strict error handler, and so print() fails to display OS data like filenames.

Example, "ls" command in Python:
---
import os
for name in sorted(os.listdir()): print(name)
---

Try it with "LANG=C python ls.py" in a directory containing non-ASCII characters and you will get unicode errors.

Issues #19846 and #19847 are examples of this annoyance.

I propose to use also the surrogateescape error handler for sys.stdout if the POSIX locale is used for LC_CTYPE at startup. Attached patch implements this idea.

With the patch, "LANG=C python ls.py" almost works as filenames and stdout are byte streams, even if the Unicode type is used.
msg206114 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-12-13 16:42
Oh, in fact, sys.stdin is also modified by the patch (as I expected).
msg206121 - (view) Author: (Sworddragon) Date: 2013-12-13 16:58
What would happen if we call this example script with LANG=C on the patch?:

---
import os
for name in sorted(os.listdir('ä')):
	print(name)
---

Would it throw an exception on os.listdir('ä')?
msg206122 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-12-13 17:03
test_ls.py: test script producing invalid filenames and then trying to display them into stdout.

Output with UTF-8 locale, UTF-8 terminal and Python 3.3 (or unpatched 3.4, it's the same):

ascii.txt
<UnicodeError 'invalid_utf8:\udcff.txt'>
<UnicodeError 'latin1:\udce9.txt'>
utf8:é€.txt

Output with C locale (ASCII), UTF-8 terminal and Python 3.3:

ascii.txt
<UnicodeError 'invalid_utf8:\udcff.txt'>
<UnicodeError 'latin1:\udce9.txt'>
<UnicodeError 'utf8:\udcc3\udca9\udce2\udc82\udcac.txt'>

Output with C locale (ASCII), UTF-8 terminal and patched Python 3.4:

ascii.txt
invalid_utf8:�.txt
latin1:�.txt
utf8:é€.txt

You get no Unicode error with LANG=C, but you get mojibake instead.
msg206123 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-12-13 17:08
os.fsencode(text) always fail if text cannot be encoded to sys.getfilesystemencoding(). surrogateescape doesn't help here.

Your example is "artificial", you should not get 'ä'. All OS data is decoded from the filesystem encoding using the surrogateescape error handler (except on Windows, where strict is used, but it's a different story, Python uses Unicode functions when available so don't worry). So all these data can always be encoded back to bytes using os.fsencode().

More generally, os.fsencode(os.fsdecode(read_data)) == read_data is always true on Unix, with any filesystem (locale) encoding.

You may get Unicode data from other sources like files or a GUI, but I don't see what can be done here.
msg206124 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-13 17:21
> When LANG=C is used to get the english language (which is a mistake,
> LC_CTYPE=C should be used instead)

I think you mean LC_MESSAGES=C here.
(but it's not only about the English language; it's also about other locale parameters such as number formatting)

I think we should start thinking about making utf-8 the default filesystem encoding in 3.5 (under Unix).
msg206131 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-12-13 17:59
Reintroducing moji-bake intentionally doesn't sound like a particularly good idea, wasn't that what python3 was supposed to help prevent?

It does seem like a utf-8 default is the Way of the Future.  Or even the present, most places.
msg206141 - (view) Author: Toshio Kuratomi (a.badger) * Date: 2013-12-13 19:44
My impression was that python3 was supposed to help get rid of UnicodeError tracebacks, not mojibake.  If mojibake was the problem then we should never have gone down the surrogateescape path for input.
msg206148 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-13 21:27
Mojibake in input can cause decoding error in other application which consumes output of Python script. In some cases this can be even worse thin UnicodeError in producer.

But for C locale this makes sense. I think we should try this experiment in 3.5. There will be much time for testing before 3.5 beta 1.
msg206168 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-14 06:07
Getting rid of mojibake was the goal, surrogateescape was about dealing with cases where the "avoid mojibake" checks were spuriously breaking round-tripping between OS APIs due to other configuration errors (with LANG=C being set, or LANG not being set at all being the main problem). Other "high mojibake risk" power tools (like changing the encoding of an already open stream) are likely to return in the future, since there *are* cases where they're the right answer (e.g. you can't right an iconv equivalent in Python 3 at the moment, we need issue 15216 implemented before that will be possible).

+1 for this solution - see issue 19846 for the long discussion which got us to this point (there are a few unrelated tangents, a couple of them my fault, but this is definitely an improvement over the status quo.
msg207210 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-01-03 06:13
Larry: I'm assuming it's way too late to make a change like this for the 3.4 release?

Slavek: assuming this change is made for 3.5 upstream, we may want to look at backporting it as a 3.4 patch in Fedora (as part of the Python-3-by-default project). Otherwise it's very easy to provoke Python 3 into throwing Unicode errors when attempting to print data provided by the OS.
msg207296 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-01-04 18:04
Yeah, unless there was a *huge* amount of support for changing this, it's way too late for 3.4.
msg207414 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2014-01-06 07:39
Nick: Sure, once there is an upstream solution that people have agreed on, I'll look into backporting it, NP. Thanks for letting me know about this.
msg210990 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-02-11 17:47
> Reintroducing moji-bake intentionally doesn't sound like a particularly good idea, wasn't that what python3 was supposed to help prevent?

Sometimes practicality beats purity :-(

I tried to convince users that their computer was "not well configured", they always replied that Python 3 fails where Perl, PHP, Python 2, C, etc. "just work".
msg213922 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-18 00:27
New changeset bc06f67234d0 by Victor Stinner in branch 'default':
Issue #19977: When the ``LC_TYPE`` locale is the POSIX locale (``C`` locale),
http://hg.python.org/cpython/rev/bc06f67234d0
msg213925 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-03-18 00:49
Test failing on "x86 OpenIndiana 3.x" buildbot:

http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/7939/steps/test/logs/stdio

======================================================================
FAIL: test_forced_io_encoding (test.test_capi.EmbeddingTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_capi.py", line 352, in test_forced_io_encoding
    self.assertEqual(out.strip(), expected_output)
AssertionError: '--- [79 chars]646:surrogateescape\nstdout: 646:surrogateesca[576 chars]lace' != '--- [79 chars]646:strict\nstdout: 646:strict\nstderr: 646:ba[540 chars]lace'
  --- Use defaults ---
  Expected encoding: default
  Expected errors: default
- stdin: 646:surrogateescape
- stdout: 646:surrogateescape
+ stdin: 646:strict
+ stdout: 646:strict
  stderr: 646:backslashreplace
  --- Set errors only ---
  Expected encoding: default
  Expected errors: surrogateescape
  stdin: 646:surrogateescape
  stdout: 646:surrogateescape
  stderr: 646:backslashreplace
  --- Set encoding only ---
  Expected encoding: latin-1
  Expected errors: default
- stdin: latin-1:surrogateescape
- stdout: latin-1:surrogateescape
+ stdin: latin-1:strict
+ stdout: latin-1:strict
  stderr: latin-1:backslashreplace
  --- Set encoding and errors ---
  Expected encoding: latin-1
  Expected errors: surrogateescape
  stdin: latin-1:surrogateescape
  stdout: latin-1:surrogateescape
  stderr: latin-1:backslashreplace
msg213927 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-03-18 00:57
New behaviour:

$ mkdir z
$ touch z/abcé
$ LC_CTYPE=C ./python -c 'import os; print(os.listdir("z")[0])'
abcé

Old behaviour, before the change (test with Python 3.3):

$ LC_CTYPE=C python3 -c 'import os; print(os.listdir("z")[0])'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 3-4: ordinal not in range(128)
msg213932 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-18 01:32
New changeset 3589980c98de by Victor Stinner in branch 'default':
Issue #19977, #19036: Always include <locale.h> in pythonrun.c
http://hg.python.org/cpython/rev/3589980c98de

New changeset 94d5025c70a3 by Victor Stinner in branch 'default':
Issue #19977: Enable test_c_locale_surrogateescape() on Windows
http://hg.python.org/cpython/rev/94d5025c70a3
msg213933 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-18 01:38
New changeset c9905e802042 by Victor Stinner in branch 'default':
Issue #19977: Fix test_capi when LC_CTYPE locale is POSIX
http://hg.python.org/cpython/rev/c9905e802042
msg215029 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-03-28 10:01
This seems to be working on the buildbots for 3.5 now (buildbot failures appear to be due to other issues).

However, I'd still like to discuss the idea of backporting this to 3.4.1.

From a Fedora point of view, it's still *very* easy to flip an environment into POSIX mode, so even if the system is appropriately configured to use UTF-8 everywhere, Python 3.4 may still blow up if a script or application ends up running under the POSIX locale.

That has long made Toshio nervous about the migration of core services to Python 3 (https://fedoraproject.org/wiki/Changes/Python_3_as_Default), and his concerns make sense to me, as that migration covers little things like the installer, package manager, post-image install initialisation, etc. I'm not sure the Fedora team can deliver on the "Users shouldn't notice any changes, except that packages in minimal buildroot and on LiveCD will be python3-, not python-." aspect of the change proposal without this behavioural tweak in the 3.4 series as well.

Note that this *isn't* a blocker for the migration - if it was, it would be mentioned in the Fedora proposal. However, I think there's a risk to the Fedora user experience if the status quo remains in place for the life of Python 3.4, and I'd hate for the first encounter Fedora users have with Python 3 to be inexplicable tracebacks from components that have been migrated.
msg215786 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-09 00:28
"However, I'd still like to discuss the idea of backporting this to 3.4.1."

THe idea of doing this change in Python 3.5 is that I have no idea of the risk of regression. To backport such change in a minor version (3.4.1), I would feel more confident with user tests of Python 3.5 or patched Python 3.4.

"That has long made Toshio nervous about the migration of core services to Python 3 (https://fedoraproject.org/wiki/Changes/Python_3_as_Default), and his concerns make sense to me, as that migration covers little things like the installer, package manager, post-image install initialisation, etc. "

Which programs in this test are or may be running with the POSIX locale?

Fedora doesn't use en_US.utf8 locale by default?
msg215812 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-09 10:52
The default locale on Fedora is indeed UTF-8 these days - the problem is that *users* are used to being able to use "LANG=C" to force the POSIX locale (whether for testing purposes or other reasons), and that currently means system utilities written in Python may fail in such situations if used with UTF-8 data from the filesystem (or elsewhere). (I believe there may also be other cases where POSIX mandates the use of the C locale, but Toshio would be in a better position than I am to confirm whether or not that is actually the case).

So perhaps this is best left in a "wait & see" mode for now - as the Fedora migration to Python 3 progresses, if the folks working on that find specific utilities where the Python 3.4 standard stream handling in the C locale appears problematic, then Slavek & Toshio can bring them up here.

The counterargument is that if we're going to change it, 3.4.1 would be a better time frame than 3.4.2. In that case, the task of identifying specific Fedora utilities of concern still falls back on Toshio & Slavek, but it would be a matter of going hunting for them specifically *now*, rather than waiting until they come up over the course of the migration.
msg215815 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-09 11:08
> The default locale on Fedora is indeed UTF-8 these days - the problem is that *users* are used to being able to use "LANG=C" to force the POSIX locale (whether for testing purposes or other reasons), and that currently means system utilities written in Python may fail in such situations if used with UTF-8 data from the filesystem (or elsewhere). (I believe there may also be other cases where POSIX mandates the use of the C locale, but Toshio would be in a better position than I am to confirm whether or not that is actually the case).

A common situation where you get a C locale is for programs started by
a crontab. If I remember correctly, these programs start with the C
locale, instead of the "system" (user?) locale.
msg217300 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-27 18:13
Additional environments where the system misreports the encoding to use (courtesy of Armin Ronacher & Graham Dumpleton on Twitter): upstart, Salt, mod_wsgi.

Note that for more complex applications (e.g. integrated web UIs, socket servers, sending email), round tripping to the standard streams won't be enough - what we really need is a better "source of truth" as to the real system encoding when POSIX compliant systems provide incorrect configuration data to the interpreter.
msg217314 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-27 19:34
Issue 21368 now suggests looking for /etc/locale.conf before falling back to ASCII+surrogateescape.
msg217315 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-27 19:48
We should not overcomplicate this. I suggest that we simply use utf-8 under the C locale.
msg217317 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-27 20:47
If you can convince Stephen Turnbull that's a good idea, sure. It's
probably more likely to be the right thing than "ASCII" or "ASCII +
surrogateescape", but in the absence of hard data, he's in a better
position than we are to judge the likely impact of that, at least in Japan.

I'm also going to hunt around on freedesktop.org to see if there's anything
more general there on the topic of encodings.
msg217329 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-27 23:33
> We should not overcomplicate this. I suggest that we simply use utf-8 under the C locale.

Do you mean utf8/strict or utf8/surrogateescape?

utf8/strict doesn't work (os.listdir raises an unicode error) if your
system is configured to use latin1 (ex: filenames are stored in this
encoding), but unfortunately your program is running in an empty
environment (so will use the POSIX locale).
msg217332 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-27 23:57
> We should not overcomplicate this. I suggest that we simply use utf-8 under the C locale.

Please open a new issue if you would prefer UTF-8. You will have to solve different technical issues. I tried to list some of them in issues #19846 and #19847.

In short, you should always decode and encode "OS data" with the same encoding. Python "file system encoding" is the locale encoding because in some places, PyUnicode_DecodeLocale[AndSize]() is used (ex: to decode PYTHONWARNINGS environment variable). A common location is PyUnicode_DecodeFSDefaultAndSize() before the Python codec is loaded. See also _Py_wchar2char() and _Py_char2wchar() functions which use the locale encoding and are used in many places.

I'm now closing the issue because the initial point (use surrogateescape error handler) is implemented in Python 3.5, and backporting such major change in Python 3.4 branch is risky right now.
msg217355 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-28 10:07
> > We should not overcomplicate this. I suggest that we simply use utf-8 under the C locale.
> 
> Do you mean utf8/strict or utf8/surrogateescape?
> 
> utf8/strict doesn't work (os.listdir raises an unicode error) if your
> system is configured to use latin1 (ex: filenames are stored in this
> encoding), but unfortunately your program is running in an empty
> environment (so will use the POSIX locale).

The issue is about stdin and stdout, I'm not sure why os.listdir would
be affected.
msg217385 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-28 17:11
Victor was referring to code like "print(os.listdir())". Those are the
motivating cases for ensuring round trips from system APIs to the standard
streams work correctly.

There's also the problem that sys.argv currently relies on the locale
encoding directly, because the filesystem encoding hasn't been worked out
at that point (see issue 8776). So this current change will also make
"print(sys.argv)" work more reliably in the POSIX locale.

The conclusion I have come to is that any further decoupling of Python 3
from the locale encoding will actually depend on getting the PEP 432
bootstrapping changes implemented, reviewed and the PEP approved, so we
have more interpreter infrastructure in place by the time the interpreter
starts trying to figure out all these boundary encoding issues.
msg217386 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-28 17:13
> The conclusion I have come to is that any further decoupling of Python 3
> from the locale encoding will actually depend on getting the PEP 432
> bootstrapping changes implemented, reviewed and the PEP approved, so we
> have more interpreter infrastructure in place by the time the interpreter
> starts trying to figure out all these boundary encoding issues.

Yeah. My proposal had more to do with the fact that we should some day
switch to utf-8 by default on all POSIX systems, regardless of what the
system advertises as "best encoding".
msg217387 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-28 17:19
Antoine Pitrou added the comment:
> Yeah. My proposal had more to do with the fact that we should some day
> switch to utf-8 by default on all POSIX systems, regardless of what the
> system advertises as "best encoding".

Yeah, that seems like a plausible future to me as well, and knowing it's a
step along that path actually gives me more motivation to get back to
working on the startup issues :)
History
Date User Action Args
2014-04-28 17:19:54ncoghlansetmessages: + msg217387
2014-04-28 17:13:17pitrousetmessages: + msg217386
2014-04-28 17:11:25ncoghlansetmessages: + msg217385
2014-04-28 10:07:29pitrousetmessages: + msg217355
2014-04-27 23:57:39hayposetstatus: open -> closed
resolution: fixed
messages: + msg217332
2014-04-27 23:33:57hayposetmessages: + msg217329
2014-04-27 20:47:39ncoghlansetmessages: + msg217317
2014-04-27 19:48:32pitrousetmessages: + msg217315
versions: + Python 3.5, - Python 3.4
2014-04-27 19:34:05ncoghlansetmessages: + msg217314
2014-04-27 18:13:18ncoghlansetmessages: + msg217300
2014-04-09 11:08:44hayposetmessages: + msg215815
2014-04-09 10:52:05ncoghlansetmessages: + msg215812
2014-04-09 00:28:33hayposetmessages: + msg215786
2014-03-28 10:01:26ncoghlansetmessages: + msg215029
versions: + Python 3.4, - Python 3.5
2014-03-20 00:48:13ishimotosetnosy: + ishimoto
2014-03-18 01:38:21python-devsetmessages: + msg213933
2014-03-18 01:32:24python-devsetmessages: + msg213932
2014-03-18 00:57:46hayposetmessages: + msg213927
2014-03-18 00:49:09hayposetmessages: + msg213925
2014-03-18 00:27:00python-devsetnosy: + python-dev
messages: + msg213922
2014-02-11 17:47:46hayposetmessages: + msg210990
2014-01-22 22:41:50ncoghlanunlinkissue20329 superseder
2014-01-22 22:36:58ncoghlanlinkissue20329 superseder
2014-01-06 07:39:50bkabrdasetmessages: + msg207414
2014-01-04 18:04:26larrysetmessages: + msg207296
2014-01-03 06:13:54ncoghlansetnosy: + larry, bkabrda
type: behavior
messages: + msg207210
2013-12-21 17:10:37jwilksetnosy: + jwilk
2013-12-20 05:03:54vadmiumsetnosy: + vadmium
2013-12-14 06:07:30ncoghlansetmessages: + msg206168
2013-12-13 21:27:00serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg206148
2013-12-13 19:44:36a.badgersetmessages: + msg206141
2013-12-13 17:59:51r.david.murraysetmessages: + msg206131
2013-12-13 17:21:05pitrousetnosy: + pitrou
messages: + msg206124
2013-12-13 17:13:47pitrousetversions: + Python 3.5, - Python 3.4
2013-12-13 17:08:27hayposetmessages: + msg206123
2013-12-13 17:03:54hayposetfiles: + test_ls.py

messages: + msg206122
2013-12-13 16:58:57Sworddragonsetmessages: + msg206121
2013-12-13 16:56:17r.david.murraysetnosy: + r.david.murray
2013-12-13 16:42:18hayposetmessages: + msg206114
title: Use "surrogateescape" error handler for sys.stdout on UNIX for the C locale -> Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale
2013-12-13 16:40:20haypocreate