classification
Title: On FreeBSD, Python 3 doesn't support support the POSIX locale
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ncoghlan, vstinner
Priority: normal Keywords: patch

Created on 2018-08-28 09:35 by vstinner, last changed 2018-08-29 09:02 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8972 merged vstinner, 2018-08-28 09:51
PR 8974 merged vstinner, 2018-08-28 11:24
PR 8975 merged vstinner, 2018-08-28 12:07
Messages (7)
msg324230 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 09:35
vstinner@freebsd$ env -i LC_ALL=POSIX ./python -m test -v test_cmd_line_script  -m test_non_ascii
== CPython 3.8.0a0 (heads/master:096329f0b2, Aug 28 2018, 09:30:55) [Clang 4.0.0 (tags/RELEASE_400/final 297347)]
== FreeBSD-11.1-RELEASE-amd64-64bit-ELF little-endian
== cwd: /usr/home/vstinner/prog/python/master/build/test_python_13073
== CPU count: 2
== encodings: locale=US-ASCII, FS=ascii
Run tests sequentially
0:00:00 load avg: 0.48 [1/1] test_cmd_line_script
test_non_ascii (test.test_cmd_line_script.CmdLineTest) ... FAIL

======================================================================
FAIL: test_non_ascii (test.test_cmd_line_script.CmdLineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/vstinner/prog/python/master/Lib/test/test_cmd_line_script.py", line 512, in test_non_ascii
    'stdout=%r stderr=%r' % (stdout, stderr))
AssertionError: "'./@test_13073_tmp\\udce7w\\udcf0.py'" != "'<filename encoding error>'"
- './@test_13073_tmp\udce7w\udcf0.py'
+ '<filename encoding error>'
 : stdout=b"'<filename encoding error>'\n" stderr=b"Failed checking if argv[0] is an import path entry\nUnicodeEncodeError: 'ascii' codec can't encode character '\\xe7' in position 17: ordinal not in range(128)"

----------------------------------------------------------------------
Ran 1 test in 0.093s

FAILED (failures=1)
test test_cmd_line_script failed
test_cmd_line_script failed

== Tests result: FAILURE ==

1 test failed:
    test_cmd_line_script

Total duration: 170 ms
Tests result: FAILURE


By the way, the UTF-8 Mode is not enabled by the POSIX locale:

vstinner@freebsd$ env -i LC_ALL=POSIX ./python -c 'import sys; print(sys.flags.utf8_mode)'
0

It contradicts the PEP 540 which says the the UTF-8 Mode is enabled by the POSIX locale:

https://www.python.org/dev/peps/pep-0540/#abstract

> This mode is off by default, but is automatically activated when using the "POSIX" locale.
msg324234 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 10:35
New changeset 5cb258950ce9b69b1f65646431c464c0c17b1510 by Victor Stinner in branch 'master':
bpo-34527: POSIX locale enables the UTF-8 Mode (GH-8972)
https://github.com/python/cpython/commit/5cb258950ce9b69b1f65646431c464c0c17b1510
msg324240 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 11:51
New changeset 65ef7425a32ee411d8047a4fad0fc6bb9ff733b1 by Victor Stinner in branch '3.7':
bpo-34527: POSIX locale enables the UTF-8 Mode (GH-8972) (GH-8974)
https://github.com/python/cpython/commit/65ef7425a32ee411d8047a4fad0fc6bb9ff733b1
msg324241 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 12:09
Note: Python 2.7 is not affected: it doesn't have "force_ascii" and "LC_ALL=POSIX ./python -m test test_cmd_line_script" pass successfully.
msg324247 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 12:53
New changeset 4ff38870b1de8a3add5357edf125c2866bc42b54 by Victor Stinner in branch '3.6':
bpo-34527: Fix FreeBSD with POSIX locale (GH-8975)
https://github.com/python/cpython/commit/4ff38870b1de8a3add5357edf125c2866bc42b54
msg324250 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-28 13:06
I wanted to quickly fix this issue since I'm working on 3 different issues where I got conflicts with these changes, and the other issues were impacted by this bug.

I sent an email to python-dev about this issue:
https://mail.python.org/pipermail/python-dev/2018-August/155052.html
msg324310 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-29 09:02
See also bpo-34485. Example:

New changeset 0b9ea4b211b24464c7d38f63e45e51c275c52dcd by Victor Stinner in branch '3.7':
[3.7] bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986) (GH-8987)
https://github.com/python/cpython/commit/0b9ea4b211b24464c7d38f63e45e51c275c52dcd
History
Date User Action Args
2018-08-29 09:02:23vstinnersetmessages: + msg324310
2018-08-28 13:06:41vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg324250

stage: patch review -> resolved
2018-08-28 12:53:34vstinnersetmessages: + msg324247
2018-08-28 12:09:09vstinnersetmessages: + msg324241
versions: + Python 3.6
2018-08-28 12:07:05vstinnersetpull_requests: + pull_request8448
2018-08-28 11:51:24vstinnersetmessages: + msg324240
2018-08-28 11:24:59vstinnersetpull_requests: + pull_request8447
2018-08-28 10:35:46vstinnersetmessages: + msg324234
2018-08-28 09:51:17vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request8445
2018-08-28 09:35:24vstinnercreate