classification
Title: test_subprocess test_undecodable_env error on AIX
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David.Edelsohn, Michael.Felt, gregory.p.smith, pitrou, sable, vstinner
Priority: normal Keywords: patch

Created on 2011-02-11 15:28 by sable, last changed 2018-08-27 00:19 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8939 merged Michael.Felt, 2018-08-26 13:18
PR 8946 merged miss-islington, 2018-08-27 00:19
Messages (12)
msg128402 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-11 15:28
The following tests fail in test_subprocess on AIX:

======================================================================
FAIL: test_undecodable_env (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/test/test_subprocess.py", line 1039, in test_undecodable_env
    self.assertEqual(stdout.decode('ascii'), ascii(value))
AssertionError: "'abc\\xff'" != "'abc\\udcff'"
- 'abc\xff'
?      ^
+ 'abc\udcff'
?      ^^^


======================================================================
FAIL: test_undecodable_env (test.test_subprocess.ProcessTestCasePOSIXPurePython)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/test/test_subprocess.py", line 1039, in test_undecodable_env
    self.assertEqual(stdout.decode('ascii'), ascii(value))
AssertionError: "'abc\\xff'" != "'abc\\udcff'"
- 'abc\xff'
?      ^
+ 'abc\udcff'
?      ^^^


----------------------------------------------------------------------
Ran 267 tests in 366.280s

FAILED (failures=2, skipped=22)

I haven't investigated yet.
msg128403 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-11 15:41
Perhaps related to the test_locale failure in issue11190.
msg128406 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-02-11 16:01
test_undecodable_code() in test_cmd_line had a similar issue: on FreeBSD, Solaris and Mac OS X, even if the locale is C (and nl_langinfo(CODESET) announces ASCII), _Py_char2wchar() (mbstowcs) decoded b'\xff' as '\xff' (use ISO-8859-1 encoding). To fix the test, I just patched the test to accept both results: b'\xff' may be decoded as '\udcff' or '\xff'.


test_undecodable_env does something like: os.environb[b'test']=b'abc\xff', but os.getenv() decodes b'abc\xff' as 'abc\xff' instead of 'abc\udcff' even if LC_ALL=C.

I don't understand why the test pass on FreeBSD, Solaris and Mac OS X, but not on AIX. It would be interesting to get the locale encoding of the child process using LC_ALL=C.
msg128543 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-02-14 13:23
> I don't understand why the test pass on FreeBSD, Solaris and
> Mac OS X, but not on AIX.

Oh, the command line and the filesystem encodings may be different. test_undecodable_env() of test_subprocess.py uses os.environ which uses sys.getfilesystemencoding(), whereas test_undecodable_code() of test_cmd_line.py uses _Py_char2wchar() which uses mbstowcs() functions (which use the locale encoding).

sys.getfilesystemencoding() is initialized from nl_langinfo(CODESET).

nl_langinfo(CODESET) gives maybe "ISO-8859-1" (or an alias to this encoding) even if LC_ALL=C.
msg128548 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-14 14:25
I am not sure this is what you want:

> LC_ALL=C ./python
Python 3.2rc3 (py3k:88417M, Feb 14 2011, 10:37:42) 
[GCC 4.2.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getfilesystemencoding()
'iso8859-1'
>>> 

What information can I provide to help solve this bug?
msg128549 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-02-14 14:32
>>> sys.getfilesystemencoding()
'iso8859-1'

Ok, I expected this result.

Can you also try:

$ LC_ALL=C ./python -c "import sys; print(ascii(sys.argv))" $(echo -ne "abc\xff")
['-c', 'abc\udcff']
$ LC_ALL=C python3.1 -c "import locale; print(locale.nl_langinfo(locale.CODESET))"
ANSI_X3.4-1968
$ LC_ALL=C python3.1 -c "import locale; print(locale.getpreferredencoding())"
ANSI_X3.4-1968

Anyway, test_undecodable_env() is not written to support ISO-8859-1 filesystem encoding. I should patch the test to check the filesystem encoding.
msg128555 - (view) Author: Sébastien Sablé (sable) Date: 2011-02-14 15:56
$ LANG=C ./python -Wd -E -bb -c "import sys; print(ascii(sys.argv))" $(echo -ne "abc\xff")['-c', 'abc\xff']
['-c', 'abc\xff']
$ LANG=C ./python -Wd -E -bb -c "import locale; print(locale.nl_langinfo(locale.CODESET))"
ISO8859-1
$ LANG=C ./python -Wd -E -bb -c "import locale; print(locale.getpreferredencoding())"
ISO8859-1
msg184314 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-03-16 13:43
ping myself
msg271281 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-07-25 15:16
Seems to be okay at least with Python 3.6 "test" version - but maybe there is better way to call these tests - just to be sure the one needed is not being skipped.

On AIX 5.3 TL7 SP0:

root@x064:[/data/prj/aixtools/python/python-3.6.0.162/Lib/test]../../python test_subprocess.py
ssss...........................................s.......................................................s.....s.........s...............................................................s.....s.........s.........................................sssssssssssss
----------------------------------------------------------------------
Ran 254 tests in 113.796s

OK (skipped=24)

On AIX 6.1 TL9 SP4:

michael@x071:[/data/prj/aixtools/python/python-3.6.0.162/Lib/test]../../python test_subprocess.py
ssss...........................................s.......................................................s.....s.........s...............................................................s.....s.........s.........................................sssssssssssss
----------------------------------------------------------------------
Ran 254 tests in 126.684s

OK (skipped=24)
michael@x071:[/data/prj/aixtools/python/python-3.6.0.162/Lib/test]

Please note that GNU libiconv is used by default (version 1.14) rather than IBM iconv.

This may be the key issue (will test 11190 as well)
msg324117 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-08-26 13:11
Short Version:
root@x065:[/data/prj/python/python3-3.8]./python -m test test_subprocess
Run tests sequentially
0:00:00 [1/1] test_subprocess
test_subprocess passed in 2 min 18 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 18 sec
Tests result: SUCCESS

...
----------------------------------------------------------------------
Ran 285 tests in 136.443s

OK (skipped=33)
test_subprocess passed in 2 min 16 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 16 sec
Tests result: SUCCESS

Posting a PR - tested manually on AIX 5.3, AIX 6.1 and AIX 7.1

imho: being able to remove the special condition for AIX is an improvement.

diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 73b57b21db..4719773b67 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -2228,15 +2228,9 @@ class POSIXProcessTestCase(BaseTestCase):
             env = os.environ.copy()
             env[key] = value
             # Use C locale to get ASCII for the locale encoding to force
-            # surrogate-escaping of \xFF in the child process; otherwise it can
-            # be decoded as-is if the default locale is latin-1.
+            # surrogate-escaping of \xFF in the child process
             env['LC_ALL'] = 'C'
-            if sys.platform.startswith("aix"):
-                # On AIX, the C locale uses the Latin1 encoding
-                decoded_value = encoded_value.decode("latin1", "surrogateescape")
-            else:
-                # On other UNIXes, the C locale uses the ASCII encoding
-                decoded_value = value
+            decoded_value = value
             stdout = subprocess.check_output(
                 [sys.executable, "-c", script],
                 env=env)
msg324128 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-08-26 17:31
someone with modern AIX access reopen if this specific issue still appears to exist after the 3.8 and 3.7 PRs finish merging.
msg324130 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-08-26 17:33
https://github.com/python/cpython/pull/8939 and https://github.com/python/cpython/pull/8946
History
Date User Action Args
2018-08-27 00:19:42miss-islingtonsetpull_requests: + pull_request8425
2018-08-26 17:33:54gregory.p.smithsetresolution: fixed
2018-08-26 17:33:33gregory.p.smithsetmessages: + msg324130
2018-08-26 17:31:29gregory.p.smithsetstatus: open -> closed


title: test_subprocess error on AIX -> test_subprocess test_undecodable_env error on AIX
nosy: + gregory.p.smith
versions: + Python 3.7
messages: + msg324128
stage: patch review -> resolved
2018-08-26 13:18:20Michael.Feltsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8412
2018-08-26 13:11:50Michael.Feltsetmessages: + msg324117
versions: + Python 3.8, - Python 3.2
2016-07-25 15:16:07Michael.Feltsetnosy: + Michael.Felt
messages: + msg271281
2013-06-19 21:23:56David.Edelsohnsetnosy: + David.Edelsohn
type: behavior
components: + Interpreter Core
2013-03-16 13:43:10vstinnersetmessages: + msg184314
2011-02-14 15:56:43sablesetnosy: pitrou, vstinner, sable
messages: + msg128555
2011-02-14 14:32:22vstinnersetnosy: pitrou, vstinner, sable
messages: + msg128549
2011-02-14 14:25:17sablesetnosy: pitrou, vstinner, sable
messages: + msg128548
2011-02-14 13:23:54vstinnersetnosy: pitrou, vstinner, sable
messages: + msg128543
2011-02-11 16:01:27vstinnersetnosy: pitrou, vstinner, sable
messages: + msg128406
2011-02-11 15:41:05pitrousetnosy: + vstinner, pitrou
messages: + msg128403
2011-02-11 15:28:17sablecreate