test_cmd_line fails on an android emulator running an x86 system image at API level 21.


======================================================================
FAIL: test_non_ascii (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_cmd_line.py", line 141, in test_non_ascii
    assert_python_ok('-c', command)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 135, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 121, in _assert_python
    err))
AssertionError: Process return code is 1
command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', '-c', "assert(ord('\xe6') == 230)"]

stdout:
---

---

stderr:
---
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
---

----------------------------------------------------------------------
Ran 33 tests in 1.943s

FAILED (failures=1, skipped=1)
test test_cmd_line failed
1 test failed:
    test_cmd_line
Total duration: 0:00:03

There is a related issue #4388 although different, and issue #22747 may also be related.

This patch follows Victor suggestion in msg230407 and brings the changes made in issue #4388 and issue #16416 to the Android platform. As a consequence, it adds a new test for Android in test_cmd_line named test_osx_android_utf8 (was previously test_osx_utf8). The platform.android_ver() function is defined by issue #26855: "add platform.android_ver() for android".

With this patch test_cmd_line does not have any failure.

Patch updated after vadmium review.

I commented retrofit_osx_2.patch on the review.

This new patch adds a dependency to issue #27027.

An interactive session confirms that the problem is indeed with the command line arguments of python invoked by subprocess (and the problem is fixed by the patch):

>>> from test.support import FS_NONASCII
>>> cmd = "assert(ord(%r) == %s)" % (FS_NONASCII, ord(FS_NONASCII))
>>> exec(cmd)
>>> import subprocess, sys
>>> subprocess.run([sys.executable, '-c', cmd])
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
CompletedProcess(args=['/data/data/org.bitbucket.pyona/python/bin/python', '-c', "assert(ord('\xe6') == 230)"], returncode=1)

The patch fixes also the following errors in test_warnings:

======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_warnings/__init__.py", line 1072, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_warnings/__init__.py", line 1072, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

----------------------------------------------------------------------

New changeset e5360d413ce4 by Xavier de Gaye in branch '3.6':
Issue #26919: On Android, operating system data is now always encoded/decoded
https://hg.python.org/cpython/rev/e5360d413ce4

Forgot the issue number in the commit message for the default branch, the corresponding changeset is 80a041d39c20.