classification
Title: on Android python fails to decode/encode command line arguments
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: 27027 Superseder:
Assigned To: xdegaye Nosy List: Alex.Willmer, python-dev, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2016-05-03 09:18 by xdegaye, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
retrofit_osx.patch xdegaye, 2016-05-13 09:07 review
retrofit_osx_2.patch xdegaye, 2016-05-13 11:11 test description changed review
retrofit_osx_3.patch xdegaye, 2016-05-20 09:08 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (10)
msg264703 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-03 09:18
test_cmd_line fails on an android emulator running an x86 system image at API level 21.


======================================================================
FAIL: test_non_ascii (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_cmd_line.py", line 141, in test_non_ascii
    assert_python_ok('-c', command)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 135, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 121, in _assert_python
    err))
AssertionError: Process return code is 1
command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', '-c', "assert(ord('\xe6') == 230)"]

stdout:
---

---

stderr:
---
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
---

----------------------------------------------------------------------
Ran 33 tests in 1.943s

FAILED (failures=1, skipped=1)
test test_cmd_line failed
1 test failed:
    test_cmd_line
Total duration: 0:00:03
msg264800 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-04 10:07
There is a related issue #4388 although different, and issue #22747 may also be related.
msg265458 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-13 09:07
This patch follows Victor suggestion in msg230407 and brings the changes made in issue #4388 and issue #16416 to the Android platform. As a consequence, it adds a new test for Android in test_cmd_line named test_osx_android_utf8 (was previously test_osx_utf8). The platform.android_ver() function is defined by issue #26855: "add platform.android_ver() for android".

With this patch test_cmd_line does not have any failure.
msg265462 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-13 11:11
Patch updated after vadmium review.
msg265550 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-05-14 22:50
I commented retrofit_osx_2.patch on the review.
msg265916 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-20 09:08
This new patch adds a dependency to issue #27027.
msg279803 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-31 16:43
An interactive session confirms that the problem is indeed with the command line arguments of python invoked by subprocess (and the problem is fixed by the patch):

>>> from test.support import FS_NONASCII
>>> cmd = "assert(ord(%r) == %s)" % (FS_NONASCII, ord(FS_NONASCII))
>>> exec(cmd)
>>> import subprocess, sys
>>> subprocess.run([sys.executable, '-c', cmd])
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
CompletedProcess(args=['/data/data/org.bitbucket.pyona/python/bin/python', '-c', "assert(ord('\xe6') == 230)"], returncode=1)
msg283340 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-12-15 19:41
The patch fixes also the following errors in test_warnings:

======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_warnings/__init__.py", line 1072, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_warnings/__init__.py", line 1072, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcc3\\udcb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

----------------------------------------------------------------------
msg283341 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-15 20:02
New changeset e5360d413ce4 by Xavier de Gaye in branch '3.6':
Issue #26919: On Android, operating system data is now always encoded/decoded
https://hg.python.org/cpython/rev/e5360d413ce4
msg283342 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-12-15 20:09
Forgot the issue number in the commit message for the default branch, the corresponding changeset is 80a041d39c20.
History
Date User Action Args
2017-03-31 16:36:20dstufftsetpull_requests: + pull_request937
2016-12-15 20:37:29xdegayesetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-12-15 20:09:59xdegayesetmessages: + msg283342
2016-12-15 20:02:34python-devsetnosy: + python-dev
messages: + msg283341
2016-12-15 19:41:26xdegayesetmessages: + msg283340
2016-10-31 16:54:40xdegayesettitle: android: test_cmd_line fails -> on Android python fails to decode/encode command line arguments
2016-10-31 16:45:25xdegayesetassignee: xdegaye
stage: commit review
components: + Interpreter Core, - Library (Lib), Cross-Build
versions: + Python 3.7
2016-10-31 16:43:59xdegayesetmessages: + msg279803
2016-05-21 07:06:39xdegayelinkissue26865 dependencies
2016-05-21 06:55:00xdegayesetdependencies: + add the 'is_android' attribute to test.support
2016-05-20 09:08:50xdegayesetfiles: + retrofit_osx_3.patch

messages: + msg265916
2016-05-14 22:50:57vstinnersetmessages: + msg265550
2016-05-13 11:11:54xdegayesetfiles: + retrofit_osx_2.patch

messages: + msg265462
2016-05-13 09:07:35xdegayesetfiles: + retrofit_osx.patch
keywords: + patch
messages: + msg265458
2016-05-04 10:07:06xdegayesetnosy: + vstinner
messages: + msg264800
2016-05-03 09:18:35xdegayecreate