classification
Title: android: test_sys fails
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: xdegaye Nosy List: Alex.Willmer, lemburg, loewis, python-dev, serhiy.storchaka, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2016-05-03 09:26 by xdegaye, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
test_ioencoding_nonascii.patch xdegaye, 2016-05-14 08:02 review
test_c_locale_surrogateescape.patch xdegaye, 2016-05-14 09:40 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (12)
msg264705 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-03 09:26
test_sys fails on an android emulator running an x86 system image at API level 21.

...
test_implementation (test.test_sys.SysModuleTest) ... ok
test_intern (test.test_sys.SysModuleTest) ... ok
test_ioencoding (test.test_sys.SysModuleTest) ... ok
test_ioencoding_nonascii (test.test_sys.SysModuleTest) ... Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe6' in position 0: ordinal not in range(
128)
FAIL
test_is_finalizing (test.test_sys.SysModuleTest) ... ok
test_lost_displayhook (test.test_sys.SysModuleTest) ... ok
...
======================================================================
FAIL: test_c_locale_surrogateescape (test.test_sys.SysModuleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_sys.py", line 709, in test_c_locale_surr
ogateescape
    'stdin: surrogateescape\n'
AssertionError: 'stdin: strict\nstdout: strict\nstderr: backslashreplace\n' != 'stdin: surrogateesca
pe\nstdout: surrogateescape\nstderr: backslashreplace\n'
- stdin: strict
- stdout: strict
+ stdin: surrogateescape
+ stdout: surrogateescape
  stderr: backslashreplace


======================================================================
FAIL: test_ioencoding_nonascii (test.test_sys.SysModuleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_sys.py", line 638, in test_ioencoding_no
nascii
    self.assertEqual(out, os.fsencode(test.support.FS_NONASCII))
AssertionError: b'' != b'\xc3\xa6'

----------------------------------------------------------------------
Ran 43 tests in 0.913s

FAILED (failures=2, skipped=2)
test test_sys failed
1 test failed:
    test_sys
Total duration: 0:00:01
msg265463 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-13 11:15
test_ioencoding_nonascii does not fail when LANG is set to en_GB.UTF-8 in the environment.
msg265465 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-13 11:47
> test_ioencoding_nonascii does not fail when LANG is set to en_GB.UTF-8 in the environment.

When LANG is not set, we have on an android emulator:

>>> from test.support import FS_NONASCII
>>> print(FS_NONASCII)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe6' in position 0: ordinal not in range(128)


But on linux when LANG is not set:

$ LANG= ./python
Python 3.6.0a0 (default:eee959fee5f5+, May 13 2016, 11:32:27) 
[GCC 6.1.1 20160501] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from test.support import FS_NONASCII
>>> print(FS_NONASCII)
None
>>> 

And test_ioencoding_nonascii is skipped accordingly on linux for the following printed reason: 'requires OS support of non-ASCII encodings'.

On Android, os.fsdecode(os.fsencode('\xe6')) does not raise UnicodeError in the test.support module. And locale.getpreferredencoding() returns 'ascii'. So it seems that this criterion (FS_NONASCII is not None) is not sufficient to decide when this test should be run and not skipped.
msg265480 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-13 15:58
> On Android, os.fsdecode(os.fsencode('\xe6')) does not raise UnicodeError in the test.support module.

Because of changeset ad6be34ce8c9.
msg265509 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-14 08:02
This patch fixes test_ioencoding_nonascii.
msg265511 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-14 09:12
About the failures in test_c_locale_surrogateescape, it seems that on Android the locale is not set correctly on startup when LC_ALL is set to C:

root@generic_x86:/data/local/tmp # LC_ALL=C python
Python 3.6.0a0 (default:eee959fee5f5+, May 14 2016, 10:19:09) 
[GCC 4.9 20150123 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale; locale.getlocale()
('en_US', 'UTF-8')
>>> locale.setlocale(locale.LC_ALL, '')
'C'
>>> locale.getlocale()
(None, None)
>>>
msg265512 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-14 09:40
This patch fixes the locale setting on startup when the LC_ALL environment variable is set to C, and as a consequence test_c_locale_surrogateescape does not fail anymore. Note that on Android HAVE_LANGINFO_H is undefined, see issue #22747.
msg265518 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-14 10:55
See also issue19058.
msg265522 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-05-14 11:49
With Serhiy's latest patch named sys_test_ioencoding.patch in issue19058, test_sys fails only in test_c_locale_surrogateescape on the android-21-x86 emulator.
msg265548 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-05-14 22:45
test_c_locale_surrogateescape.patch LGTM.
msg279822 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-10-31 19:36
For the record, on the Android emulator we have now (not sure where this change has been made):
>>> sys.getfilesystemencoding()
'utf-8'
>>> locale.getpreferredencoding(False)
'ascii'

So test_ioencoding_nonascii succeeds now. Anyway the problem with this test is being fixed at issue 19058.
msg280917 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-11-16 06:35
New changeset 73bd6bbbb5e2 by Xavier de Gaye in branch '3.6':
Issue #26920: Fix not getting the locale's charset upon initializing the interpreter,
https://hg.python.org/cpython/rev/73bd6bbbb5e2

New changeset f358d849c14e by Xavier de Gaye in branch 'default':
Issue #26920: Merge 3.6
https://hg.python.org/cpython/rev/f358d849c14e
History
Date User Action Args
2017-03-31 16:36:16dstufftsetpull_requests: + pull_request908
2016-11-16 07:19:52xdegayesetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-11-16 06:35:11python-devsetnosy: + python-dev
messages: + msg280917
2016-10-31 19:44:08xdegayesetassignee: xdegaye
stage: commit review
components: + Interpreter Core, - Extension Modules, Cross-Build
versions: + Python 3.7
2016-10-31 19:36:07xdegayesetmessages: + msg279822
2016-05-21 07:06:39xdegayelinkissue26865 dependencies
2016-05-14 22:45:27vstinnersetmessages: + msg265548
2016-05-14 11:49:35xdegayesetmessages: + msg265522
2016-05-14 10:55:18serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg265518
2016-05-14 09:40:53xdegayesetfiles: + test_c_locale_surrogateescape.patch

messages: + msg265512
2016-05-14 09:12:11xdegayesetmessages: + msg265511
2016-05-14 08:02:19xdegayesetfiles: + test_ioencoding_nonascii.patch
keywords: + patch
messages: + msg265509
2016-05-13 15:58:25xdegayesetmessages: + msg265480
2016-05-13 11:47:38xdegayesetnosy: + vstinner
messages: + msg265465
2016-05-13 11:15:11xdegayesetmessages: + msg265463
2016-05-03 09:26:22xdegayecreate