android: test_ctypes crashes on armv7 and aarch64 #71129

xdegaye · 2016-05-03T15:53:09Z

BPO	26942
Nosy	@amauryfa, @abalkin, @meadori, @xdegaye, @moreati, @yan12125
Files	libffi-pr240.patch libffi-pr240.patch: Version 2

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-10-16.09:54:15.941>
created_at = <Date 2016-05-03.15:53:09.471>
labels = ['build', 'library', 'type-crash']
title = 'android: test_ctypes crashes on armv7 and aarch64'
updated_at = <Date 2016-10-16.09:54:15.941>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2016-10-16.09:54:15.941>
actor = 'xdegaye'
assignee = 'none'
closed = True
closed_date = <Date 2016-10-16.09:54:15.941>
closer = 'xdegaye'
components = ['Library (Lib)', 'Cross-Build']
creation = <Date 2016-05-03.15:53:09.471>
creator = 'xdegaye'
dependencies = []
files = ['43894', '43945']
hgrepos = []
issue_num = 26942
keywords = ['patch']
message_count = 15.0
messages = ['264745', '265823', '265829', '270032', '271355', '271357', '271367', '271368', '271392', '271649', '271662', '271675', '275584', '278759', '278760']
nosy_count = 6.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'meador.inge', 'xdegaye', 'Alex.Willmer', 'yan12125']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue26942'
versions = ['Python 3.6']

xdegaye · 2016-05-03T15:53:09Z

test_ctypes crashes on an android emulator running an armv7 system image (but not on x86) at API level 21.

143|root@generic:/data/local/tmp # python -m test -v test_ctypes
== CPython 3.6.0a0 (default:f4c6dab59cd8+, May 3 2016, 17:24:17) [GCC 4.9 20140827 (prerelease)]
== Linux-3.4.67-01422-gd3ffcc7-dirty-armv7l-with-libc little-endian
== hash algorithm: fnv 32bit
== /data/local/tmp/test_python_2301
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0,
no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomiza
tion=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_ctypes
test_anon (ctypes.test.test_anon.AnonTest) ... ok
test_anon_nonmember (ctypes.test.test_anon.AnonTest) ... ok
test_anon_nonseq (ctypes.test.test_anon.AnonTest) ... ok
test_nested (ctypes.test.test_anon.AnonTest) ... ok
test (ctypes.test.test_array_in_pointer.Test) ... ok
test_2 (ctypes.test.test_array_in_pointer.Test) ... ok
test_bad_subclass (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_cache (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_classcache (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_from_address (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_from_addressW (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_numeric_arrays (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_simple (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_subclass (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_byval (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ... ok
test_callbacks (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ... Fatal Python error
: Segmentation fault

Current thread 0xb6f2eec8 (most recent call first):
File "/sdcard/org.bitbucket.pyona/lib/python3.6/ctypes/test/test_as_parameter.py", line 85 in test
_callbacks
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/case.py", line 600 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/case.py", line 648 in __call__
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/runner.py", line 176 in run
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/init.py", line 1802 in _run_suite
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/init.py", line 1836 in run_unitte
st
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 166 in test_run
ner
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 167 in runtest_
inner
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 131 in runtest
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 332 in run_tests_s
equential
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 402 in run_tests
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 462 in _main
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 442 in main
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 504 in main
File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/main.py", line 2 in <module>
File "/sdcard/org.bitbucket.pyona/lib/python3.6/runpy.py", line 85 in _run_code
File "/sdcard/org.bitbucket.pyona/lib/python3.6/runpy.py", line 184 in _run_module_as_main
Segmentation fault

xdegaye · 2016-05-18T10:34:27Z

The crash occurs at the same line that the crash reported in issue
bpo-17786. Line 85 in ctypes/test/test_as_parameter.py was line 87 at changeset ae5c4a9118b8a3f490f77f2084d46163ca229aef.

xdegaye · 2016-05-18T14:57:49Z

Running the following interactive statements [1]:

>> import unittest, ctypes.test.test_as_parameter
>> unittest.main(module=ctypes.test.test_as_parameter, defaultTest='BasicWrapTestCase', verbosity=2)

The corresponding attached gdb session:

(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0xb6a181a0 in ?? ()
(gdb) bt 7
#0 0xb6a181a0 in ?? ()
#1 0xb605de7a in _testfunc_callback_i_if (value=-10, func=0xb6a181a0)
at cpython/Modules/_ctypes/_ctypes_test.c:234
#2 0xb6072238 in ffi_call_SYSV ()
at cpython/Modules/_ctypes/libffi/src/arm/sysv.S:188
#3 0xb6072a52 in ffi_call (cif=cif@entry=0xbe91ac04,
fn=fn@entry=0xb605de6d <_testfunc_callback_i_if>, rvalue=rvalue@entry=0xbe91ac88,
avalue=avalue@entry=0xbe91ac78)
at cpython/Modules/_ctypes/libffi/src/arm/ffi.c:339
#4 0xb606e2a0 in _call_function_pointer (flags=flags@entry=-1240871416,
pProc=0xb605de6d <_testfunc_callback_i_if>, pProc@entry=0xb606956b <PyCFuncPtr_call+350>,
avalues=avalues@entry=0xbe91ac78, atypes=atypes@entry=0xbe91ac68, restype=0xb60db8e0,
resmem=resmem@entry=0xbe91ac88, argcount=argcount@entry=2)
at cpython/Modules/_ctypes/callproc.c:811
#5 0xb606e894 in _ctypes_callproc (pProc=pProc@entry=0xb605de6d <_testfunc_callback_i_if>,
argtuple=argtuple@entry=(<AsParamPropertyWrapper(_param=-10) at remote 0xb5f012d8>, <AsParamPropertyWrapper(_param=<CFunctionType at remote 0xb5ef5c70>) at remote 0xb5ef1fc0>),
flags=<optimized out>, argtypes=argtypes@entry=0x0, restype=<optimized out>,
restype@entry=<_ctypes.PyCSimpleType at remote 0xb6097228>, checker=checker@entry=0x0)
at cpython/Modules/_ctypes/callproc.c:1149
#6 0xb606956a in PyCFuncPtr_call (self=0xb5ef5e90, inargs=<optimized out>, kwds=<optimized out>)
at cpython/Modules/_ctypes/_ctypes.c:3856
(More stack frames follow...)
(gdb) up
#1 0xb605de7a in _testfunc_callback_i_if (value=-10, func=0xb6a181a0)
at cpython/Modules/_ctypes/_ctypes_test.c:234
234 sum += func(value);
(gdb) list
229
230 EXPORT(int) _testfunc_callback_i_if(int value, int (func)(int))
231 {
232 int sum = 0;
233 while (value != 0) {
234 sum += func(value);
235 value /= 2;
236 }
237 return sum;
238 }
(gdb) p func
$2 = (int ()(int)) 0xb6a181a0
(gdb) disassemble func
No function contains specified address.
(gdb)

[1] Gdb is attached after the import statement because gdb fails with SIGILL in __dl_notify_gdb_of_libraries both with arm and armv7, whenever a library is loaded. This is gdb from the Android ndk started by ndk-gdb.py from the r11c ndk. A workaround is to add the following lines in the gdbinit script:
break __dl_rtld_db_dlactivity
commands
silent
return
sharedlibrary
continue
end

yan12125 · 2016-07-09T09:08:13Z

Maybe a libffi issue. The crash is still with libffi git-master and CPython hg-tip + --with-system-libffi. Reported to libffi/libffi#262

yan12125 · 2016-07-26T10:14:33Z

Found libffi PR240 that fixes closures on Android:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -m test.test_ctypes
...................................................................s......s..............s....................ssssssssssssssssssssss..................................ssssssssssssssssssssssssssss.s...sssOpenGL libraries:
('GL', None)
('GLU', None)
('gle', None)
sss...........................s.s............s.........libc_name is None
ss.sssss...s...s...........s....s..................................................................s.................................sssss..ss..........................s....................ss.sssssss
----------------------------------------------------------------------
Ran 456 tests in 2.090s

OK (skipped=92)

See yan12125/python3-android@1daebca for details.

By http://comments.gmane.org/gmane.comp.lib.ffi.general/1235, SELinux affects the result, too. PR240 of libffi assumes SELinux is disabled. I have disabled SELinux on my phone for some root applications. I'm not sure whether PR240 works for phones with SELinux or not.

xdegaye · 2016-07-26T11:43:02Z

Thanks for looking into that problem.
Can you provide a patch ?

yan12125 · 2016-07-26T13:56:31Z

By msg264746, only ARM fails, so I patch libffi for arm and aarch64 triplets only

yan12125 · 2016-07-26T14:00:32Z

Test results against patched libffi in Modules/_ctypes:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -m test.test_ctypes
...................................................................s......s..............s....................ssssssssssssssssssssss..................................ssssssssssssssssssssssssssss.s...sssOpenGL libraries:
('GL', None)
('GLU', None)
('gle', None)
sss...........................s.s............s.........libc_name is None
ss.sssss...s...s...........s....s..................................................................s.................................sssss..ss..........................s....................ssFsssssss
======================================================================
FAIL: test_struct_by_value (ctypes.test.test_win32.Structures)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/local/tmp/python3/lib/python3.6/ctypes/test/test_win32.py", line 133, in test_struct_by_value
    self.assertEqual(ret.left, left.value)
AssertionError: -200 != 10

Ran 456 tests in 1.970s

FAILED (failures=1, skipped=92)

A failure occurs with libffi 3.1 while all tests passes with libffi git-master. bpo-23085 may be the solution.

xdegaye · 2016-07-26T16:17:12Z

Nice, the patch fixes the problem when python is built with gcc :)
Running test_ctypes on the Android emulator when python is built for the arm architecture or the armv7 architecture gives in both cases the same successfull result:

Ran 456 tests in 30.424s

OK (skipped=92)
test.test_ctypes passed in 41 sec
1 test OK.
Total duration: 0:00:43

I did not try with clang, stopped by the problem in bpo-27627 for armv7 and in bpo-27606 for arm.

xdegaye · 2016-07-29T21:16:06Z

By msg264746, only ARM fails, so I patch libffi for arm and aarch64 triplets only

Why not for all Android architectures (-linux-android) as it is done in PR120 ?

yan12125 · 2016-07-30T04:03:28Z

You're right. I thought the default malloc() implementation is better, and now I think a unified implementation on Android brings less surprises.

xdegaye · 2016-07-30T10:13:01Z

We should wait for the pull request to be merged in the libffi development repo before committing the patch. The PR is at libffi/libffi#265.

yan12125 · 2016-09-10T06:55:42Z

Since bpo-27976, this one can be closed as third-party, just like bpo-27323.

xdegaye · 2016-10-16T09:49:58Z

At least for non-Darwin POSIX builds:

Building _ctypes with the bundled copy of libffi is deprecated in 3.6 and the default is to use a system copy of libffi, bpo-27976.
The bundled libffi is removed in 3.7, bpo-27979.

As this crash happens with the bundled libffi, closing this issue as won't fix.

xdegaye · 2016-10-16T09:51:18Z

Thanks Chi Hsuan Yen for your contributions with this issue.

xdegaye mannequin added stdlib Python modules in the Lib dir build The build process and cross-build type-crash A hard crash of the interpreter, possibly with a core dump labels May 3, 2016

yan12125 mannequin changed the title ~~android: test_ctypes crashes on armv7~~ android: test_ctypes crashes on armv7 and aarch64 Jul 28, 2016

xdegaye mannequin closed this as completed Oct 16, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

android: test_ctypes crashes on armv7 and aarch64 #71129

android: test_ctypes crashes on armv7 and aarch64 #71129

xdegaye mannequin commented May 3, 2016

xdegaye mannequin commented May 3, 2016

xdegaye mannequin commented May 18, 2016

xdegaye mannequin commented May 18, 2016

yan12125 mannequin commented Jul 9, 2016

yan12125 mannequin commented Jul 26, 2016

xdegaye mannequin commented Jul 26, 2016

yan12125 mannequin commented Jul 26, 2016

yan12125 mannequin commented Jul 26, 2016

xdegaye mannequin commented Jul 26, 2016

xdegaye mannequin commented Jul 29, 2016

yan12125 mannequin commented Jul 30, 2016

xdegaye mannequin commented Jul 30, 2016

yan12125 mannequin commented Sep 10, 2016

xdegaye mannequin commented Oct 16, 2016

xdegaye mannequin commented Oct 16, 2016

android: test_ctypes crashes on armv7 and aarch64 #71129

android: test_ctypes crashes on armv7 and aarch64 #71129

Comments

xdegaye mannequin commented May 3, 2016

xdegaye mannequin commented May 3, 2016

xdegaye mannequin commented May 18, 2016

xdegaye mannequin commented May 18, 2016

yan12125 mannequin commented Jul 9, 2016

yan12125 mannequin commented Jul 26, 2016

xdegaye mannequin commented Jul 26, 2016

yan12125 mannequin commented Jul 26, 2016

yan12125 mannequin commented Jul 26, 2016

xdegaye mannequin commented Jul 26, 2016

xdegaye mannequin commented Jul 29, 2016

yan12125 mannequin commented Jul 30, 2016

xdegaye mannequin commented Jul 30, 2016

yan12125 mannequin commented Sep 10, 2016

xdegaye mannequin commented Oct 16, 2016

xdegaye mannequin commented Oct 16, 2016