test_ctypes crashes on an android emulator running an armv7 system image (but not on x86) at API level 21.


143|root@generic:/data/local/tmp # python -m test -v test_ctypes                                   
== CPython 3.6.0a0 (default:f4c6dab59cd8+, May 3 2016, 17:24:17) [GCC 4.9 20140827 (prerelease)]
==   Linux-3.4.67-01422-gd3ffcc7-dirty-armv7l-with-libc little-endian
==   hash algorithm: fnv 32bit
==   /data/local/tmp/test_python_2301
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, 
no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomiza
tion=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_ctypes
test_anon (ctypes.test.test_anon.AnonTest) ... ok
test_anon_nonmember (ctypes.test.test_anon.AnonTest) ... ok
test_anon_nonseq (ctypes.test.test_anon.AnonTest) ... ok
test_nested (ctypes.test.test_anon.AnonTest) ... ok
test (ctypes.test.test_array_in_pointer.Test) ... ok
test_2 (ctypes.test.test_array_in_pointer.Test) ... ok
test_bad_subclass (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_cache (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_classcache (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_from_address (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_from_addressW (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_numeric_arrays (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_simple (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_subclass (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_byval (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ... ok
test_callbacks (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ... Fatal Python error
: Segmentation fault

Current thread 0xb6f2eec8 (most recent call first):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/ctypes/test/test_as_parameter.py", line 85 in test
_callbacks
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/case.py", line 600 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/case.py", line 648 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/runner.py", line 176 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/__init__.py", line 1802 in _run_suite
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/__init__.py", line 1836 in run_unitte
st
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 166 in test_run
ner
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 167 in runtest_
inner
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 131 in runtest
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 332 in run_tests_s
equential
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 402 in run_tests
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 462 in _main
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 442 in main
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 504 in main
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/__main__.py", line 2 in <module>
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/runpy.py", line 85 in _run_code
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/runpy.py", line 184 in _run_module_as_main
Segmentation fault

The crash occurs at the same line that the crash reported in issue
#17786. Line 85 in ctypes/test/test_as_parameter.py was line 87 at changeset  ae5c4a9118b8a3f490f77f2084d46163ca229aef.

Running the following interactive statements [1]:

>>> import unittest, ctypes.test.test_as_parameter
>>> unittest.main(module=ctypes.test.test_as_parameter, defaultTest='BasicWrapTestCase', verbosity=2)

The corresponding attached gdb session:

(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0xb6a181a0 in ?? ()
(gdb) bt 7
#0  0xb6a181a0 in ?? ()
#1  0xb605de7a in _testfunc_callback_i_if (value=-10, func=0xb6a181a0)
    at cpython/Modules/_ctypes/_ctypes_test.c:234
#2  0xb6072238 in ffi_call_SYSV ()
    at cpython/Modules/_ctypes/libffi/src/arm/sysv.S:188
#3  0xb6072a52 in ffi_call (cif=cif@entry=0xbe91ac04, 
    fn=fn@entry=0xb605de6d <_testfunc_callback_i_if>, rvalue=rvalue@entry=0xbe91ac88, 
    avalue=avalue@entry=0xbe91ac78)
    at cpython/Modules/_ctypes/libffi/src/arm/ffi.c:339
#4  0xb606e2a0 in _call_function_pointer (flags=flags@entry=-1240871416, 
    pProc=0xb605de6d <_testfunc_callback_i_if>, pProc@entry=0xb606956b <PyCFuncPtr_call+350>, 
    avalues=avalues@entry=0xbe91ac78, atypes=atypes@entry=0xbe91ac68, restype=0xb60db8e0, 
    resmem=resmem@entry=0xbe91ac88, argcount=argcount@entry=2)
    at cpython/Modules/_ctypes/callproc.c:811
#5  0xb606e894 in _ctypes_callproc (pProc=pProc@entry=0xb605de6d <_testfunc_callback_i_if>, 
    argtuple=argtuple@entry=(<AsParamPropertyWrapper(_param=-10) at remote 0xb5f012d8>, <AsParamPropertyWrapper(_param=<CFunctionType at remote 0xb5ef5c70>) at remote 0xb5ef1fc0>), 
    flags=<optimized out>, argtypes=argtypes@entry=0x0, restype=<optimized out>, 
    restype@entry=<_ctypes.PyCSimpleType at remote 0xb6097228>, checker=checker@entry=0x0)
    at cpython/Modules/_ctypes/callproc.c:1149
#6  0xb606956a in PyCFuncPtr_call (self=0xb5ef5e90, inargs=<optimized out>, kwds=<optimized out>)
    at cpython/Modules/_ctypes/_ctypes.c:3856
(More stack frames follow...)
(gdb) up
#1  0xb605de7a in _testfunc_callback_i_if (value=-10, func=0xb6a181a0)
    at cpython/Modules/_ctypes/_ctypes_test.c:234
234	        sum += func(value);
(gdb) list
229	
230	EXPORT(int) _testfunc_callback_i_if(int value, int (*func)(int))
231	{
232	    int sum = 0;
233	    while (value != 0) {
234	        sum += func(value);
235	        value /= 2;
236	    }
237	    return sum;
238	}
(gdb) p func
$2 = (int (*)(int)) 0xb6a181a0
(gdb) disassemble func
No function contains specified address.
(gdb)


[1] Gdb is attached after the import statement because gdb fails with SIGILL in __dl_notify_gdb_of_libraries both with arm and armv7, whenever a library is loaded. This is gdb from the Android ndk started by ndk-gdb.py from the r11c ndk. A workaround is to add the following lines in the gdbinit script:
    break __dl_rtld_db_dlactivity
    commands
    silent
    return
    sharedlibrary
    continue
    end

Maybe a libffi issue. The crash is still with libffi git-master and CPython hg-tip + `--with-system-libffi`. Reported to https://github.com/libffi/libffi/issues/262

Found libffi PR240 that fixes closures on Android:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -m test.test_ctypes
...................................................................s......s..............s....................ssssssssssssssssssssss..................................ssssssssssssssssssssssssssss.s...sssOpenGL libraries:
         ('GL', None)
         ('GLU', None)
         ('gle', None)
sss...........................s.s............s.........libc_name is None
ss.sssss...s...s...........s....s..................................................................s.................................sssss..ss..........................s....................ss.sssssss
----------------------------------------------------------------------
Ran 456 tests in 2.090s

OK (skipped=92)

See https://github.com/yan12125/python3-android/commit/1daebca128e9ca1a9a39ef143ded7090cfa71cc9 for details.

By http://comments.gmane.org/gmane.comp.lib.ffi.general/1235, SELinux affects the result, too. PR240 of libffi assumes SELinux is disabled. I have disabled SELinux on my phone for some root applications. I'm not sure whether PR240 works for phones with SELinux or not.

Thanks for looking into that problem.
Can you provide a patch ?

By msg264746, only ARM fails, so I patch libffi for arm and aarch64 triplets only

Test results against patched libffi in Modules/_ctypes:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -m test.test_ctypes              
...................................................................s......s..............s....................ssssssssssssssssssssss..................................ssssssssssssssssssssssssssss.s...sssOpenGL libraries:
         ('GL', None)
         ('GLU', None)
         ('gle', None)
sss...........................s.s............s.........libc_name is None
ss.sssss...s...s...........s....s..................................................................s.................................sssss..ss..........................s....................ssFsssssss
======================================================================
FAIL: test_struct_by_value (ctypes.test.test_win32.Structures)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/local/tmp/python3/lib/python3.6/ctypes/test/test_win32.py", line 133, in test_struct_by_value
    self.assertEqual(ret.left, left.value)
AssertionError: -200 != 10

----------------------------------------------------------------------
Ran 456 tests in 1.970s

FAILED (failures=1, skipped=92)

A failure occurs with libffi 3.1 while all tests passes with libffi git-master. Issue23085 may be the solution.

Nice, the patch fixes the problem when python is built with gcc :)
Running test_ctypes on the Android emulator when python is built for the arm architecture or the armv7 architecture gives in both cases the same successfull result:

    Ran 456 tests in 30.424s

    OK (skipped=92)
    test.test_ctypes passed in 41 sec
    1 test OK.
    Total duration: 0:00:43

I did not try with clang, stopped by the problem in issue 27627 for armv7 and in issue 27606 for arm.

> By msg264746, only ARM fails, so I patch libffi for arm and aarch64 triplets only

Why not for all Android architectures (*-linux-android*) as it is done in PR120 ?

You're right. I thought the default malloc() implementation is better, and now I think a unified implementation on Android brings less surprises.

We should wait for the pull request to be merged in the libffi development repo before committing the patch. The PR is at https://github.com/libffi/libffi/pull/265.

Since issue27976, this one can be closed as third-party, just like issue27323.

At least for non-Darwin POSIX builds:
* Building _ctypes with the bundled copy of libffi is deprecated in 3.6 and the default is to use a system copy of libffi, issue 27976.
* The bundled libffi is removed in 3.7, issue 27979.

As this crash happens with the bundled libffi, closing this issue as won't fix.

Thanks Chi Hsuan Yen for your contributions with this issue.