classification
Title: android: test_ctypes crashes on armv7 and aarch64
Type: crash Stage: resolved
Components: Cross-Build, Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, Chi Hsuan Yen, amaury.forgeotdarc, belopolsky, meador.inge, xdegaye
Priority: normal Keywords: patch

Created on 2016-05-03 15:53 by xdegaye, last changed 2016-10-16 09:54 by xdegaye. This issue is now closed.

Files
File name Uploaded Description Edit
libffi-pr240.patch Chi Hsuan Yen, 2016-07-26 13:56 review
libffi-pr240.patch Chi Hsuan Yen, 2016-07-30 04:03 Version 2 review
Messages (15)
msg264745 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-03 15:53
test_ctypes crashes on an android emulator running an armv7 system image (but not on x86) at API level 21.


143|root@generic:/data/local/tmp # python -m test -v test_ctypes                                   
== CPython 3.6.0a0 (default:f4c6dab59cd8+, May 3 2016, 17:24:17) [GCC 4.9 20140827 (prerelease)]
==   Linux-3.4.67-01422-gd3ffcc7-dirty-armv7l-with-libc little-endian
==   hash algorithm: fnv 32bit
==   /data/local/tmp/test_python_2301
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, 
no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomiza
tion=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_ctypes
test_anon (ctypes.test.test_anon.AnonTest) ... ok
test_anon_nonmember (ctypes.test.test_anon.AnonTest) ... ok
test_anon_nonseq (ctypes.test.test_anon.AnonTest) ... ok
test_nested (ctypes.test.test_anon.AnonTest) ... ok
test (ctypes.test.test_array_in_pointer.Test) ... ok
test_2 (ctypes.test.test_array_in_pointer.Test) ... ok
test_bad_subclass (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_cache (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_classcache (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_from_address (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_from_addressW (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_numeric_arrays (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_simple (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_subclass (ctypes.test.test_arrays.ArrayTestCase) ... ok
test_byval (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ... ok
test_callbacks (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ... Fatal Python error
: Segmentation fault

Current thread 0xb6f2eec8 (most recent call first):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/ctypes/test/test_as_parameter.py", line 85 in test
_callbacks
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/case.py", line 600 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/case.py", line 648 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 122 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/suite.py", line 84 in __call__
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/unittest/runner.py", line 176 in run
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/__init__.py", line 1802 in _run_suite
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/__init__.py", line 1836 in run_unitte
st
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 166 in test_run
ner
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 167 in runtest_
inner
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/runtest.py", line 131 in runtest
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 332 in run_tests_s
equential
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 402 in run_tests
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 462 in _main
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 442 in main
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/libregrtest/main.py", line 504 in main
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/__main__.py", line 2 in <module>
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/runpy.py", line 85 in _run_code
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/runpy.py", line 184 in _run_module_as_main
Segmentation fault
msg265823 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-18 10:34
The crash occurs at the same line that the crash reported in issue
#17786. Line 85 in ctypes/test/test_as_parameter.py was line 87 at changeset  ae5c4a9118b8a3f490f77f2084d46163ca229aef.
msg265829 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-05-18 14:57
Running the following interactive statements [1]:

>>> import unittest, ctypes.test.test_as_parameter
>>> unittest.main(module=ctypes.test.test_as_parameter, defaultTest='BasicWrapTestCase', verbosity=2)

The corresponding attached gdb session:

(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0xb6a181a0 in ?? ()
(gdb) bt 7
#0  0xb6a181a0 in ?? ()
#1  0xb605de7a in _testfunc_callback_i_if (value=-10, func=0xb6a181a0)
    at cpython/Modules/_ctypes/_ctypes_test.c:234
#2  0xb6072238 in ffi_call_SYSV ()
    at cpython/Modules/_ctypes/libffi/src/arm/sysv.S:188
#3  0xb6072a52 in ffi_call (cif=cif@entry=0xbe91ac04, 
    fn=fn@entry=0xb605de6d <_testfunc_callback_i_if>, rvalue=rvalue@entry=0xbe91ac88, 
    avalue=avalue@entry=0xbe91ac78)
    at cpython/Modules/_ctypes/libffi/src/arm/ffi.c:339
#4  0xb606e2a0 in _call_function_pointer (flags=flags@entry=-1240871416, 
    pProc=0xb605de6d <_testfunc_callback_i_if>, pProc@entry=0xb606956b <PyCFuncPtr_call+350>, 
    avalues=avalues@entry=0xbe91ac78, atypes=atypes@entry=0xbe91ac68, restype=0xb60db8e0, 
    resmem=resmem@entry=0xbe91ac88, argcount=argcount@entry=2)
    at cpython/Modules/_ctypes/callproc.c:811
#5  0xb606e894 in _ctypes_callproc (pProc=pProc@entry=0xb605de6d <_testfunc_callback_i_if>, 
    argtuple=argtuple@entry=(<AsParamPropertyWrapper(_param=-10) at remote 0xb5f012d8>, <AsParamPropertyWrapper(_param=<CFunctionType at remote 0xb5ef5c70>) at remote 0xb5ef1fc0>), 
    flags=<optimized out>, argtypes=argtypes@entry=0x0, restype=<optimized out>, 
    restype@entry=<_ctypes.PyCSimpleType at remote 0xb6097228>, checker=checker@entry=0x0)
    at cpython/Modules/_ctypes/callproc.c:1149
#6  0xb606956a in PyCFuncPtr_call (self=0xb5ef5e90, inargs=<optimized out>, kwds=<optimized out>)
    at cpython/Modules/_ctypes/_ctypes.c:3856
(More stack frames follow...)
(gdb) up
#1  0xb605de7a in _testfunc_callback_i_if (value=-10, func=0xb6a181a0)
    at cpython/Modules/_ctypes/_ctypes_test.c:234
234	        sum += func(value);
(gdb) list
229	
230	EXPORT(int) _testfunc_callback_i_if(int value, int (*func)(int))
231	{
232	    int sum = 0;
233	    while (value != 0) {
234	        sum += func(value);
235	        value /= 2;
236	    }
237	    return sum;
238	}
(gdb) p func
$2 = (int (*)(int)) 0xb6a181a0
(gdb) disassemble func
No function contains specified address.
(gdb)


[1] Gdb is attached after the import statement because gdb fails with SIGILL in __dl_notify_gdb_of_libraries both with arm and armv7, whenever a library is loaded. This is gdb from the Android ndk started by ndk-gdb.py from the r11c ndk. A workaround is to add the following lines in the gdbinit script:
    break __dl_rtld_db_dlactivity
    commands
    silent
    return
    sharedlibrary
    continue
    end
msg270032 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-07-09 09:08
Maybe a libffi issue. The crash is still with libffi git-master and CPython hg-tip + `--with-system-libffi`. Reported to https://github.com/libffi/libffi/issues/262
msg271355 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-07-26 10:14
Found libffi PR240 that fixes closures on Android:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -m test.test_ctypes
...................................................................s......s..............s....................ssssssssssssssssssssss..................................ssssssssssssssssssssssssssss.s...sssOpenGL libraries:
         ('GL', None)
         ('GLU', None)
         ('gle', None)
sss...........................s.s............s.........libc_name is None
ss.sssss...s...s...........s....s..................................................................s.................................sssss..ss..........................s....................ss.sssssss
----------------------------------------------------------------------
Ran 456 tests in 2.090s

OK (skipped=92)

See https://github.com/yan12125/python3-android/commit/1daebca128e9ca1a9a39ef143ded7090cfa71cc9 for details.

By http://comments.gmane.org/gmane.comp.lib.ffi.general/1235, SELinux affects the result, too. PR240 of libffi assumes SELinux is disabled. I have disabled SELinux on my phone for some root applications. I'm not sure whether PR240 works for phones with SELinux or not.
msg271357 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-26 11:43
Thanks for looking into that problem.
Can you provide a patch ?
msg271367 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-07-26 13:56
By msg264746, only ARM fails, so I patch libffi for arm and aarch64 triplets only
msg271368 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-07-26 14:00
Test results against patched libffi in Modules/_ctypes:

shell@ASUS_Z00E_2:/data/local/tmp $ python3.6 -m test.test_ctypes              
...................................................................s......s..............s....................ssssssssssssssssssssss..................................ssssssssssssssssssssssssssss.s...sssOpenGL libraries:
         ('GL', None)
         ('GLU', None)
         ('gle', None)
sss...........................s.s............s.........libc_name is None
ss.sssss...s...s...........s....s..................................................................s.................................sssss..ss..........................s....................ssFsssssss
======================================================================
FAIL: test_struct_by_value (ctypes.test.test_win32.Structures)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/local/tmp/python3/lib/python3.6/ctypes/test/test_win32.py", line 133, in test_struct_by_value
    self.assertEqual(ret.left, left.value)
AssertionError: -200 != 10

----------------------------------------------------------------------
Ran 456 tests in 1.970s

FAILED (failures=1, skipped=92)

A failure occurs with libffi 3.1 while all tests passes with libffi git-master. Issue23085 may be the solution.
msg271392 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-26 16:17
Nice, the patch fixes the problem when python is built with gcc :)
Running test_ctypes on the Android emulator when python is built for the arm architecture or the armv7 architecture gives in both cases the same successfull result:

    Ran 456 tests in 30.424s

    OK (skipped=92)
    test.test_ctypes passed in 41 sec
    1 test OK.
    Total duration: 0:00:43

I did not try with clang, stopped by the problem in issue 27627 for armv7 and in issue 27606 for arm.
msg271649 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-29 21:16
> By msg264746, only ARM fails, so I patch libffi for arm and aarch64 triplets only

Why not for all Android architectures (*-linux-android*) as it is done in PR120 ?
msg271662 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-07-30 04:03
You're right. I thought the default malloc() implementation is better, and now I think a unified implementation on Android brings less surprises.
msg271675 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-30 10:13
We should wait for the pull request to be merged in the libffi development repo before committing the patch. The PR is at https://github.com/libffi/libffi/pull/265.
msg275584 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-09-10 06:55
Since issue27976, this one can be closed as third-party, just like issue27323.
msg278759 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-10-16 09:49
At least for non-Darwin POSIX builds:
* Building _ctypes with the bundled copy of libffi is deprecated in 3.6 and the default is to use a system copy of libffi, issue 27976.
* The bundled libffi is removed in 3.7, issue 27979.

As this crash happens with the bundled libffi, closing this issue as won't fix.
msg278760 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-10-16 09:51
Thanks Chi Hsuan Yen for your contributions with this issue.
History
Date User Action Args
2016-10-16 09:54:15xdegayesetstatus: open -> closed
2016-10-16 09:51:17xdegayesetmessages: + msg278760
2016-10-16 09:49:58xdegayesetresolution: wont fix
messages: + msg278759
stage: patch review -> resolved
2016-09-10 06:55:41Chi Hsuan Yensetmessages: + msg275584
2016-07-30 10:13:00xdegayesetmessages: + msg271675
2016-07-30 04:03:28Chi Hsuan Yensetfiles: + libffi-pr240.patch

messages: + msg271662
2016-07-29 21:16:05xdegayesetmessages: + msg271649
stage: patch review
2016-07-28 02:30:45Chi Hsuan Yensettitle: android: test_ctypes crashes on armv7 -> android: test_ctypes crashes on armv7 and aarch64
2016-07-26 16:17:12xdegayesetmessages: + msg271392
2016-07-26 14:00:32Chi Hsuan Yensetmessages: + msg271368
2016-07-26 13:56:31Chi Hsuan Yensetfiles: + libffi-pr240.patch
keywords: + patch
messages: + msg271367
2016-07-26 11:43:02xdegayesetmessages: + msg271357
2016-07-26 10:14:33Chi Hsuan Yensetmessages: + msg271355
2016-07-09 09:08:13Chi Hsuan Yensetnosy: + Chi Hsuan Yen
messages: + msg270032
2016-05-21 07:06:39xdegayelinkissue26865 dependencies
2016-05-18 14:57:49xdegayesetmessages: + msg265829
2016-05-18 10:34:26xdegayesetmessages: + msg265823
2016-05-03 15:53:09xdegayecreate