This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ctypes: test_ctypes test_callbacks() crash on AArch64 with SELinux enabled
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 2.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: fweimer, vstinner
Priority: normal Keywords:

Created on 2019-02-18 16:25 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
bug.py vstinner, 2019-02-18 16:25
bug2.py vstinner, 2019-02-18 16:56
Messages (8)
msg335847 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-18 16:25
Attached bug.py does crash *randomly* on AArch64. The code is extract from ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase.test_callbacks test.

Example with Python 2.7.15 and Python 3.6.8 on RHEL8:

# python2 bug.py
Illegal instruction (core dumped)
[root@cav-thunderx2s-cn88xx-01 ~]# python3 bug.py
...
OK
[root@cav-thunderx2s-cn88xx-01 ~]# python3 bug.py
Illegal instruction (core dumped)

I can reproduce the crash on Python 2.7.16rc compiled manually:
./configure --enable-unicode=ucs4 --with-system-ffi && make

RHEL8 currently uses libffi-3.1-18.el8.aarch64.

(I tried optimization levels -O0, -O1, -O2, -O3: I am always able to *randomly* trigger the crash.)

Original bug report, Python 2 crash on RHEL8:
https://bugzilla.redhat.com/show_bug.cgi?id=1652930

--

I don't know if it's related but I also saw the following error which has been reported in bpo-30991.

FAIL: test_pass_by_value (ctypes.test.test_structures.StructureTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/src/python-3.6.2/Lib/ctypes/test/test_structures.py", line 416, in test_pass_by_value
    self.assertEqual(s.first, 0xdeadbeef)
AssertionError: 195948557 != 3735928559
msg335848 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-18 16:33
> I can reproduce the crash on Python 2.7.16rc compiled manually:
> ./configure --enable-unicode=ucs4 --with-system-ffi && make

I can also reproduce the crash on Python 3.7.2 compiled manually with:
./configure --with-pydebug && make
(_ctypes uses system libffi)
msg335851 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-18 16:56
bug2.py: example simplified a little bit more. Changes:

* unittest is no longer used
* move local variables to globals, to make sure that memory is not released in any way during function calls
msg335853 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-18 17:09
Sadly, even with debug symbols on RHEL8 and gcc -g -O0 -fno-omit-frame-pointer, gdb is unable to retrieve the backtrace when the crash occurs :-(

Program received signal SIGILL, Illegal instruction.
0x0000ffffbe7a0058 in ?? ()
(gdb) where
#0  0x0000ffffbe7a0058 in ?? ()
#1  0x0000ffffffffe8a0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
msg335855 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-18 17:46
# gdb -args ./python ../bug2.py 
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-5.el8

(gdb) run
...

Program received signal SIGILL, Illegal instruction.
0x0000ffffbded0058 in ?? ()

(gdb) disassemble $pc,$pc+10
Dump of assembler code from 0xffffbded0058 to 0xffffbded0062:
=> 0x0000ffffbded0058:  ldr     x16, 0xffffbded0064
   0x0000ffffbded005c:  adr     x17, 0xffffbded006c
   0x0000ffffbded0060:  br      x16
End of assembler dump.

(gdb) print *(void**)0xffffbded0064
$1 = (void *) 0xffffbdea4f64 <ffi_closure_SYSV>


According to /proc/95475/maps, 0xffffbded0064 lives in this memory mapping:

ffffbded0000-ffffbdee0000 r-xs 00000000 fd:00 34943874   /tmp/ffiQhRWVt (deleted)
msg335860 - (view) Author: Florian Weimer (fweimer) Date: 2019-02-18 19:58
We believe this is a libffi bug.  On certain aarch64 implementations, it is necessary to flush both the writable mapping and the executable mapping.  I have a patch which I confirmed to work.
msg335861 - (view) Author: Florian Weimer (fweimer) Date: 2019-02-18 20:02
Filed as an libffi issue: https://github.com/libffi/libffi/issues/470
msg335920 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-19 10:32
Florian Weimer wrote a fix for libffi, so the bug is in libffi and not Python/ctypes. I close the issue. There are enough open issues to track this bug :-)
https://github.com/libffi/libffi/issues/470
History
Date User Action Args
2022-04-11 14:59:11adminsetgithub: 80205
2019-02-19 10:32:33vstinnersetstatus: open -> closed
title: ctypes: test_ctypes test_callbacks() crash on AArch64 -> ctypes: test_ctypes test_callbacks() crash on AArch64 with SELinux enabled
messages: + msg335920

resolution: third party
stage: resolved
2019-02-18 20:02:52fweimersetmessages: + msg335861
2019-02-18 19:58:42fweimersetmessages: + msg335860
2019-02-18 17:46:45vstinnersetmessages: + msg335855
2019-02-18 17:25:19fweimersetnosy: + fweimer
2019-02-18 17:09:48vstinnersetmessages: + msg335853
2019-02-18 16:56:52vstinnersetfiles: + bug2.py

messages: + msg335851
2019-02-18 16:33:52vstinnersetmessages: + msg335848
2019-02-18 16:25:21vstinnercreate