"import ctypes" causes segfault on read-only filesystem

This regression was introduced in python-2.6.6 and exists in all the later versions.

To reproduce run python -c "import ctypes" on read-only filesystem:


(gdb) file python3.2
Reading symbols from /usr/bin/python3.2...done.
(gdb) run -c "import ctypes"
Starting program: /usr/bin/python3.2 -c "import ctypes"
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0xb7af605c in CThunkObject_dealloc (_self=0xb7b35344)
    at /var/tmp/portage/dev-lang/python-3.2_pre20110123/work/python-3.2_pre20110123/Modules/_ctypes/callbacks.c:18
18      /var/tmp/portage/dev-lang/python-3.2_pre20110123/work/python-3.2_pre20110123/Modules/_ctypes/callbacks.c: No such file or directory.
        in /var/tmp/portage/dev-lang/python-3.2_pre20110123/work/python-3.2_pre20110123/Modules/_ctypes/callbacks.c
(gdb) bt
#0  0xb7af605c in CThunkObject_dealloc (_self=0xb7b35344)
    at /var/tmp/portage/dev-lang/python-3.2_pre20110123/work/python-3.2_pre20110123/Modules/_ctypes/callbacks.c:18
#1  0xb7af63b4 in _ctypes_alloc_callback (callable=0xb7b10bec, converters=0xb7c4e02c, restype=0xb810c544, flags=257)
    at /var/tmp/portage/dev-lang/python-3.2_pre20110123/work/python-3.2_pre20110123/Modules/_ctypes/callbacks.c:439
#2  0xb7af1f57 in PyCFuncPtr_new (type=0xb810b0bc, args=0xb7b3618c, kwds=0x0)
    at /var/tmp/portage/dev-lang/python-3.2_pre20110123/work/python-3.2_pre20110123/Modules/_ctypes/_ctypes.c:3339
#3  0xb7ea2355 in type_call (type=0xb810b0bc, args=0xb7b3618c, kwds=0x0) at Objects/typeobject.c:676
#4  0xb7e4f34e in PyObject_Call (func=0xb810b0bc, arg=0xb7b3618c, kw=0x0) at Objects/abstract.c:2149
#5  0xb7eedee3 in do_call (f=0xb80fdb44, throwflag=0) at Python/ceval.c:4095
#6  call_function (f=0xb80fdb44, throwflag=0) at Python/ceval.c:3898
#7  PyEval_EvalFrameEx (f=0xb80fdb44, throwflag=0) at Python/ceval.c:2673
#8  0xb7ef0639 in PyEval_EvalCodeEx (_co=0xb7b159d0, globals=0xb7bf40b4, locals=0xb7bf40b4, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3311
#9  0xb7ef08b6 in PyEval_EvalCode (co=0xb7b159d0, globals=0xb7bf40b4, locals=0xb7bf40b4) at Python/ceval.c:761
#10 0xb7f0121c in PyImport_ExecCodeModuleWithPathnames (name=0xbfffd9fb "ctypes", co=0xb7b159d0,
    pathname=0xbfffa89b "/usr/lib/python3.2/ctypes/__pycache__/__init__.cpython-32.pyc",
    cpathname=0xbfffa89b "/usr/lib/python3.2/ctypes/__pycache__/__init__.cpython-32.pyc") at Python/import.c:809
#11 0xb7f03ce8 in load_source_module (name=<value optimized out>, pathname=<value optimized out>, fp=0xb8020b28) at Python/import.c:1339
#12 0xb7f044f8 in load_package (name=<value optimized out>, pathname=<value optimized out>) at Python/import.c:1435
#13 0xb7f04da7 in import_submodule (mod=<value optimized out>, subname=<value optimized out>, fullname=0xbfffd9fb "ctypes") at Python/import.c:2894
#14 0xb7f050b4 in load_next (mod=<value optimized out>, altmod=<value optimized out>, p_name=0xbfffd9ec, buf=0xbfffd9fb "ctypes", p_buflen=0xbfffd9f4)
    at Python/import.c:2706
#15 0xb7f05774 in import_module_level (name=0x0, globals=<value optimized out>, locals=0xb7c2035c, fromlist=0xb7f98ca0, level=0) at Python/import.c:2422
#16 0xb7f05d14 in PyImport_ImportModuleLevel (name=0xb7c0f8e8 "ctypes", globals=0xb7c2035c, locals=0xb7c2035c, fromlist=0xb7f98ca0, level=0)
    at Python/import.c:2474
#17 0xb7ee73c1 in builtin___import__ (self=0xb7c6726c, args=0xb7c7b9bc, kwds=0x0) at Python/bltinmodule.c:168
#18 0xb7e907fe in PyCFunction_Call (func=0xb7c6730c, arg=0xb7c7b9bc, kw=0xb7b35344) at Objects/methodobject.c:84
#19 0xb7e4f34e in PyObject_Call (func=0xb7c6730c, arg=0xb7c7b9bc, kw=0x0) at Objects/abstract.c:2149
#20 0xb7ee802f in PyEval_CallObjectWithKeywords (func=0xb7c6730c, arg=0xb7c7b9bc, kw=0x0) at Python/ceval.c:3755
#21 0xb7eec962 in PyEval_EvalFrameEx (f=0xb8072564, throwflag=0) at Python/ceval.c:2332
#22 0xb7ef0639 in PyEval_EvalCodeEx (_co=0xb7bdb7f0, globals=0xb7c2035c, locals=0xb7c2035c, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    kwdefs=0x0, closure=0x0) at Python/ceval.c:3311
#23 0xb7ef08b6 in PyEval_EvalCode (co=0xb7bdb7f0, globals=0xb7c2035c, locals=0xb7c2035c) at Python/ceval.c:761
#24 0xb7f0eabc in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=0xb7c2035c, locals=0xb7c2035c, flags=0xbfffefa8,
    arena=0xb8071030) at Python/pythonrun.c:1760
#25 0xb7f0edf9 in PyRun_StringFlags (str=0xb7bf5330 "import ctypes\n", start=257, globals=0xb7c2035c, locals=0xb7c2035c, flags=0xbfffefa8)
    at Python/pythonrun.c:1694
#26 0xb7f11006 in PyRun_SimpleStringFlags (command=0xb7bf5330 "import ctypes\n", flags=0xbfffefa8) at Python/pythonrun.c:1267
#27 0xb7f2477c in run_command (argc=3, argv=0xb8001018) at Modules/main.c:258
#28 Py_Main (argc=3, argv=0xb8001018) at Modules/main.c:647
#29 0xb7fffc4f in main (argc=3, argv=0xbffff0d4) at ./Modules/python.c:82
(gdb) quit

I cannot reproduce this with the current py3k branch (Ubuntu 10.04 32 bit).

How to reproduce:

# mkdir /mnt/readonly
# mount --bind / /mnt/readonly
# mount -o remount,ro /mnt/readonly
# mount -t proc proc /mnt/readonly/proc
# chroot /mnt/readonly python3.2 -c "import ctypes"
Segmentation fault

If your python build expected to have this bug, you'll see something like this (the -1 EROFS lines):

# chroot /mnt/readonly strace -f -e trace=open python3.2 -c "import ctypes" 2>&1 | grep ffi
open("/usr/lib/libffi.so.5", O_RDONLY)  = 5
open("/tmp/.private/root/ffiicoh8G", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 ENOENT (No such file or directory)
open("/tmp/ffiFjqUa9", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EROFS (Read-only file system)
open("/var/tmp/ffidTdydB", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EROFS (Read-only file system)
open("/dev/shm/ffiemIcg3", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EROFS (Read-only file system)
open("/root/ffiXfWRiv", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EROFS (Read-only file system)

Thanks for the instructions. You're right, this way I can reproduce the segfault.

This patch fixes issue with unitialized variable which makes ctypes crash in error handler.

Note that for you it merely turns "Segmentation fault" into MemoryError exception.  Python ships with buggy version of libffi, which tries to allocate memory using mmap() over a temporary file (the one seen in strace output above) instead of just doing mmap(MAP_ANONYMOUS|MAP_PRIVATE).  The bug is fixed in this libffi commit:

  https://github.com/atgreen/libffi/commit/eaf444eabc4c78703c0f98ac0197b1619c1b1bef

Until Python's libffi is updated, you can do "mount -t tmpfs none /dev/shm".  It's good to have "tmpfs" mounted anyway, just like "proc" and "sysfs".

It would be nice to add a unit test to this patch.  I don't think messing with read-only filesystems is feasible in regrtest, but it seems from reading _ctypes_alloc_callback() source that similar behavior can be triggered by passing a failing converters list to a ctypes function constructor.

A nit-pick: it would be easier to verify that all CThunkObject fields are initialized if initialization was done in the same order as declaration:


typedef struct {
    PyObject_VAR_HEAD
    ...
    PyObject *callable;
    PyObject *restype;
    SETFUNC setfunc;
    ffi_type *ffi_restype;
    ffi_type *atypes[1];
} CThunkObject;


In other words, "p->restype = NULL;" should go after "p->callable = NULL;".

New changeset ab4975520c7d by Victor Stinner in branch '3.5':
ctypes: fix CThunkObject_new()
https://hg.python.org/cpython/rev/ab4975520c7d

New changeset da9898e7e90d by Victor Stinner in branch 'default':
Merge 3.5 (issue #11048)
https://hg.python.org/cpython/rev/da9898e7e90d

New changeset 7d692840c152 by Victor Stinner in branch '2.7':
ctypes: fix CThunkObject_new()
https://hg.python.org/cpython/rev/7d692840c152

Sorry, I'm unable to reproduce the crash using the read-only filesystem. But an assertion confirmed me that the restype is not initialized which is an obvious bug.


ctypes-erofs-crash.diff: LGTM. By the way, the flags field is not initialized neither, but it's probably less important (at least for the destructor).


Alexander Belopolsky: "It would be nice to add a unit test to this patch."

I tried to write a short unit test triggering the bug, but it really depends on low-level implementation details for a very specific corner case. I would prefer to avoid such complex unit test and just fix the bug. We waited for an unit test the last 5 years, it's too late now :-)


"A nit-pick: it would be easier to verify that all CThunkObject fields are initialized if initialization was done in the same order as declaration:"

Right, I rewrote the patch as you suggested. I noticed that flags is not initialized neither (also fixed in my patch).


I pushed a fix.


Thanks marcin.bachry for the initial patch! Thanks Pavel Labushev for your bug report! ... but sorry for the very long delay :-( It looks like nobody reproduced the bug recently, maybe it was fixed in libffi in the meanwhile?