classification
Title: test_ctypes: segfault with suncc
Type: crash Stage: resolved
Components: ctypes Versions: Python 3.3
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Justin.Venus, doko, georg.brandl, loewis, meador.inge, pitrou, skrah, vstinner
Priority: normal Keywords:

Created on 2011-09-07 12:45 by skrah, last changed 2012-07-07 18:42 by skrah. This issue is now closed.

Files
File name Uploaded Description Edit
python3x_ctypes-systemffi.txt.gz Justin.Venus, 2012-06-26 04:52
Messages (15)
msg143670 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-07 12:45
stefan@opensolaris:~/hg/cpython$ ./python -m test -uall -v test_ctypes
== CPython 3.3.0a0 (default:5c8b6e03ebfe, Sep 7 2011, 13:41:08) [C]
==   Solaris-2.11-i86pc-i386-32bit little-endian
==   /export/home/stefan/hg/cpython/build/test_python_28849
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0)
[1/1] test_ctypes
test_cast (ctypes.test.test_memfunctions.MemFunctionsTest) ... ok
test_memmove (ctypes.test.test_memfunctions.MemFunctionsTest) ... ok
test_memset (ctypes.test.test_memfunctions.MemFunctionsTest) ... ok
test_string_at (ctypes.test.test_memfunctions.MemFunctionsTest) ... ok
test_wstring_at (ctypes.test.test_memfunctions.MemFunctionsTest) ... ok
test_chararray (ctypes.test.test_delattr.TestCase) ... ok
test_simple (ctypes.test.test_delattr.TestCase) ... ok
test_struct (ctypes.test.test_delattr.TestCase) ... ok
test_struct_by_value (ctypes.test.test_win32.Structures) ... ok
test_buffers (ctypes.test.test_unicode.StringTestCase) ... ok
test_func (ctypes.test.test_unicode.StringTestCase) ... ok
test_wcslen (ctypes.test.test_unicode.StringTestCase) ... ok
test_buffers (ctypes.test.test_unicode.UnicodeTestCase) ... ok
test_wcslen (ctypes.test.test_unicode.UnicodeTestCase) ... ok
test_compare (ctypes.test.test_simplesubclasses.Test) ... ok
test_ignore_retval (ctypes.test.test_simplesubclasses.Test) ... Fatal Python error: Segmentation fault

Current thread 0x00000001:
  File "/export/home/stefan/hg/cpython/Lib/ctypes/test/test_simplesubclasses.py", line 24 in test_ignore_retval
  File "/export/home/stefan/hg/cpython/Lib/unittest/case.py", line 386 in _executeTestPart
  File "/export/home/stefan/hg/cpython/Lib/unittest/case.py", line 441 in run
  File "/export/home/stefan/hg/cpython/Lib/unittest/case.py", line 493 in __call__
  File "/export/home/stefan/hg/cpython/Lib/unittest/suite.py", line 105 in run
  File "/export/home/stefan/hg/cpython/Lib/unittest/suite.py", line 67 in __call__
  File "/export/home/stefan/hg/cpython/Lib/unittest/suite.py", line 105 in run
  File "/export/home/stefan/hg/cpython/Lib/unittest/suite.py", line 67 in __call__
  File "/export/home/stefan/hg/cpython/Lib/unittest/suite.py", line 105 in run
  File "/export/home/stefan/hg/cpython/Lib/unittest/suite.py", line 67 in __call__
  File "/export/home/stefan/hg/cpython/Lib/unittest/runner.py", line 168 in run
  File "/export/home/stefan/hg/cpython/Lib/test/support.py", line 1293 in _run_suite
  File "/export/home/stefan/hg/cpython/Lib/test/support.py", line 1327 in run_unittest
  File "/export/home/stefan/hg/cpython/Lib/test/test_ctypes.py", line 13 in test_main
  File "/export/home/stefan/hg/cpython/Lib/test/regrtest.py", line 1140 in runtest_inner
  File "/export/home/stefan/hg/cpython/Lib/test/regrtest.py", line 916 in runtest
  File "/export/home/stefan/hg/cpython/Lib/test/regrtest.py", line 708 in main
  File "/export/home/stefan/hg/cpython/Lib/test/__main__.py", line 13 in <module>
  File "/export/home/stefan/hg/cpython/Lib/runpy.py", line 73 in _run_code
  File "/export/home/stefan/hg/cpython/Lib/runpy.py", line 160 in _run_module_as_main
Segmentation Fault (core dumped)
msg143671 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-07 12:51
Code of the test:

    from ctypes import *
    def test_ignore_retval(self):
        # Test if the return value of a callback is ignored
        # if restype is None
        proto = CFUNCTYPE(None)
        def func():
            return (1, "abc", None)
        cb = proto(func)
        self.assertEqual(None, cb())

The crash occurs on self.assertEqual(None, cb()). I suppose that it occurs on cb().

Is CFUNCTYPE correct? Or should it be PYFUNCTYPE?
msg143691 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-07 15:30
The segfault occurs both with CFUNCTYPE and PYFUNCTYPE in
ffi_prep_incoming_args_SYSV(). I'm not very familiar with
either suncc or the dbx debugger, and I can't get dbx to
step into that function (compiled with -g, libffi is not
stripped as far as I can see). Anyway, here's what I got:

(dbx) stop at ffi.c:403 
(2) stop at "ffi.c":403
(dbx) run
Running: python 
(process id 29932)
Python 3.3.0a0 (default:5c8b6e03ebfe+, Sep  7 2011, 15:48:46) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
Reading _struct.so
Reading libscf.so.1
Reading libuutil.so.1
Reading libgen.so.1
Reading libmd.so.1
Reading libmp.so.2
>>> proto = CFUNCTYPE(None)
>>> def func():
        return (1, "abc", None)
...
>>> cb = proto(func)
>>> cb()
stopped in ffi_closure_SYSV_inner at line 403 in file "ffi.c"
  403     ffi_prep_incoming_args_SYSV(args, respp, arg_area, cif);
(dbx) step
signal SEGV (no mapping at the fault address) in ffi_closure_SYSV_inner at line 403 in file "ffi.c"
  403     ffi_prep_incoming_args_SYSV(args, respp, arg_area, cif);
msg143838 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-10 14:09
I see what is going on.  I don't think the 'libffi' code can be executed correctly when built by 'suncc'.  The code currently requires '__attribute__((regparm(1)))', which AFAICT is not implemented by 'suncc';  only 'gcc'.

Take a look at 'ffi_closure_SYSV_inner' in 'Modules/_ctypes/libffi/src/x86/ffi.c':

void FFI_HIDDEN ffi_closure_raw_SYSV (ffi_raw_closure *)
     __attribute__ ((regparm(1)));

...

/* This function is jumped to by the trampoline */

unsigned int FFI_HIDDEN
ffi_closure_SYSV_inner (closure, respp, args)
     ffi_closure *closure;
     void **respp;
     void *args;
{
  /* our various things...  */
  ffi_cif       *cif;
  void         **arg_area;

  cif         = closure->cif;
  arg_area    = (void**) alloca (cif->nargs * sizeof (void*));

The '__attribute__ ((regparm(1)))' causes the first parameter to be passed in a register and the rest of the parameters to be spilled to the stack.  For most (all?) x86 compilers, '%eax' is the first register used to pass parameters when the calling convention requires registers.

'suncc' ignores this attribute (you can see warnings by greping the build logs) and expects 'closure' to be on the stack where as the trampoline code puts it in '%eax'.  This causes a bogus 'closure' and a huge value for 'cif->nargs', which overflows the stack with 'alloca'.

If you build Python on OpenSolaris with 'gcc' (pkg install gcc-dev), then everything works fine.  Does 'libffi' claim to support Solaris Studio?  If not, then this can be closed.

Also see issue1544339.  This issue was run into there as well (although no one ever really diagnosed it).
msg143839 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-10 14:30
According to the latest release notes (https://github.com/atgreen/libffi/blob/master/README) the Solaris compiler should be supported for building:

3.0.10 Aug-23-11
        Add support for Apple's iOS.
        Add support for ARM VFP ABI.
        Add RTEMS support for MIPS and M68K.
        Fix instruction cache clearing problems on
        ARM and SPARC.
        Fix the N64 build on mips-sgi-irix6.5.
        Enable builds with Microsoft's compiler.
        Enable x86 builds with Oracle's Solaris compiler.
        Fix support for calling code compiled with Oracle's Sparc
        Solaris compiler.
        Testsuite fixes for Tru64 Unix.
        Additional platform support.

So 'x86' builds work with 'suncc', but I guess no runtime testing was performed, which is consistent with (http://www.moxielogic.org/wiki/index.php?title=Libffi_3.0.11).  I will open a bug with 'libffi'.
msg143842 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-09-10 16:43
Nice debugging work! I compiled --with-system-ffi and there the
issue seems fixed.

Do we have a libffi-3.0.10 release candidate in Modules/_ctypes?
The README says libffi-3.0.10, but the last commit for ffi.c was
17 months ago:

http://hg.python.org/cpython/log/b239b7d9f752/Modules/_ctypes/libffi/src/x86/ffi.c
msg143848 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-11 01:51
I am not sure, but it does seem that way considering the most recent libffi says that 3.0.10 was released on Aug-23-11.  What is the process for updating libffi anyway?

Also, I posted a question concerning the Solaris Studio support on libffi-discuss: http://sourceware.org/ml/libffi-discuss/2011/msg00185.html.
msg163942 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-25 11:35
Ping.  Could we update libffi for 3.3 final?
msg163947 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-06-25 11:46
Coincidentally, I've just asked the Solaris buildbot owner if he could
try out the system libffi. I'm not sure if even the latest libffi
supports suncc out of the box. Meador, did you get any answers on
libffi-discuss?
msg163974 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2012-06-25 14:41
Stefan, nope -- I never got a reply.  I will try again.
msg164034 - (view) Author: Justin Venus (Justin.Venus) Date: 2012-06-26 04:52
I can confirm (see attachement) that `--with-system-ffi` prevents the segfault on test_ctypes with the solaris compiler on Solaris11 x86.  I did not have to update the system at all for this switch to work.
msg164035 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-06-26 05:09
Antoine, is it possible to add --with-system-ffi to the build rules
of the Solaris/suncc bot? This will prevent the segfaults.
msg164062 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-06-26 09:54
> Antoine, is it possible to add --with-system-ffi to the build rules
> of the Solaris/suncc bot? This will prevent the segfaults.

Ok, done.
msg164782 - (view) Author: Justin Venus (Justin.Venus) Date: 2012-07-07 04:28
I can confirm this issue is resolved on Solaris 11 x86 and has been since '--with-system-ffi' was added to the build rules.
msg164920 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-07-07 18:42
Given that --with-system-ffi works, I don't think we need the additional
burden of maintaining patches for libffi/suncc.
History
Date User Action Args
2012-07-07 18:42:37skrahsetstatus: open -> closed
resolution: wont fix
messages: + msg164920

stage: needs patch -> resolved
2012-07-07 04:28:42Justin.Venussetmessages: + msg164782
2012-06-26 09:54:53pitrousetmessages: + msg164062
2012-06-26 05:09:44skrahsetnosy: + pitrou
messages: + msg164035
2012-06-26 04:52:49Justin.Venussetfiles: + python3x_ctypes-systemffi.txt.gz

messages: + msg164034
2012-06-25 14:41:05meador.ingesetmessages: + msg163974
2012-06-25 14:10:02skrahsetnosy: + Justin.Venus
2012-06-25 11:46:43skrahsetmessages: + msg163947
2012-06-25 11:35:15georg.brandlsetnosy: + doko, loewis, georg.brandl
messages: + msg163942
2011-09-11 01:51:29meador.ingesetmessages: + msg143848
2011-09-10 16:43:22skrahsetmessages: + msg143842
2011-09-10 14:30:02meador.ingesetmessages: + msg143839
2011-09-10 14:09:03meador.ingesetnosy: + meador.inge
messages: + msg143838
2011-09-08 00:46:24meador.ingesetstage: needs patch
2011-09-07 15:30:27skrahsetmessages: + msg143691
2011-09-07 12:51:20vstinnersetnosy: + vstinner
messages: + msg143671
2011-09-07 12:45:14skrahcreate