classification
Title: test_ascii_formatd fails on Mac when built with Intel compiler
Type: crash Stage:
Components: ctypes Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christopher.hogan, r.david.murray, vstinner, zach.ware
Priority: normal Keywords:

Created on 2015-11-09 18:30 by christopher.hogan, last changed 2016-01-08 20:11 by christopher.hogan.

Messages (12)
msg254398 - (view) Author: Chris Hogan (christopher.hogan) * Date: 2015-11-09 18:30
I successfully built Python on OS X 10.10 with the Intel compiler v15.  However, running the regression tests gives me a "segmentation fault: 11" on test_ascii_formated.  I think this has to do with libffi, because when I build with the --use-system-ffi flag (using a libffi that I built) all the regression tests pass, but I get a seg fault when I try to import ctypes.  Using python's libffi I can import ctypes, but I get seg faults in the tests.  Looking at libffi's homepage, it looks like x86-64 Mac OS X isn't on the list of officially supported platforms.  I tried contacting the libffi-discuss mailing list about this but have received no response in a month.  Please let me know what other information I can provide.
msg254399 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-11-09 20:15
This doesn't seem to be a problem on the ICC buildbot; David, did you do anything special for libffi/ctypes?
msg254400 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-11-09 20:15
Also, I haven't had an issue with this on my own machine.
msg254404 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-11-09 20:56
Chris, how exactly are you running this test that fails and where does the test come from?  I may be overlooking something but I do not see any spelling of a test_ascii_formatted in the Python 3.5 source including its embedded ctypes.  The ctypes shipped with Python certainly is supported on 64-bit OS X.
msg254406 - (view) Author: Chris Hogan (christopher.hogan) * Date: 2015-11-09 21:04
Ned, the test is in Lib/test/test_ascii_formatd.py.  Sorry, I spelled it wrong originally.  Also, I guess it's only in 2.7.  My mistake.
msg254408 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-09 21:12
See also issue #24999 about ICC compiler.
msg254425 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-11-10 00:50
Thanks for the updates.  Since it's not at all clear what the issue is or that it is Mac-specific, I'm going to bow out of this and remove the Mac tag for now.
msg254448 - (view) Author: Chris Hogan (christopher.hogan) * Date: 2015-11-10 14:50
I guess since no one can reproduce the error, it must be a problem with how I'm building it.  We build our own dependencies (zlib, bzip, sqlite3, openssl, etc.) with the Intel compiler, so it could be any number of things causing the issue. Closing as not a bug.
msg254452 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-11-10 16:01
Someone can correct me if I'm wrong, but my expectation would be that since we use ICC for the main compile on the ICC buildbots, we also compile the bundled ctypes using it.  It is hard to see how any of the other modules would affect this.
msg254823 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-11-17 23:52
We've figured out that the difference is the optimization level.  Without --with-pydebug, I can reproduce the failure.
msg256709 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-12-18 21:18
This one is officially weird.

I can reliably reproduce the segfault in test_ascii_formatd on OSX with ICC 15 or 16, using the included libffi in Modules/_ctypes/libffi_osx.  There are also segfaults in test_ctypes, test_multiprocessing, and test_threading (all stemming from an issue in ctypes).

Using libffi from Homebrew and --with-system-ffi, all the segfaults go away, and in fact all tests pass (with --with-icc from #25827, which adds '-fp-model strict' to CFLAGS).  The bundled 'nearly vanilla' libffi doesn't work with OSX; it doesn't make it through configure.

Somehow, argcount in _ctypes_callproc is being corrupted, but applying this patch somehow makes test_ascii_formatd pass:

diff -r 937774aa1853 Modules/_ctypes/callproc.c
--- a/Modules/_ctypes/callproc.c	Mon Dec 14 03:41:59 2015 +0000
+++ b/Modules/_ctypes/callproc.c	Fri Dec 18 02:10:04 2015 -0600
@@ -1168,6 +1168,7 @@
         PyErr_NoMemory();
         goto cleanup;
     }
+    printf("line 1171, argcount %zd", argcount);
     for (i = 0; i < argcount; ++i) {
         atypes[i] = args[i].ffi_type;
         if (atypes[i]->type == FFI_TYPE_STRUCT)


I have absolutely no idea why that should work.  Adding that line before the previous statement still results in segfault, and adding it anywhere after line 1178 also still results in segfault.  Even with the magic patch, test_ctypes still segfaults elsewhere.  I'm out of my depth as to what could be causing this or how to further diagnose it.

As a workaround, would it be kosher to mandate --with-system-ffi on Darwin when CC=icc?  As I recall, the custom libffi is necessary for regular OSX builds to accommodate fat binaries; how concerned are we about supporting fat binaries with ICC?
msg257775 - (view) Author: Chris Hogan (christopher.hogan) * Date: 2016-01-08 20:11
The Intel compiler team is looking into this issue.  I'll keep everyone updated here.
History
Date User Action Args
2016-01-08 20:11:17christopher.hogansetmessages: + msg257775
2015-12-18 21:18:39zach.waresetmessages: + msg256709
components: + ctypes
2015-11-18 15:43:25christopher.hogansetstatus: closed -> open
resolution: not a bug ->
2015-11-17 23:52:34r.david.murraysetmessages: + msg254823
2015-11-10 16:01:35r.david.murraysetmessages: + msg254452
2015-11-10 14:50:20christopher.hogansetstatus: open -> closed
resolution: not a bug
messages: + msg254448
2015-11-10 00:50:45ned.deilysetnosy: - ned.deily
2015-11-10 00:50:27ned.deilysetnosy: - ronaldoussoren
messages: + msg254425
components: - macOS, Tests
2015-11-09 21:12:15vstinnersetnosy: + vstinner
messages: + msg254408
2015-11-09 21:04:45christopher.hogansetmessages: + msg254406
versions: - Python 3.5
2015-11-09 20:56:42ned.deilysetmessages: + msg254404
2015-11-09 20:15:45zach.waresetmessages: + msg254400
2015-11-09 20:15:20zach.waresetnosy: + r.david.murray, zach.ware
messages: + msg254399
2015-11-09 18:30:53christopher.hogancreate