Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_ascii_formatd fails on Mac when built with Intel compiler #69775

Closed
christopherhogan mannequin opened this issue Nov 9, 2015 · 13 comments
Closed

test_ascii_formatd fails on Mac when built with Intel compiler #69775

christopherhogan mannequin opened this issue Nov 9, 2015 · 13 comments
Labels
topic-ctypes type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@christopherhogan
Copy link
Mannequin

christopherhogan mannequin commented Nov 9, 2015

BPO 25589
Nosy @vstinner, @bitdancer, @zware, @serhiy-storchaka

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-31.15:15:10.880>
created_at = <Date 2015-11-09.18:30:53.459>
labels = ['ctypes', 'type-crash']
title = 'test_ascii_formatd fails on Mac when built with Intel compiler'
updated_at = <Date 2020-05-31.15:15:10.877>
user = 'https://bugs.python.org/christopherhogan'

bugs.python.org fields:

activity = <Date 2020-05-31.15:15:10.877>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-31.15:15:10.880>
closer = 'serhiy.storchaka'
components = ['ctypes']
creation = <Date 2015-11-09.18:30:53.459>
creator = 'christopher.hogan'
dependencies = []
files = []
hgrepos = []
issue_num = 25589
keywords = []
message_count = 13.0
messages = ['254398', '254399', '254400', '254404', '254406', '254408', '254425', '254448', '254452', '254823', '256709', '257775', '370481']
nosy_count = 5.0
nosy_names = ['vstinner', 'r.david.murray', 'zach.ware', 'serhiy.storchaka', 'christopher.hogan']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue25589'
versions = ['Python 2.7']

@christopherhogan
Copy link
Mannequin Author

christopherhogan mannequin commented Nov 9, 2015

I successfully built Python on OS X 10.10 with the Intel compiler v15. However, running the regression tests gives me a "segmentation fault: 11" on test_ascii_formated. I think this has to do with libffi, because when I build with the --use-system-ffi flag (using a libffi that I built) all the regression tests pass, but I get a seg fault when I try to import ctypes. Using python's libffi I can import ctypes, but I get seg faults in the tests. Looking at libffi's homepage, it looks like x86-64 Mac OS X isn't on the list of officially supported platforms. I tried contacting the libffi-discuss mailing list about this but have received no response in a month. Please let me know what other information I can provide.

@christopherhogan christopherhogan mannequin added OS-mac tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump labels Nov 9, 2015
@zware
Copy link
Member

zware commented Nov 9, 2015

This doesn't seem to be a problem on the ICC buildbot; David, did you do anything special for libffi/ctypes?

@zware
Copy link
Member

zware commented Nov 9, 2015

Also, I haven't had an issue with this on my own machine.

@ned-deily
Copy link
Member

Chris, how exactly are you running this test that fails and where does the test come from? I may be overlooking something but I do not see any spelling of a test_ascii_formatted in the Python 3.5 source including its embedded ctypes. The ctypes shipped with Python certainly is supported on 64-bit OS X.

@christopherhogan
Copy link
Mannequin Author

christopherhogan mannequin commented Nov 9, 2015

Ned, the test is in Lib/test/test_ascii_formatd.py. Sorry, I spelled it wrong originally. Also, I guess it's only in 2.7. My mistake.

@vstinner
Copy link
Member

vstinner commented Nov 9, 2015

See also issue bpo-24999 about ICC compiler.

@ned-deily
Copy link
Member

Thanks for the updates. Since it's not at all clear what the issue is or that it is Mac-specific, I'm going to bow out of this and remove the Mac tag for now.

@ned-deily ned-deily removed OS-mac tests Tests in the Lib/test dir labels Nov 10, 2015
@christopherhogan
Copy link
Mannequin Author

christopherhogan mannequin commented Nov 10, 2015

I guess since no one can reproduce the error, it must be a problem with how I'm building it. We build our own dependencies (zlib, bzip, sqlite3, openssl, etc.) with the Intel compiler, so it could be any number of things causing the issue. Closing as not a bug.

@christopherhogan christopherhogan mannequin closed this as completed Nov 10, 2015
@christopherhogan christopherhogan mannequin added the invalid label Nov 10, 2015
@bitdancer
Copy link
Member

Someone can correct me if I'm wrong, but my expectation would be that since we use ICC for the main compile on the ICC buildbots, we also compile the bundled ctypes using it. It is hard to see how any of the other modules would affect this.

@bitdancer
Copy link
Member

We've figured out that the difference is the optimization level. Without --with-pydebug, I can reproduce the failure.

@christopherhogan christopherhogan mannequin reopened this Nov 18, 2015
@christopherhogan christopherhogan mannequin removed the invalid label Nov 18, 2015
@zware
Copy link
Member

zware commented Dec 18, 2015

This one is officially weird.

I can reliably reproduce the segfault in test_ascii_formatd on OSX with ICC 15 or 16, using the included libffi in Modules/_ctypes/libffi_osx. There are also segfaults in test_ctypes, test_multiprocessing, and test_threading (all stemming from an issue in ctypes).

Using libffi from Homebrew and --with-system-ffi, all the segfaults go away, and in fact all tests pass (with --with-icc from bpo-25827, which adds '-fp-model strict' to CFLAGS). The bundled 'nearly vanilla' libffi doesn't work with OSX; it doesn't make it through configure.

Somehow, argcount in _ctypes_callproc is being corrupted, but applying this patch somehow makes test_ascii_formatd pass:

diff -r 937774aa1853 Modules/_ctypes/callproc.c
--- a/Modules/_ctypes/callproc.c	Mon Dec 14 03:41:59 2015 +0000
+++ b/Modules/_ctypes/callproc.c	Fri Dec 18 02:10:04 2015 -0600
@@ -1168,6 +1168,7 @@
         PyErr_NoMemory();
         goto cleanup;
     }
+    printf("line 1171, argcount %zd", argcount);
     for (i = 0; i < argcount; ++i) {
         atypes[i] = args[i].ffi_type;
         if (atypes[i]->type == FFI_TYPE_STRUCT)

I have absolutely no idea why that should work. Adding that line before the previous statement still results in segfault, and adding it anywhere after line 1178 also still results in segfault. Even with the magic patch, test_ctypes still segfaults elsewhere. I'm out of my depth as to what could be causing this or how to further diagnose it.

As a workaround, would it be kosher to mandate --with-system-ffi on Darwin when CC=icc? As I recall, the custom libffi is necessary for regular OSX builds to accommodate fat binaries; how concerned are we about supporting fat binaries with ICC?

@christopherhogan
Copy link
Mannequin Author

christopherhogan mannequin commented Jan 8, 2016

The Intel compiler team is looking into this issue. I'll keep everyone updated here.

@serhiy-storchaka
Copy link
Member

Python 2.7 is no longer supported.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-ctypes type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Development

No branches or pull requests

5 participants