classification
Title: C Extensions on Darwin that link against libpython are likely to crash
Type: crash Stage: resolved
Components: Build, C API, Extension Modules, macOS Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, froody, ned.deily, ronaldoussoren, steve.dower
Priority: low Keywords:

Created on 2020-12-10 20:21 by froody, last changed 2021-02-03 21:44 by froody.

Messages (9)
msg382841 - (view) Author: Tom Birch (froody) Date: 2020-12-10 20:21
After https://github.com/python/cpython/pull/12946, there exists an issue on MacOS due to the two-level namespace for symbol resolution. If a C extension links against libpython.dylib, all symbols dependencies from the Python C API will be bound to libpython. When the C extension is loaded into a process launched by running the `python` binary, there will be two active copies of the python runtime. This can lead to crashes if objects from one runtime are used by the other.

https://developer.apple.com/library/archive/documentation/Porting/Conceptual/PortingUnix/compiling/compiling.html#//apple_ref/doc/uid/TP40002850-BCIHJBBF

See issue/test case here: https://github.com/PDAL/python/pull/76
msg382860 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-11 10:22
Does this affect unix-style builds with --enable-shared or framework builds?

For the latter the python executable is linked to the framework, and lib python.dylib is an alias for the framework. 


A related known problem is that linking an extension to a different interpreter than it was loaded into. That is, link to /opt/lib/libpython.dylib, and use with a framework build (of v.v.).  That's one reason why linking extensions to libpython is not commonly done.

It might be interesting to experiment with "-bundle_loader ..." in the link flags instead of linking to libpython, this checks symbols not found in one of the linked to libraries against the bundle_loader ("like it was one of the dynamic libraries the bundle was linked with").  I don't know if this is a recursive check that will also look at libraries linked to by the bundle loader.
msg382878 - (view) Author: Tom Birch (froody) Date: 2020-12-11 19:37
> Does this affect unix-style builds with --enable-shared or framework builds?

I believe the answer is no, since in both those cases the `python` executable doesn't contain definitions for any of the libpython symbols. In my testing I was using a python binary from anaconda, where the `python` executable defines all the symbols found in libpython.
msg382883 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-12-11 20:00
Stop adding me to this issue.
msg382885 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-12-11 20:03
Sorry, somehow bpo decided I added two people to this issue?  Weird.  Anyway I have removed them (Ned and "froody").
msg382907 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-12 10:18
I've removed two spammy comments and adjusted the categories (the latter explains Larry's annoyance, someone selected all categories which automatically added him to the nosy list).
msg382908 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-12 10:26
Linking extensions to libpython with a static build of python will cause problems (as you described in your initial message).

The not very satisfactory response is "don't do that then".  Note that distutils won't link against libpython by default.

Longer term I want to look into "-bundle_loader ..." because that might allow us to drop "-undefined dynamic_lookup" from the linker flags and hence give a nicer development experience (build-time error instead of runtime error when symbols cannot be found).
msg386226 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:03
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
msg386458 - (view) Author: Tom Birch (froody) Date: 2021-02-03 21:42
Steve Dower: this issue is independent of distutils, reopening
History
Date User Action Args
2021-02-03 21:44:00froodysetcomponents: + Extension Modules, C API
2021-02-03 21:42:22froodysetstatus: closed -> open

messages: + msg386458
components: - Distutils
2021-02-03 18:03:15steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386226

resolution: out of date
stage: resolved
2020-12-12 10:26:03ronaldoussorensetmessages: + msg382908
2020-12-12 10:18:51ronaldoussorensetmessages: + msg382907
2020-12-12 10:17:37ronaldoussorensetnosy: + froody, ned.deily, dstufft, eric.araujo, - asvetlov, yselivanov, Alex.Willmer, zhtw1234
components: + Distutils, - 2to3 (2.x to 3.x conversion tool), ctypes, Cross-Build, asyncio, Argument Clinic, C API
versions: - Python 3.6, Python 3.7
2020-12-12 10:16:45ronaldoussorensetmessages: - msg382879
2020-12-12 10:16:24ronaldoussorensetmessages: - msg382880
2020-12-11 20:03:52larrysetnosy: - larry
2020-12-11 20:03:44larrysetnosy: + larry, - ned.deily, froody
messages: + msg382885
2020-12-11 20:02:09larrysetnosy: - larry
2020-12-11 20:00:31larrysetnosy: ronaldoussoren, larry, ned.deily, asvetlov, yselivanov, Alex.Willmer, zhtw1234, froody
messages: + msg382883
2020-12-11 19:59:09zhtw1234setfiles: - IMAG0629_1.jpg
2020-12-11 19:58:11zhtw1234setnosy: + larry
messages: + msg382880
2020-12-11 19:56:37larrysetnosy: - larry
2020-12-11 19:54:24zhtw1234setfiles: + IMAG0629_1.jpg
versions: + Python 3.6, Python 3.7, Python 3.10
nosy: + zhtw1234, yselivanov, Alex.Willmer, asvetlov, larry

messages: + msg382879

components: + Build, 2to3 (2.x to 3.x conversion tool), ctypes, Cross-Build, asyncio, Argument Clinic
2020-12-11 19:37:12froodysetmessages: + msg382878
2020-12-11 10:22:08ronaldoussorensetpriority: normal -> low

messages: + msg382860
2020-12-10 20:21:24froodycreate