Author indygreg
Recipients Maxime Belanger, Mika_Hawkins, corona10, dirn, eamanu, erlendaasland, indygreg, jmr, jreese, lawrence-danna-apple, mattip, miss-islington, mistydemeo, ned.deily, nieder, python-dev, ronaldoussoren, skrah
Date 2021-02-22.18:32:32
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1614018753.78.0.0290293150468.issue41100@roundup.psfhosted.org>
In-reply-to
Content
I ran into a minor issue with the ctypes extension and the _dyld_shared_cache_contains_path dependency that was added as part of this work (commit e8b1c038b14b5fc8120aab62c9bf5fb840274cb6).

When building on Intel macOS 10.15 using an 11.x SDK, the configure check for this symbol passes (#define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH 1) and the extension compiles (albeit with a lot of deprecation warnings spewing from the Apple SDK headers). However, at run-time, we blow up due to a missing symbol error:

```
2021-02-21T21:57:21.4116370Z cpython-3.9> clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/_decimal.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/basearith.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/constants.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/context.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/convolute.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/crt.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/difradix2.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/fnt.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/fourstep.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/io.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/mpalloc.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/mpdecimal.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/numbertheory.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/sixstep.o build/temp.macosx-11.0-x86_64-3.9/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/Python-3.9.2/Modules/_decimal/libmpdec/transpose.o -L/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpasb48bzk/tools/pyhost/lib -L/usr/local/lib -lm -o build/lib.macosx-11.0-x86_64-3.9/_decimal.cpython-39-darwin.so
2021-02-21T21:57:27.1635100Z cpython-3.9> *** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path
2021-02-21T21:57:27.1637220Z cpython-3.9>   Referenced from: build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so (which was built for Mac OS X 11.0)
2021-02-21T21:57:27.1638560Z cpython-3.9>   Expected in: /usr/lib/libSystem.B.dylib
2021-02-21T21:57:27.1639770Z cpython-3.9>  in build/lib.macosx-11.0-x86_64-3.9/_ctypes.cpython-39-darwin.so
```

(This build output is from https://github.com/indygreg/python-build-standalone/runs/1947611547?check_suite_focus=true)

The missing symbol is provided by libdyld.dylib, which doesn't appear to be linked directly by CPython's build system.

I'm only able to reproduce this on a GitHub Actions 10.5 machine using the 11.1 SDK: it seems to work fine on both an 11.1 Intel and M1 device.

I reckon the correct fix is to have _ctypes link libdyld is this build configuration. I was also able to work around it in python-build-standalone by unsetting HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH (hacky solution but this feature isn't needed by the build that was failing). And of course linking a non-Apple libffi should also work, which is potentially why pyenv/Homebrew/etc users aren't noticing.

python-build-standalone uses its own Clang built from source, so it is entirely possible there's something differing between the Apple Clang and my Clang causing this to fail. However, at least 1 other person ran into this in the wild (https://stackoverflow.com/questions/65853539/symbol-not-found-dyld-shared-cache-contains-path-on-osx-after-installl-from), so it might be generic.

While I'm here, I've taught python-build-standalone to cross-compile CPython 3.9 from both Intel and M1 Apple devices (without using universal binaries). I also have some (hacky) support for targeting iPhone/iWatch/iTV. If there's interest, I could start sending patches upstream. Let me know!
History
Date User Action Args
2021-02-22 18:32:33indygregsetrecipients: + indygreg, ronaldoussoren, ned.deily, skrah, jmr, jreese, python-dev, dirn, mattip, Maxime Belanger, eamanu, corona10, miss-islington, mistydemeo, erlendaasland, nieder, lawrence-danna-apple, Mika_Hawkins
2021-02-22 18:32:33indygregsetmessageid: <1614018753.78.0.0290293150468.issue41100@roundup.psfhosted.org>
2021-02-22 18:32:33indygreglinkissue41100 messages
2021-02-22 18:32:32indygregcreate