Python-binaries compiled on either Big Sur or Catalina - and moved to the other MacOS-version will not work as expected when code depends on ctypes.util.find_library.

Example symptom of this issue: jupyterlab/jupyterlab#9863
I have personally faced this when using Python from nixpkgs - since nixpkgs Python has been built on Catalina - and I'm using Big Sur.

Scenario 1: Compile on Catalina, copy binaries to BigSur, and call ctypes.util.find_library('c')
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
None

Scenario 2: Compile on Big Sur, copy binaries to Catalina, and call ctypes.util.find_library('c'):
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:28:48) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path
  Referenced from: /usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so (which was built for Mac OS X 11.4)
  Expected in: /usr/lib/libSystem.B.dylib
 in /usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so

The problem with moving from Catalina to Big Sur is a known issue, AFAIK there's an open issue for this.

The problem is that Big Sur moved system libraries into a big blob (which Apple calls the shared library cache). Ctypes uses an API that's new in macOS 11 to check if a library is in that cache, but only when compiled with the the macOS 11 SDK or later as the API is not available in earlier SDKs.

Moving from Big Sur to earlier version should work fine, but only if you set the deployment target correctly during the build. This is how the "universal2" installers on python.org are build.

Anyways, the solution for "build on older macOS version, deploy to Big Sur" is to dynamically look for the relevant API (_dyld_shared_cache_contains_path) and use it when available. But only in that scenario, the current code path using explicit weak linking should be kept for those building using a recent SDK (cleaner code, better error reporting).

This should be a fairly easy patch, but I don't know when I'll get around to looking into this further.

Alternatively we could require that Python is build using the macOS 11 SDK (or later) when targeting Big Sur.

I'm dropping 3.8 from the list of versions because it is in "bug fix only" mode and won't receive a patch for this. IIRC 3.8 also doesn't support Big Sur in the first place, we've only back ported Big Sur support to 3.9.

An alternative to using _dyld_shared_cache_contains_path is to use dlopen to check for library existence (which is what Apple recommends in their change notes: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes).

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

I have created a PR which modifies the current find_library from using _dyld_shared_cache_contains_path to dlopen. It passes all of the existing find_library-tests:
#27251

There might be downsides to using dlopen (performance?) or something else I haven't considered. The huge upside however, is that the function is basically available on all Unix-systems.

The disadvantage of using dlopen is that this function has side effects, and those can affect program behaviour. Because of this I'm against switching to using dlopen to probe for libraries.

You are absolutely right! From the manpage of dlopen(3) on MacOS Big Sur:

dlopen() examines the mach-o file specified by path. If the file is compatible with the current process and has not already been loaded into the current process, it is loaded and linked. After being linked, if it contains any initializer functions, they are called, before dlopen() returns. dlopen() can load dynamic libraries and bundles. It returns a handle that can be used with dlsym() and dlclose(). A second call to dlopen() with the same path will return the same handle, but the internal reference count for the handle will be incremented. Therefore all dlopen() calls should be balanced with a dlclose() call.

Essentially, if the shared library contains initializer functions that have some kind of side-effects, dlopen will also trigger these side-effects.

Basic example:

// mylib.c
#include <stdio.h>
void myinit(void) {
    printf("Hello from mylib\n");
}
__attribute__((section("__DATA,__mod_init_func"))) typeof(myinit) *__init = myinit;

---

// main.c
#include <dlfcn.h>
#include <stdio.h>
int main(void) {
    void* handle = dlopen("./mylib.dyld", RTLD_LAZY);
    if (handle == NULL) printf("Failed to load"); 
}

$ clang mylib.c -shared -o mylib.dyld
$ clang main.c -o main
$ ./main
Hello from mylib

Okay, I decided to look into how I could do dynamic loading as you suggested.

Here is a POC (executable) for using _dyld_shared_cache_contains_path when available:

#include <stdio.h>
#include <dlfcn.h>

void* libsystemb_handle;
typedef bool (*_dyld_shared_cache_contains_path_f)(const char* path);
_dyld_shared_cache_contains_path_f _dyld_shared_cache_contains_path;

bool _dyld_shared_cache_contains_path_fallback(const char* name) {
    return false;
}

__attribute__((constructor)) void load_libsystemb(void) {
    if (
        (libsystemb_handle = dlopen("/usr/lib/libSystem.B.dylib", RTLD_LAZY)) == NULL ||
        (_dyld_shared_cache_contains_path = dlsym(libsystemb_handle, "_dyld_shared_cache_contains_path")) == NULL
    ) {
        _dyld_shared_cache_contains_path = _dyld_shared_cache_contains_path_fallback;
    }
}

__attribute__((destructor)) void unload_libsystemb(void) {
    if (libsystemb_handle != NULL) {
        dlclose(libsystemb_handle);
    }
}

int main(int argc, char ** argv) {
    printf("Library exists in cache: %d\n", _dyld_shared_cache_contains_path(argv[1]));
}

A fallback function is used when _dyld_shared_cache_contains_path cannot be loaded, which always returns false. If there is no cache - the (nonexistent) cache also does not contain whatever path you pass it.

The constructor function is called when the Python extension is loaded - ensuring that _dyld_shared_cache_contains_path is defined and callable. I've read that C extension modules cannot be autoreloaded (https://ipython.org/ipython-doc/3/config/extensions/autoreload.html) - so this might mean there is no need for a deconstructor? Instead the OS would handle cleanup once the process exits?

This could be compiled on either MacOS Catalina or Big Sur, and run without problems on the other MacOS version.

Regarding the "explicit weak linking" when building on MacOS Big Sur and later; wouldn't this mean that a Big Sur build wouldn't work on Catalina?

Would you be positive towards a PR with the approach I demonstrated here?

Regarding the "explicit weak linking" when building on MacOS Big Sur and later; wouldn't this mean that a Big Sur build wouldn't work on Catalina?

No, if it is done correctly. I think you are trying to solve the wrong problem here. As Ronald noted earlier, we now fully support building Python on a newer version of macOS to run correctly on that version and on older versions (for current python.org-provided macOS binary installers, we support one build that runs on macOS 10.9 through 11 Big Sur). The key to this is weak-linking with the help of Apple-provided availability macros and by properly setting the MACOSX_DEPLOYMENT_TARGET variable when running ./configure to the oldest desired supported macOS release.

Because this area is neither well-understood nor well-documented, let me try to get it written down here at the risk of covering some familiar ground.

To support more than one version of macOS when building Python, there are basically two approaches: either build on the oldest targeted system and trust that Apple will continue to provide compatibility when running older binaries on new systems; or, build on the newest targeted system and dynamically test at runtime whether specific newer OS features are available and gracefully handle cases where they are not available (which we call "weak-linking" here for short).

Prior to Python 3.9.1, we did not support the latter approach, i.e. weak-linking for many APIs / features added in recent macOS releases. So our practice and recommendation was to always build on the oldest macOS release to be supported. That's the approach we took for many years, for example, with the macOS 64-bit Intel installer variant for 10.9+ systems. Because Apple has had a very good track record of providing compatibility on newer systems (at least for the mostly C-based APIs CPython uses), that approached worked reasonably well. The main drawback was that certain new features added to Python, primarily in the os module, were not available when using the python.org installer binaries on newer (post-10.9) systems. That was not ideal but, for the most part, the missing features weren't commonly used yet and this was essentially only an issue if you were using the python.org-supplied binaries; you could always use or build a Python targeted for the system in use.

However, things changed with macOS 11 Big Sur and the removal of system library files which broke ctype's find_library() when searching for system files, the subject of this issue. There were a number of other changes needed in CPython to fully support Big Sur, as documented in bpo-41100 and others. As part of that work, Ronald and Lawrence D'Anna bit the bullet and went through the interpreter and the standard library to finally properly support weak-linking for multiple macOS versions. That means, as of 3.9.1 with the introduction of Big Sur support, it is finally possible to build on newer systems but still work properly on older ones. For 3.9.1, we introduced a new python.org installer variant, the "universal2" variant, that provides Intel and Apple Silicon fat binaries that should work on all Macs that can run macOS 10.9 through at least 11 with newer features conditionally tested at runtime.

So our recommendation has changed as of 3.9.1 to now use the second approach above (which previously could cause Python segfaults when running on older systems) and to deprecate and phase out the use of the first approach (which still works as before - i.e. missing some features - with the notable exception of find_library() with system libraries on Big Sur). Note that the find_library() issue is only one reason for that change in recommendation.

How does this work? Here's a quick demo using current head of Python 3.10 (although you should see similar results with Python 3.9.x as of 3.9.1), the latest versions of macOS 11, 10.15, and 10.9. We'll build on 11 in all cases, then deploy and run test_ctypes and test_posix on 11, 10.15, and 10.9.

--------------------------------------------

1. Default case, no MACOSX_DEPLOYMENT_TARGET specified. If the deployment target is not specified, configure normally uses the operating system version the build is running on, so in this case, 11 Big Sur.

$ sw_vers
ProductName:	macOS
ProductVersion:	11.5.1
BuildVersion:	20G80
$ ./configure --prefix=/tmp/py && make -j3 && make install
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 11.5
[...]

# run on 11, works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 16:46:59) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.86 Run tests sequentially
0:00:00 load avg: 1.86 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 762 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.87 Run tests sequentially
0:00:00 load avg: 1.87 [1/1] test_posix

== Tests result: SUCCESS ==

1 test OK.

Total duration: 758 ms
Tests result: SUCCESS

# same build, copied to 10.15 system: find_library test fails and test_posix fails
% sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H1323
sysadmin@pyb15 ~ % /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 16:46:59) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 0.97 Run tests sequentially
0:00:00 load avg: 0.97 [1/1] test_ctypes
test_ctypes skipped -- dlopen(/tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path
  Referenced from: /tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so (which was built for Mac OS X 11.5)
  Expected in: /usr/lib/libSystem.B.dylib
 in /tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so
test_ctypes skipped

== Tests result: SUCCESS ==

1 test skipped:
test_ctypes

Total duration: 72 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.04 Run tests sequentially
0:00:00 load avg: 2.04 [1/1] test_posix
dyld: lazy symbol binding failed: Symbol not found: _preadv
Referenced from: /tmp/py/bin/python3.10 (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: _preadv
Referenced from: /tmp/py/bin/python3.10 (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib

Fatal Python error: Aborted

Current thread 0x000000011373bdc0 (most recent call first):
File "/tmp/py/lib/python3.10/test/test_posix.py", line 302 in test_preadv
File "/tmp/py/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
File "/tmp/py/lib/python3.10/unittest/case.py", line 592 in run
File "/tmp/py/lib/python3.10/unittest/case.py", line 652 in __call__
File "/tmp/py/lib/python3.10/unittest/suite.py", line 122 in run
File "/tmp/py/lib/python3.10/unittest/suite.py", line 84 in __call__
File "/tmp/py/lib/python3.10/unittest/suite.py", line 122 in run
File "/tmp/py/lib/python3.10/unittest/suite.py", line 84 in __call__
File "/tmp/py/lib/python3.10/test/support/testresult.py", line 169 in run
File "/tmp/py/lib/python3.10/test/support/init.py", line 971 in _run_suite
File "/tmp/py/lib/python3.10/test/support/init.py", line 1096 in run_unittest
File "/tmp/py/lib/python3.10/test/test_posix.py", line 2176 in test_main
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 297 in _runtest_inner2
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 335 in _runtest_inner
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 215 in _runtest
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 245 in runtest
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 437 in run_tests_sequential
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 535 in run_tests
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 708 in _main
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 655 in main
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 733 in main
File "/tmp/py/lib/python3.10/test/main.py", line 2 in <module>
File "/tmp/py/lib/python3.10/runpy.py", line 86 in _run_code
File "/tmp/py/lib/python3.10/runpy.py", line 196 in _run_module_as_main

Extension modules: _testcapi (total: 1)

# same build, copied to 10.9 system: interpreter fails to launch
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.9.5
BuildVersion: 13F1911
$ /tmp/py/bin/python3.10
dyld: lazy symbol binding failed: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib

Trace/BPT trap: 5

--------------------------------------------
2. Build on Big Sur 11 with MACOSX_DEPLOYMENT_TARGET=10.15

$ ./configure --prefix=/tmp/py MACOSX_DEPLOYMENT_TARGET=10.15
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 10.15
[...]

# run on 11, still works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 17:00:45) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 736 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_posix

== Tests result: SUCCESS ==

1 test OK.

# same build, copied to 10.15 system: test_ctypes and test_posix both now pass

% /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 17:00:45) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
% /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.29 Run tests sequentially
0:00:00 load avg: 1.29 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 908 ms
Tests result: SUCCESS
% /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.09 Run tests sequentially
0:00:00 load avg: 1.09 [1/1] test_posix

== Tests result: SUCCESS ==

1 test OK.

Total duration: 812 ms
Tests result: SUCCESS

# same build, copied to 10.9: still fails to launch
$ /tmp/py/bin/python3.10
dyld: lazy symbol binding failed: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib

Trace/BPT trap: 5

--------------------------------------------
3. Build on Big Sur 11 with MACOSX_DEPLOYMENT_TARGET=10.9

$ ./configure --prefix=/tmp/py MACOSX_DEPLOYMENT_TARGET=10.9
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 10.9
[...]

# run on 11, still works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 17:07:23) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 2.31 Run tests sequentially
0:00:00 load avg: 2.31 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 646 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.12 Run tests sequentially
0:00:00 load avg: 2.12 [1/1] test_posix

== Tests result: SUCCESS ==

1 test OK.

Total duration: 767 ms
Tests result: SUCCESS

# same build run on 10.15, still works as expected
% /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 17:07:23) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.65 Run tests sequentially
0:00:00 load avg: 1.65 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 964 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.68 Run tests sequentially
0:00:00 load avg: 1.68 [1/1] test_posix

== Tests result: SUCCESS ==

1 test OK.

Total duration: 795 ms
Tests result: SUCCESS

# same build run on 10.9, now also works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug  4 2021, 17:07:23) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 0.68 Run tests sequentially
0:00:00 load avg: 0.68 [1/1] test_ctypes

== Tests result: SUCCESS ==

1 test OK.

Total duration: 398 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 0.62 Run tests sequentially
0:00:00 load avg: 0.62 [1/1] test_posix

== Tests result: SUCCESS ==

1 test OK.

Total duration: 543 ms
Tests result: SUCCESS

--------------------------------------------

To summarize, we believe that building on current systems and targeting older systems by using MACOS_DEPLOYMENT_TARGET should work properly and, going forward from 3.9.1, is the recommended and supported method for current multi-version macOS builds for 3.9.x+. (For the record, note that, while 3.8.10, the final maintenance release of the 3.8 series, did gain support for running natively on macOS 11 when built on macOS 11, the much more invasive weak-linking support was not backported. 3.8.x is now in the security-fix-only phase of its life cycle.)

So, what are the remaining issues? I believe them to be and in rough priority order:

1. Better document how to build Python on macOS for multiple releases. Mac/README.txt was updated to include a section on weak linking support but it should be expanded and should give more concrete examples along the lines of above. We should do that for the next releases: 3.9.7 and 3.10.0. We should also re-iterate that building Python for multiple macOS versions and/or architectures is only supported when building on macOS using the Apple-supplied build tools (I will make sure this gets done.)

2. Make the universal2 10.9+ python.org installer variant the default download for 3.9.7 and mark the Intel-only 10.9+ variant as deprecated for 3.9.7 and drop in a future 3.9.x maintenance release. (I will also make sure this gets done.) With 3.10.0 pre-releases, we are already only providing the 10.9+ universal2 variant.

3. Consider accepting a PR to support find_library() on macOS 11+ in the legacy and deprecated case of building on older versions of macOS to also run on newer versions. The current revision of PR 27251 is still problematic; as Ronald noted before, "the current code path using explicit weak linking should be kept for those building using a recent SDK (cleaner code, better error reporting)." And the description is misleading as demonstrated above.

Opinions?

BTW, thanks again for all your work so far on this issue, Tobias. Because of it, I think we will end up with some major improvements for both builders and users of Python.

https://stackoverflow.com/questions/25352389/what-is-the-difference-between-macosx-deployment-target-and-mmacosx-version-min

This makes a lot of sense now.

Thank you so much for the thorough explanation Ned - and for highlighting where my assumptions were wrong!

I didn’t realise I needed to specify MACOSX_DEPLOYMENT_TARGET to enable backwards compatibility. Is there a reason for not setting this to as old a version as possible by default when running ./configure? (Bigger binaries? Or something else?)

If I understand correctly, the following two statements are now equivalent (in Python >= 3.9, but not in Python == 3.8), after the added Python-support for weak linking you mention:

if (__builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)) { // ...
If (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) { // ...

From:

And in order to support the deprecated case of building on an older MacOS-version, I would want to do something like the following:

#ifdef __APPLE__
#ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH
// leave current implementation as is
#else
// dynamic loading implementation
#endif
#endif

That way, dynamic loading will only be used when building on old MacOS, and we keep the current code path using weak linking when possible.

This means we will still get useful compiler errors if for example Apple decides to suddenly remove the _dyld_shared_cache_contains_path symbol again in the future, or change its signature - rather than having to wait for the test suite to fail. It makes sense to prioritise good error reporting for the latest version of MacOS, since this will reduce the debugging time for CPython developers whenever Apple introduces new breaking changes to MacOS.

I realised that I needed to define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME in the source file myself - as this is not defined after running the configure-script. I've updated the PR and its description to only focus on the legacy/deprecated approach on building on Catalina and running on Big Sur.

Now a dynamic loading version of _dyld_shared_cache_contains_path is only used when compiling on MacOS < 11 (Catalina or older). And the weak-linking approach is used when HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH is defined (MacOS >= 11).

New changeset 71853a7 by Tobias Bergkvist in branch 'main':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (bpo-27251)
71853a7

New changeset 7234e67 by Miss Islington (bot) in branch '3.8':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (GH-27251) (GH-28054)
7234e67

New changeset 4b55837 by Miss Islington (bot) in branch '3.9':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (GH-27251) (GH-28053)
4b55837

New changeset 41c87c4 by Miss Islington (bot) in branch '3.10':
bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (GH-27251)
41c87c4

I think patch broke building on macOS < 11 when building with a 11.0+ SDK and targeting macOS < 11? This build configuration previously used to work with 3.9.6.

clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1 -DPy_BUILD_CORE_BUILTIN -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o Modules/callproc.o
python-3.9> ./Modules/_ctypes/callproc.c:1459:15: error: redefinition of '_dyld_shared_cache_contains_path' as different kind of symbol
cpython-3.9> static bool (_dyld_shared_cache_contains_path)(const char *path);
cpython-3.9> ^
cpython-3.9> /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13: note: previous definition is here
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char path) __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:42: error: non-object type 'bool (const char *)' is not assignable
cpython-3.9> _dyld_shared_cache_contains_path = dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:9: error: '_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer [-Werror,-Wunguarded-availability-new]
cpython-3.9> _dyld_shared_cache_contains_path = dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13: note: '_dyld_shared_cache_contains_path' has been marked as being introduced in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path) __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1464:9: note: enclose '_dyld_shared_cache_contains_path' in a __builtin_available check to silence this warning
cpython-3.9> _dyld_shared_cache_contains_path = dlsym(libsystem_b_handle, "_dyld_shared_cache_contains_path");
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1482:10: error: '_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer [-Werror,-Wunguarded-availability-new]
cpython-3.9> if (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) {
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1474:5: note: expanded from macro 'HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME'
cpython-3.9> _dyld_shared_cache_contains_path != NULL
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13: note: '_dyld_shared_cache_contains_path' has been marked as being introduced in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path) __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1482:10: note: enclose '_dyld_shared_cache_contains_path' in a __builtin_available check to silence this warning
cpython-3.9> if (HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME) {
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1474:5: note: expanded from macro 'HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME'
cpython-3.9> _dyld_shared_cache_contains_path != NULL
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> ./Modules/_ctypes/callproc.c:1495:14: error: '_dyld_shared_cache_contains_path' is only available on macOS 11.0 or newer [-Werror,-Wunguarded-availability-new]
cpython-3.9> r = _dyld_shared_cache_contains_path(name_str);
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cpython-3.9> /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/mach-o/dyld.h:121:13: note: '_dyld_shared_cache_contains_path' has been marked as being introduced in macOS 11.0 here, but the deployment target is macOS 10.9.0
cpython-3.9> extern bool _dyld_shared_cache_contains_path(const char* path) __API_AVAILABLE(macos(11.0), ios(14.0), watchos(7.0), tvos(14.0)) DYLD_DRIVERKIT_UNAVAILABLE;
cpython-3.9> ^
cpython-3.9> ./Modules/_ctypes/callproc.c:1495:14: note: enclose '_dyld_shared_cache_contains_path' in a __builtin_available check to silence this warning
cpython-3.9> r = _dyld_shared_cache_contains_path(name_str);
cpython-3.9> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It seems like _dyld_shared_cache_contains_path exists, while the define flag HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH has not been set.

Essentially, the define-flags being used does not seem to agree with the SDK version you are using.

Could you try adding HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH like this and see if it builds correctly?

clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1 -HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH=1 -DPy_BUILD_CORE_BUILTIN -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o Modules/callproc.o

I made a typo (forgetting the -D):

clang -Wno-unused-result -Wsign-compare -g -O0 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpkfji88v7/tools/deps/include/uuid -Werror=unguarded-availability-new -DMACOSX -DUSING_MALLOC_CLOSURE_DOT_C=1 -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1 -DHAVE_DYLD_SHARED_CACHE_CONTAINS_PATH=1 -DPy_BUILD_CORE_BUILTIN -I_ctypes/darwin -c ./Modules/_ctypes/callproc.c -o Modules/callproc.o

Oh, this might be my custom Modules file not passing the HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH define properly. I suspect I was working around this bug before by disabling the dyld bits. Sorry for the noise! (I should have reproduced with CPython's build system.)

I spoke too soon: you can reproduce this with CPython's build system and I think this is a legit regression.

I think the function sniffing logic is subtly wrong.

Here is the logic as written:

#ifdef __APPLE__
  #ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH
    #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \
      __builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)
  #else
    static bool (*_dyld_shared_cache_contains_path)(const char *path);

  #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \
      _dyld_shared_cache_contains_path != NULL
  #endif
#endif

The fundamental problem is that HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH comes from configure. Configure is using the target settings to probe for function presence. If I set the deployment target to 10.9, it says checking for _dyld_shared_cache_contains_path... no. But if I set the deployment target to 11.0, it says checking for _dyld_shared_cache_contains_path... yes. Because we may be targeting a macOS without the function, the function appears as not available to configure. It may exist in the SDK, but it is masked behind availability checks for the test compile.

The logic as written assumes that !HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH means the SDK doesn't contain the function at all. This just isn't true for SDKs >= 11.0.

If you look at posixmodule.c, the logic for checking for function availability is typically this pattern:

#ifdef __APPLE__
  #ifdef HAVE_BUILTIN_AVAILABLE
    #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME \
      __builtin_available(macOS 11.0, iOS 14.0, tvOS 14.0, watchOS 7.0, *)
  #else
    #ifdef HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH 
      #define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH_RUNTIME (_dyld_shared_cache_contains_path != NULL)
    #endif
  #endif
#endif

This other pattern also reveals another regression with this patch: __builtin_available() may not be available on older compilers. See bpo-42692. I'm unsure what "older compilers" actually is. But someone is bound to complain about this (if they haven't already).

Do these build regressions warrant an unplanned 3.9.8 release?

I don't think we have ever claimed to support building on an older system with a newer SDK, as in building on 10.15 with an 11 SDK. I am sure there have been problems with trying to do this in the past for some releases. It *may* work but there are no guarantees. If you want to build on an older system you should use the SDK for that system even if a newer version of Xcode / Command Line Tools is released that provides the newer SDK. That said, I have no objection to trying to fix this issue but I think we should avoid claiming to support such configurations and I don't see this issue as needing to make a new release. I am willing to be convinced otherwise :)

On Sep 1, 2021, at 22:58, Ned Deily <report@bugs.python.org> wrote:

I don't think we have ever claimed to support building on an older system with a newer SDK, as in building on 10.15 with an 11 SDK.

My initial report was from a 10.15 Intel machine in GitHub Actions. My comment a few hours ago was from an 11.3 M1 cross-compiling to x86_64 and targeting 10.9 from a 11.X SDK.

I cannot reproduce this on an 11.5 Intel MacBook Pro using an 11.5 SDK targeting x86_64 10.9.

However, I can reproduce on a similarly configured M1 using the same OS and SDK but cross-compiling to single arch x86_64.

The issue here may reside in how configure handles cross-compiling for Apple. The current logic in configure is a bit opinionated about how things should work and this patch may have tickled things into not working.

My opinion is that configure should support various cross-compiling scenarios like this (including building non-fat x86_64 only binaries from an arm64 machine). However, it is clear from the implementation and comments in this issue and elsewhere that the CPython core devs want to limit which exact cross-compiling configurations are supported on Apple.

If the core developers say various cross-compiling scenarios aren't supported build configuration any more, I'll accept that and work around the limitation on my end. However, do note there is a significant possibility that Apple stops selling Intel machines for various product models in a few weeks. I suspect various people will want to continue their practice of building x86_64 only binaries for the indefinite future. Regardless of when the ARM only transition occurs, arbitrary restrictions like not supporting Apple arm64 -> x86_64 cross-compiling will disappoint a growing cohort of users over time. I would encourage investing in less opinionated configure logic to support Apple cross-compiling. I could potentially contribute patches in this area, since I've already taught python-build-standalone to cross-compile more robustly (including to targets like iOS).

Thanks for the updates!

If the core developers say various cross-compiling scenarios aren't supported build configuration any more, I'll accept that and work around the limitation on my end. However, do note there is a significant possibility that Apple stops selling Intel machines for various product models in a few weeks. I suspect various people will want to continue their practice of building x86_64 only binaries for the indefinite future.

I think you make very good points. I also think that we all agree that we do need to work towards better supporting the coming primarily non-Intel Mac world. I would rather put it that there are cross-compiling scenarios that were never supported before (but also not necessarily fully documented as such) but may have worked coincidentally some of the time; moving forward, we need to identify the important scenarios and make sure we fully support them. We're not there yet but I have some ideas on how to do that which I will try to get written down soon.

The easiest workaround for the compilation issue mentioned earlier is likely to add

#define _dyld_shared_cache_contains_path _Py__dyld_shared_cache_contains_path just before static void *libsystem_b_handle; in callproc.c. That way the local symbol is renamed and won't cause an issue. I'm willing to review a PR for this, but only if someone else can test as I don't have a test VM with the configuration needed to trigger this issue.

That said, I'm not convinced yet that this is scenario worth supporting:

• For now Apple still supports recent versions of macOS and Xcode on Intel systems
• Once Apple stops doing so they'll also drop Intel support from the latests SDKs if history is any indication of the future (that is, at some point the SDKs included in Xcode dropped support for PPC and later 32-bit Intel).
• It will still be possible to build using the latest/last Xcode that supports Intel, and that doesn't need this workaround.

I can test the workaround PR for you if you prepare it, @ronaldoussoren.

I have a 10.15 machine like @indygreg.

Ignore my previous message, it was not entirely correct. Some time outside gave some insight in the issue:

• Building on 10.15 with the 11.0 SDK is a configuration that we support, that's a setup with the latest Xcode that's supported on 10.15 and is the compiler that will get installed when you update the system and applications.
• The bug is likely in the configure phase, the code path that uses dlsym to find _dyld_shared_cache_contains_path should not be used when building using the 11.0 SDK.

I have a 10.15 VM with the command-line tools that include the 10.15 SDK, I'm creating a clone in which I will update the developer tools to the latest version and that should enable me to reproduce the issue.

To be continued...