> Regarding the "explicit weak linking" when building on MacOS Big Sur and later; wouldn't this mean that a Big Sur build wouldn't work on Catalina?
No, if it is done correctly. I think you are trying to solve the wrong problem here. As Ronald noted earlier, we now fully support building Python on a newer version of macOS to run correctly on that version and on older versions (for current python.org-provided macOS binary installers, we support one build that runs on macOS 10.9 through 11 Big Sur). The key to this is weak-linking with the help of Apple-provided availability macros and by properly setting the MACOSX_DEPLOYMENT_TARGET variable when running ./configure to the oldest desired supported macOS release.
Because this area is neither well-understood nor well-documented, let me try to get it written down here at the risk of covering some familiar ground.
To support more than one version of macOS when building Python, there are basically two approaches: either build on the oldest targeted system and trust that Apple will continue to provide compatibility when running older binaries on new systems; or, build on the newest targeted system and dynamically test at runtime whether specific newer OS features are available and gracefully handle cases where they are not available (which we call "weak-linking" here for short).
Prior to Python 3.9.1, we did not support the latter approach, i.e. weak-linking for many APIs / features added in recent macOS releases. So our practice and recommendation was to always build on the oldest macOS release to be supported. That's the approach we took for many years, for example, with the macOS 64-bit Intel installer variant for 10.9+ systems. Because Apple has had a very good track record of providing compatibility on newer systems (at least for the mostly C-based APIs CPython uses), that approached worked reasonably well. The main drawback was that certain new features added to Python, primarily in the os module, were not available when using the python.org installer binaries on newer (post-10.9) systems. That was not ideal but, for the most part, the missing features weren't commonly used yet and this was essentially only an issue if you were using the python.org-supplied binaries; you could always use or build a Python targeted for the system in use.
However, things changed with macOS 11 Big Sur and the removal of system library files which broke ctype's find_library() when searching for system files, the subject of this issue. There were a number of other changes needed in CPython to fully support Big Sur, as documented in Issue41100 and others. As part of that work, Ronald and Lawrence D'Anna bit the bullet and went through the interpreter and the standard library to finally properly support weak-linking for multiple macOS versions. That means, as of 3.9.1 with the introduction of Big Sur support, it is finally possible to build on newer systems but still work properly on older ones. For 3.9.1, we introduced a new python.org installer variant, the "universal2" variant, that provides Intel and Apple Silicon fat binaries that should work on all Macs that can run macOS 10.9 through at least 11 with newer features conditionally tested at runtime.
So our recommendation has changed as of 3.9.1 to now use the second approach above (which previously could cause Python segfaults when running on older systems) and to deprecate and phase out the use of the first approach (which still works as before - i.e. missing some features - with the notable exception of find_library() with system libraries on Big Sur). Note that the find_library() issue is only one reason for that change in recommendation.
How does this work? Here's a quick demo using current head of Python 3.10 (although you should see similar results with Python 3.9.x as of 3.9.1), the latest versions of macOS 11, 10.15, and 10.9. We'll build on 11 in all cases, then deploy and run test_ctypes and test_posix on 11, 10.15, and 10.9.
--------------------------------------------
1. Default case, no MACOSX_DEPLOYMENT_TARGET specified. If the deployment target is not specified, configure normally uses the operating system version the build is running on, so in this case, 11 Big Sur.
$ sw_vers
ProductName: macOS
ProductVersion: 11.5.1
BuildVersion: 20G80
$ ./configure --prefix=/tmp/py && make -j3 && make install
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 11.5
[...]
# run on 11, works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 16:46:59) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.86 Run tests sequentially
0:00:00 load avg: 1.86 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 762 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.87 Run tests sequentially
0:00:00 load avg: 1.87 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 758 ms
Tests result: SUCCESS
# same build, copied to 10.15 system: find_library test fails and test_posix fails
% sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.7
BuildVersion: 19H1323
sysadmin@pyb15 ~ % /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 16:46:59) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 0.97 Run tests sequentially
0:00:00 load avg: 0.97 [1/1] test_ctypes
test_ctypes skipped -- dlopen(/tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path
Referenced from: /tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib
in /tmp/py/lib/python3.10/lib-dynload/_ctypes.cpython-310-darwin.so
test_ctypes skipped
== Tests result: SUCCESS ==
1 test skipped:
test_ctypes
Total duration: 72 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.04 Run tests sequentially
0:00:00 load avg: 2.04 [1/1] test_posix
dyld: lazy symbol binding failed: Symbol not found: _preadv
Referenced from: /tmp/py/bin/python3.10 (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _preadv
Referenced from: /tmp/py/bin/python3.10 (which was built for Mac OS X 11.5)
Expected in: /usr/lib/libSystem.B.dylib
Fatal Python error: Aborted
Current thread 0x000000011373bdc0 (most recent call first):
File "/tmp/py/lib/python3.10/test/test_posix.py", line 302 in test_preadv
File "/tmp/py/lib/python3.10/unittest/case.py", line 549 in _callTestMethod
File "/tmp/py/lib/python3.10/unittest/case.py", line 592 in run
File "/tmp/py/lib/python3.10/unittest/case.py", line 652 in __call__
File "/tmp/py/lib/python3.10/unittest/suite.py", line 122 in run
File "/tmp/py/lib/python3.10/unittest/suite.py", line 84 in __call__
File "/tmp/py/lib/python3.10/unittest/suite.py", line 122 in run
File "/tmp/py/lib/python3.10/unittest/suite.py", line 84 in __call__
File "/tmp/py/lib/python3.10/test/support/testresult.py", line 169 in run
File "/tmp/py/lib/python3.10/test/support/__init__.py", line 971 in _run_suite
File "/tmp/py/lib/python3.10/test/support/__init__.py", line 1096 in run_unittest
File "/tmp/py/lib/python3.10/test/test_posix.py", line 2176 in test_main
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 297 in _runtest_inner2
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 335 in _runtest_inner
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 215 in _runtest
File "/tmp/py/lib/python3.10/test/libregrtest/runtest.py", line 245 in runtest
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 437 in run_tests_sequential
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 535 in run_tests
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 708 in _main
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 655 in main
File "/tmp/py/lib/python3.10/test/libregrtest/main.py", line 733 in main
File "/tmp/py/lib/python3.10/test/__main__.py", line 2 in <module>
File "/tmp/py/lib/python3.10/runpy.py", line 86 in _run_code
File "/tmp/py/lib/python3.10/runpy.py", line 196 in _run_module_as_main
Extension modules: _testcapi (total: 1)
# same build, copied to 10.9 system: interpreter fails to launch
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.9.5
BuildVersion: 13F1911
$ /tmp/py/bin/python3.10
dyld: lazy symbol binding failed: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
Trace/BPT trap: 5
--------------------------------------------
2. Build on Big Sur 11 with MACOSX_DEPLOYMENT_TARGET=10.15
$ ./configure --prefix=/tmp/py MACOSX_DEPLOYMENT_TARGET=10.15
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 10.15
[...]
# run on 11, still works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:00:45) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 736 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
# same build, copied to 10.15 system: test_ctypes and test_posix both now pass
% /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:00:45) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
% /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.29 Run tests sequentially
0:00:00 load avg: 1.29 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 908 ms
Tests result: SUCCESS
% /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.09 Run tests sequentially
0:00:00 load avg: 1.09 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 812 ms
Tests result: SUCCESS
# same build, copied to 10.9: still fails to launch
$ /tmp/py/bin/python3.10
dyld: lazy symbol binding failed: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
dyld: Symbol not found: _getentropy
Referenced from: /tmp/py/bin/python3.10
Expected in: /usr/lib/libSystem.B.dylib
Trace/BPT trap: 5
--------------------------------------------
3. Build on Big Sur 11 with MACOSX_DEPLOYMENT_TARGET=10.9
$ ./configure --prefix=/tmp/py MACOSX_DEPLOYMENT_TARGET=10.9
[...]
checking which MACOSX_DEPLOYMENT_TARGET to use... 10.9
[...]
# run on 11, still works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:07:23) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 2.31 Run tests sequentially
0:00:00 load avg: 2.31 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 646 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 2.12 Run tests sequentially
0:00:00 load avg: 2.12 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 767 ms
Tests result: SUCCESS
# same build run on 10.15, still works as expected
% /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:07:23) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 1.65 Run tests sequentially
0:00:00 load avg: 1.65 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 964 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 1.68 Run tests sequentially
0:00:00 load avg: 1.68 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 795 ms
Tests result: SUCCESS
# same build run on 10.9, now also works as expected
$ /tmp/py/bin/python3.10
Python 3.10.0rc1+ (heads/3.10:536e35ae6a, Aug 4 2021, 17:07:23) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ /tmp/py/bin/python3.10 -m test test_ctypes
0:00:00 load avg: 0.68 Run tests sequentially
0:00:00 load avg: 0.68 [1/1] test_ctypes
== Tests result: SUCCESS ==
1 test OK.
Total duration: 398 ms
Tests result: SUCCESS
$ /tmp/py/bin/python3.10 -m test test_posix
0:00:00 load avg: 0.62 Run tests sequentially
0:00:00 load avg: 0.62 [1/1] test_posix
== Tests result: SUCCESS ==
1 test OK.
Total duration: 543 ms
Tests result: SUCCESS
--------------------------------------------
To summarize, we believe that building on current systems and targeting older systems by using MACOS_DEPLOYMENT_TARGET should work properly and, going forward from 3.9.1, is the recommended and supported method for current multi-version macOS builds for 3.9.x+. (For the record, note that, while 3.8.10, the final maintenance release of the 3.8 series, did gain support for running natively on macOS 11 when built on macOS 11, the much more invasive weak-linking support was not backported. 3.8.x is now in the security-fix-only phase of its life cycle.)
So, what are the remaining issues? I believe them to be and in rough priority order:
1. Better document how to build Python on macOS for multiple releases. Mac/README.txt was updated to include a section on weak linking support but it should be expanded and should give more concrete examples along the lines of above. We should do that for the next releases: 3.9.7 and 3.10.0. We should also re-iterate that building Python for multiple macOS versions and/or architectures is only supported when building on macOS using the Apple-supplied build tools (I will make sure this gets done.)
2. Make the universal2 10.9+ python.org installer variant the default download for 3.9.7 and mark the Intel-only 10.9+ variant as deprecated for 3.9.7 and drop in a future 3.9.x maintenance release. (I will also make sure this gets done.) With 3.10.0 pre-releases, we are already only providing the 10.9+ universal2 variant.
3. Consider accepting a PR to support find_library() on macOS 11+ in the legacy and deprecated case of building on older versions of macOS to also run on newer versions. The current revision of PR 27251 is still problematic; as Ronald noted before, "the current code path using explicit weak linking should be kept for those building using a recent SDK (cleaner code, better error reporting)." And the description is misleading as demonstrated above.
Opinions?
BTW, thanks again for all your work so far on this issue, Tobias. Because of it, I think we will end up with some major improvements for both builders and users of Python.
https://stackoverflow.com/questions/25352389/what-is-the-difference-between-macosx-deployment-target-and-mmacosx-version-min |
➜
