Title: build on macOS 11 (beta) does not find system-supplied third-party libraries
Type: Stage:
Components: macOS Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, ronaldoussoren
Priority: high Keywords:

Created on 2020-06-25 15:51 by ned.deily, last changed 2020-06-26 09:39 by ronaldoussoren.

Messages (4)
msg372379 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-25 15:51
When building on macOS 11 (beta), a number of modules that should normally build on macOS fail because the system-supplied third-party libraries are not found.

The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_gdbm                 _hashlib              _lzma
_ssl                  ossaudiodev           readline
spwd                  zlib

The list should look like this (with no additional third-party libs supplied from another source like Homebrew or MacPorts):

The necessary bits to build these optional modules were not found:
_gdbm                 _hashlib              _ssl
ossaudiodev           spwd

The problem is due to a change in the 11 beta versus 10.15 or earlier systems: 
"New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache."

This breaks tests in using find_library_file() to determine if a library is present and in what directory it exists. depends on Lib/distutils/ to do the dirty work. A similar problem arose on earlier macOS releases when header files could no longer be installed in the systems /usr/include; had to be taught to look in the SDK being used implicitly or explicitly by the compiler preprocessor.  We could probably do something like that here while trying to avoid changes that might break downstream supplements/replacements to distutils, for example, setuptools.

There is a workaround: explicitly specify the SDK location to ./configure (you also need to specify the universal archs setting):

./configure \
    --enable-universalsdk=$(xcodebuild -version -sdk macosx Path) \
    --with-universal-archs=intel-64 \
msg372415 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-06-26 08:27
These build for me (master branch), because the support for .tbd files in the macOS SDK still works.

The ones that don't build for me:

- Bits not found: _gdbm, _hashlib, _ssl, ossaudiodev, spwd
- Failed to build: _ctypes, _decimal, _lzma

This was with a "Universal 2" build (which explains _ctypes) and without 3party libraries (which explains _gdbm, _ssl and _lzma).  ossaudiodev and spwd are not supported on macOS even on older versions of the OS.

I can get _decimal to build with a trivial patch to Modules/_decimal/ (I'll create a PR later today). The PR will obviously be untested w.r.t. actually supporting ARM64.
msg372416 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-26 09:21
I should have specified that the failures I saw were just with the default intel-64 arch on the beta.

After thinking about it a bit and reviewing the code, I think the best approach is to handle this in as part of the compiler flags customization.  What's been happening over the years is that Apple has been deprecating and then removing system header files and system library files on running systems and instead forcing the use of an SDK to build with for both. That's fine but, to continue to support a range of systems, we should be able to handle the various default configurations when the user does not explicitly specify a particular SDK to use. The Apple compiler chain does the right thing under the covers so that C compilations just work.'s macosx_sdk_root() now has initialization code to discover the path to the system header files that the compiler is using; it calls the compiler with -v on a dummy program and then scans the compiler output for the include paths searched until it finds either /usr/include or a path ending in .sdk/usr/include. (It's a bit of a hack but a fairly widely-used hack that works with both clang and gcc, at least, and for the Apple-supplied compilers, takes into account the dynamic specification of SDK via xcrun.)

I think the thing to do is move that sdk path detection into _osx_support and then, at the end of other customization steps, if there isn't already a valid -isysroot provided by the user (either through ./configure or CFLAGS et al, add a -isysroot for the discovered default SDK path (or /) into the appropriate customized compiler and linker flags. It would also provide a function to return the SDK path for (the equivalent of the current macosx_sdk_root()) and others to use. and Distutils should then do the right things for both header and library file paths on any configuration (whether header files are installed or not and whether lib files are installed or not, Command Line Tools or Xcode) and, with luck, any other Distutils replacements that also use _osx_support will also do the right thing.

I had contemplated removing _osx_support at some point in the future but, now with the return of universal files, I'm glad we didn't.
msg372417 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-06-26 09:39
I see, and (as expected) get the same output when I build like that.

I agree that we should teach _osx_support about this. I guess we should add a function that returns the SDK root, which can then be used in distutils (and setuptools, ...) instead of looking for -isysroot in CFLAGS.  The new function can than look for the default SDK when there is no sysroot flag.

For 3.10 we should look into cleaning up, it contains some logic that is no longer valid or necessary (such as looking for the architectures supported by /usr/lib/libSystem.dylib).  We should also avoid adding "-isysroot" to the CFLAGS/LDFLAGS to avoid problems when building on user machines that might have a different version of Xcode.
Date User Action Args
2020-06-26 09:39:07ronaldoussorensetmessages: + msg372417
2020-06-26 09:21:10ned.deilysetmessages: + msg372416
2020-06-26 08:27:11ronaldoussorensetmessages: + msg372415
2020-06-25 15:51:04ned.deilycreate