classification
Title: setup.py test for macOS SDK files may incorrectly classify files in other file systems
Type: compile error Stage: needs patch
Components: Extension Modules, macOS Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: andrewfg1992, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2020-06-26 18:11 by andrewfg1992, last changed 2020-08-11 16:14 by andrewfg1992.

Files
File name Uploaded Description Edit
configure_and_make_output.txt andrewfg1992, 2020-06-26 18:11 Output of the ./configure and make commands
Messages (13)
msg372437 - (view) Author: Andrew (andrewfg1992) Date: 2020-06-26 18:11
Quick repoduction steps:
1) Log into a mac with macOS version 10.15.1 (10.15.x may work)
2) For build tools, use Xcode11. A minimal xcode command-tools installation also reproduced for me.
3) Download and decompress the latest python 3.8.2 source
4) run "./configure" in the top-level source folder
5) run "make" in the top-level source folder

I believe this error may also occur when using python 2.7.x, 3.7.x, and others. Curiously, the errors do not occur when I use the latest 3.6.10 source.

A text file containing the output of "./configure" and "make" is attached.

Main Description:
On Mac 10.15.1 with Xcode11 I encounter compilation errors when building the latest Python 3.8.2 source. When the build reaches the "build_ext" section in setup.py, none of the extension modules build, with each compilation attempt producing an error. For example, when the _struct extension module attempts compilation I get:

--- Error Syndrome ---
building '_struct' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I./Include -I. -I/usr/local/include -I/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2/Include -I/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2 -c _struct.c -o build/temp.macosx-10.15-x86_64-3.8/_struct.o
clang: error: no such file or directory: '_struct.c'
clang: error: no input files
------

After many similar errors, we get to the main build report where we see the failed extension modules:

--- Snippet of build report ---
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 _hashlib              _ssl               
ossaudiodev           spwd                                     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           


Failed to build these modules:
_asyncio              _bisect               _blake2            
_bz2                  _codecs_cn            _codecs_hk         
_codecs_iso2022       _codecs_jp            _codecs_kr         
_codecs_tw            _contextvars          _crypt             
_csv                  _ctypes               _ctypes_test       
_curses               _curses_panel         _datetime          
_dbm                  _decimal              _elementtree       
_heapq                _json                 _lsprof            
_lzma                 _md5                  _multibytecodec    
_multiprocessing      _opcode               _pickle            
_posixshmem           _posixsubprocess      _queue             
_random               _scproxy              _sha1              
_sha256               _sha3                 _sha512            
_socket               _sqlite3              _statistics        
_struct               _testbuffer           _testcapi          
_testimportmultiple   _testinternalcapi     _testmultiphase    
_tkinter              _uuid                 _xxsubinterpreters 
_xxtestfuzz           array                 audioop            
binascii              cmath                 fcntl              
grp                   math                  mmap               
nis                   parser                pyexpat            
readline              resource              select             
syslog                termios               unicodedata        
xxlimited             zlib
------

The part "-c _struct.c" in the error syndrome stands out to me because there is not a prefix like the other modules that get built successfully before the "build_ext" section. In the successful cases, we see a prefix like "-c ./Modules/faulthandler.c". This can be seen in my attached log.

Also, when I use the Python 3.6.10 source, the build for _struct looks like this and goes fine on Mac 10.15.1. This seems to be the only version of python that works for me on 10.15.1 that I know of.

-------
building '_struct' extension
creating build/temp.macosx-10.15-x86_64-3.6/System
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -I./Include -I. -I/usr/local/include -I/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Include -I/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10 -c /System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules/_struct.c -o build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules/_struct.o
gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules/_struct.o -L/usr/local/lib -o build/lib.macosx-10.15-x86_64-3.6/_struct.cpython-36m-darwin.so
-----

In this successful case, you can see absolute paths are used to compile the .c source, and some temp folders are created for the .o output. I wonder if there is a workaround where we pass a flag to the configure script to produce the same effects? Obviously it would be nice if a plain build worked though.

Also, on Mac 10.14.5, the builds of any python succeeds similarly with no such errors. I am wondering why I am seeing this new pathing and filing behavior on mac 10.15.1 for most versions of Python. Are there any viable workarounds? Thanks for reading!
msg372453 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-27 07:19
To be honest, I've never seen a failure quite like that before so thanks for that! A couple of observations: you appear to be using an old version of Xcode (11.0.0) that may not be fully supported on 10.15.x. So I would try upgrading to the latest released Xcode for 10.15, which is Xcode 11.5 at the moment or make sure your copy of the Command Like Tools is really up to date (you don't need a full-blown Xcode to build python). xcode-select -p should show which you have selected.

You should see one of the following:

$ cc --version

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

or

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

But that may be a secondary matter. The most interesting things in the snippet you display are the paths to the header files. for example:

/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2

which are absolute paths that suggest the build process you are using is somehow using macOS sandboxes. Yet the paths to the actual source file being compiled amd its output file are unprefixed relative paths. So I suggest you figure out why those sandbox paths are in there.  A straightforward build from within the source tarball expanded in a non-sandboxed directory, for example under $HOME or /tmp, should work just fine.

Keep in mind that in any case you will still run into some missing modules like _ssl and _hashlib unless you supply copies of third-party libraries that macOS does not or no longer provides; the Python Developer's Guide has some suggestions (https://devguide.python.org/setup/#macos-and-os-x).
msg372607 - (view) Author: Andrew (andrewfg1992) Date: 2020-06-29 17:48
Thanks for your reply Ned.

I think you are right about mac OS sandboxes messing something up. Python builds most of the extension modules fine if I do a straightforward build in the /tmp directory like you said.

Do you know of any easy ways to disable mac OS sandboxes from being used in the python build from the command line? Alternatively, how do I open the python project in xcode11 GUI? I think I might be able to disable sandboxes from there. If you are not very familiar with mac OS sandboxes like me, feel free to leave that up to me to figure out.

Regarding you question about my command line tools version:

I upgraded the my xcode command line tools installation to 11.4 - I believe this is the latest version that's supported on mac 10.15.1

I still get the same error behavior though when building outside of /tmp.

In this case, here is the compiler that's getting used.

% gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.29)
Target: x86_64-apple-darwin19.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
msg372619 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 19:25
> Do you know of any easy ways to disable mac OS sandboxes from being used in the python build from the command line?

Please show the exact steps you used to build this, in particular, the full ./configure statement and any relevant env variable settings. I don't know how to disable the sandboxes because I've never seen something like this before and I don't know for sure how to reproduce. If I had to take a guess, I suspect it has to do with the location of your source tree which looks like it might be at a unconventional location at the root level: /mathworks/devel.  As you may know, macOS 10.15 Catalina has made a lot of under the cover changes to file system structure including the splitting of / into immutable and mutable subvolumes (/System/Volumes/Data). You may want to try moving /mathworks to another location rather than directly under /, perhaps under /opt or under /Users.
msg372621 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 19:31
PS. Or try doing an out-of-tree build. Perhaps the problem is just due to trying to do the build under that unconventional location under /.  

cd ~/build
./path/to/sourcetree/configure ...
make ...
msg372689 - (view) Author: Andrew (andrewfg1992) Date: 2020-06-30 13:38
The build steps were, unpack source, ./configure, make, using default settings on a company machine.

The configure_and_make_output.txt I attached should show the whole ./configure output. The only difference is that I was using xcode 11.0 for that output- the errors and behavior are identical though.

The error only seems to occur when building to a network location. This is most of our user space because we oftentimes do builds across platforms with the same source. Buliding at locations or out of source builds to /tmp or /Users/Shared works fine though. /mathworks is a network location which was originally mounted at root, but now we have it mounted at /System/Volumes/Data/mathworks. From there, all other network locations are available through other mount points. We also have some related symlinks at /System/Volumes/Data. I think I need to see how the network access move to /Systems/Volumes/Data might have changed and broken things for the python build. Likely, something deep in the python build is not working well with our new network setup. I will see if I can debug deeper into setup.py and PyBuildExt.build_extensions() to see if it's not working right with our network setup and to get you a way to reproduce this behavior. It's hard to see exactly how the paths get passed to xcode and the compiler though. I'll see what I can get you.
msg374898 - (view) Author: Andrew (andrewfg1992) Date: 2020-08-05 19:11
So I believe I've found the problem as to why we can't get the extension modules to build in our network area.

It seems to be a problem with setup.py's find_file() and is_macosx_sdk_path() functions when the extension modules are building. You can find these functions around line 182 of setup.py.

For some background, our network location /mathworks is usually mounted at root, but due to the restrictions on macOS Catalina, it mounts at /System/Volumes/Data and we then add a symlink to the root directory so "/mathworks -> /System/Volumes/Data/mathworks" as to not break old code and scripts. So the real problem is that we cannot find extension module files under /System/Volumes/Data.

This problem occurs because in setup.py, when we attempt to find an extension module file with find_file() wherein the file registers as being in the mac sdk path since the path starts with "/System". This is a problem because find_file() then prepends the file with the sdk path which isn't quite right and fails the existence check. For example a source file that actually exists at "/System/Volumes/Data/mathworks/hub/scratch/aflewell/Python-3.8.2/Modules/" find_file() calls os.path.exists() on "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Volumes/Data/mathworks/hub/scratch/aflewell/Python-3.8.2/Modules/" which fails since the path does not exist at or after the "Volumes" part. This causes find_file to think the file does not exist at all, so the compiler ends up trying to compile the file without an absolute path or relative path to it, which fails.

This can be fixed by changing is_macosx_sdk_path() so it does not consider /System/Volumes/Data as part of the sdk path:

Change from:

def is_macosx_sdk_path(path):
    """
    Returns True if 'path' can be located in an OSX SDK
    """
    return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
                or (path.startswith('/System/') )
                or path.startswith('/Library/') )

to

def is_macosx_sdk_path(path):
    """
    Returns True if 'path' can be located in an OSX SDK
    """
    return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
                or (path.startswith('/System/') and not path.startswith('/System/Volumes/Data'))
                or path.startswith('/Library/') )

This change prevents the sdk path from being prepended to the expected path to the file.

I'm not sure if this is the right fix, so I'd like your opinion on it. I don'r know much about mac's developer sdk and how it works, but it doesn't seem like anything under /System/Volumes should be included in or locatable in the sdk. Anyways, it would be nice to have an official fix for this in the official python source.

Regarding reproduction, you may be able to reproduce by trying to build python under /Systems/Volumes/Data, if not, it may have to do with the mounting of our network filesystem. Thanks for your help.
msg374980 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-08-07 07:25
Thanks for the detailed analysis. That does seem like a reasonable explanation: the code in setup.py was not expecting to find anything other than system files under /System.  For Apple to put other fs mount points in the /System hierarchy seems ... odd but there it is. At first glance, rather than singling out /Systems/Volumes/Data, changing the inclusion test from /System to /System/Library is probably sufficient.

In the most recent SDKs, there is also /System/iOSSupport which is probably never going to come up in Python build contexts but it does point out that making assumptions here about the SDK contents is fragile.  So perhaps an even better solution would be for setup.py to dynamically build a filter based on the contents of the SDK in use, i.e. examine the top level of System and usr in the SDK?

As a reminder when implementing: /usr/local is a bit tricky.  IIRC, recent versions of the compiler chain include /usr/local/include and /usrlocal/bin in the default search paths only in cases where an SDK was not explicitly named, the idea being that you don't want to include references to /usr/local files if you are building something to distribute to the world but it's fine (and necessary) to include /usr/local if you are just building something to run locally on the build system.  I believe /Library works the same way. The tool chains from older version of Xcode and/or CLTs may work differently. So any changes we make here should try to do the right thing on the build tools for all supported versions of macOS.
msg374986 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-08-07 09:40
> As a reminder when implementing: ....

The whole configure machinery needs work :-(.

Disabling looking in /usr/local should IMHO be an explicit choice when building, not something inferred from the presence of "-isysroot" in the compiler flags because that flag can be left out from most builds as Xcode hasn't shipped with multiple SDKs for a long while.
msg374987 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-08-07 09:43
> For some background, our network location /mathworks is usually mounted at root, but due to the restrictions on macOS Catalina, it mounts at /System/Volumes/Data and we then add a symlink to the root directory so "/mathworks -> /System/Volumes/Data/mathworks" as to not break old code and scripts. So the real problem is that we cannot find extension module files under /System/Volumes/Data.

Could you tell more about this? Why is this share mounted under /System/Volumes/Data and not under /Volumes? 

With the direction Apple is moving in I'm a bit surprised that this works at all, the direction seems to be toward a completely read-only tree in /System in /usr (except for /usr/local).
msg374988 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-08-07 10:01
> Why is this share mounted under /System/Volumes/Data and not under /Volumes?

See man 5 synthetic_conf for more details.
msg374990 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-08-07 10:20
That doesn't answer my question, synthetic.conf should still make it possible to have the mountpoint in /, and that's even used as an example in the manpage.
msg375183 - (view) Author: Andrew (andrewfg1992) Date: 2020-08-11 16:13
Hi Ronald. The basic idea is that >= macOS 10.15.1 Apple has made most of the filesystem read only on a folder by folder basis. They separate user-writable onto a separate filesystem, treat it like a disk, and they mount it under /System/Volumes/ as "Data". Likewise, I think the idea is that other volumes get mounted under /System/Volumes, however, since "/System/Volumes/Data" is now "firmlinked" to root, I imagine you can find mounts to the writable filesystem under "/System/Volumes/Data/Volumes" which firmlinks to /Volumes.

About the read-only system volume in macOS Catalina:
https://support.apple.com/en-us/HT210650

More info on the container / firmlink scheme
https://bombich.com/kb/ccc5/working-apfs-volume-groups
History
Date User Action Args
2020-08-11 16:14:00andrewfg1992setmessages: + msg375183
2020-08-07 10:20:46ronaldoussorensetmessages: + msg374990
2020-08-07 10:01:15ned.deilysetmessages: + msg374988
2020-08-07 09:43:14ronaldoussorensetmessages: + msg374987
2020-08-07 09:40:20ronaldoussorensetmessages: + msg374986
2020-08-07 07:32:03ned.deilysettitle: Python extension modules fail to build on Mac 10.15.1 (Catalina) -> setup.py test for macOS SDK files may incorrectly classify files in other file systems
2020-08-07 07:25:11ned.deilysetnosy: + ronaldoussoren
messages: + msg374980

components: + macOS
resolution: works for me ->
stage: needs patch
2020-08-05 19:11:04andrewfg1992setmessages: + msg374898
2020-06-30 13:38:54andrewfg1992setmessages: + msg372689
2020-06-29 19:31:38ned.deilysetmessages: + msg372621
2020-06-29 19:25:59ned.deilysetmessages: + msg372619
2020-06-29 17:48:44andrewfg1992setstatus: pending -> open

messages: + msg372607
2020-06-27 07:19:58ned.deilysetstatus: open -> pending
resolution: works for me
messages: + msg372453
2020-06-26 18:11:08andrewfg1992create