classification
Title: Python extension modules fail to build on Mac 10.15.1 (Catalina)
Type: compile error Stage:
Components: Extension Modules Versions: Python 3.8
process
Status: open Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: andrewfg1992, ned.deily
Priority: normal Keywords:

Created on 2020-06-26 18:11 by andrewfg1992, last changed 2020-06-30 13:38 by andrewfg1992.

Files
File name Uploaded Description Edit
configure_and_make_output.txt andrewfg1992, 2020-06-26 18:11 Output of the ./configure and make commands
Messages (6)
msg372437 - (view) Author: Andrew (andrewfg1992) Date: 2020-06-26 18:11
Quick repoduction steps:
1) Log into a mac with macOS version 10.15.1 (10.15.x may work)
2) For build tools, use Xcode11. A minimal xcode command-tools installation also reproduced for me.
3) Download and decompress the latest python 3.8.2 source
4) run "./configure" in the top-level source folder
5) run "make" in the top-level source folder

I believe this error may also occur when using python 2.7.x, 3.7.x, and others. Curiously, the errors do not occur when I use the latest 3.6.10 source.

A text file containing the output of "./configure" and "make" is attached.

Main Description:
On Mac 10.15.1 with Xcode11 I encounter compilation errors when building the latest Python 3.8.2 source. When the build reaches the "build_ext" section in setup.py, none of the extension modules build, with each compilation attempt producing an error. For example, when the _struct extension module attempts compilation I get:

--- Error Syndrome ---
building '_struct' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include/internal -I./Include -I. -I/usr/local/include -I/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2/Include -I/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2 -c _struct.c -o build/temp.macosx-10.15-x86_64-3.8/_struct.o
clang: error: no such file or directory: '_struct.c'
clang: error: no input files
------

After many similar errors, we get to the main build report where we see the failed extension modules:

--- Snippet of build report ---
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 _hashlib              _ssl               
ossaudiodev           spwd                                     
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           


Failed to build these modules:
_asyncio              _bisect               _blake2            
_bz2                  _codecs_cn            _codecs_hk         
_codecs_iso2022       _codecs_jp            _codecs_kr         
_codecs_tw            _contextvars          _crypt             
_csv                  _ctypes               _ctypes_test       
_curses               _curses_panel         _datetime          
_dbm                  _decimal              _elementtree       
_heapq                _json                 _lsprof            
_lzma                 _md5                  _multibytecodec    
_multiprocessing      _opcode               _pickle            
_posixshmem           _posixsubprocess      _queue             
_random               _scproxy              _sha1              
_sha256               _sha3                 _sha512            
_socket               _sqlite3              _statistics        
_struct               _testbuffer           _testcapi          
_testimportmultiple   _testinternalcapi     _testmultiphase    
_tkinter              _uuid                 _xxsubinterpreters 
_xxtestfuzz           array                 audioop            
binascii              cmath                 fcntl              
grp                   math                  mmap               
nis                   parser                pyexpat            
readline              resource              select             
syslog                termios               unicodedata        
xxlimited             zlib
------

The part "-c _struct.c" in the error syndrome stands out to me because there is not a prefix like the other modules that get built successfully before the "build_ext" section. In the successful cases, we see a prefix like "-c ./Modules/faulthandler.c". This can be seen in my attached log.

Also, when I use the Python 3.6.10 source, the build for _struct looks like this and goes fine on Mac 10.15.1. This seems to be the only version of python that works for me on 10.15.1 that I know of.

-------
building '_struct' extension
creating build/temp.macosx-10.15-x86_64-3.6/System
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -I./Include -I. -I/usr/local/include -I/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Include -I/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10 -c /System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules/_struct.c -o build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules/_struct.o
gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username/python36_source/Python-3.6.10/Modules/_struct.o -L/usr/local/lib -o build/lib.macosx-10.15-x86_64-3.6/_struct.cpython-36m-darwin.so
-----

In this successful case, you can see absolute paths are used to compile the .c source, and some temp folders are created for the .o output. I wonder if there is a workaround where we pass a flag to the configure script to produce the same effects? Obviously it would be nice if a plain build worked though.

Also, on Mac 10.14.5, the builds of any python succeeds similarly with no such errors. I am wondering why I am seeing this new pathing and filing behavior on mac 10.15.1 for most versions of Python. Are there any viable workarounds? Thanks for reading!
msg372453 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-27 07:19
To be honest, I've never seen a failure quite like that before so thanks for that! A couple of observations: you appear to be using an old version of Xcode (11.0.0) that may not be fully supported on 10.15.x. So I would try upgrading to the latest released Xcode for 10.15, which is Xcode 11.5 at the moment or make sure your copy of the Command Like Tools is really up to date (you don't need a full-blown Xcode to build python). xcode-select -p should show which you have selected.

You should see one of the following:

$ cc --version

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

or

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

But that may be a secondary matter. The most interesting things in the snippet you display are the paths to the header files. for example:

/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2

which are absolute paths that suggest the build process you are using is somehow using macOS sandboxes. Yet the paths to the actual source file being compiled amd its output file are unprefixed relative paths. So I suggest you figure out why those sandbox paths are in there.  A straightforward build from within the source tarball expanded in a non-sandboxed directory, for example under $HOME or /tmp, should work just fine.

Keep in mind that in any case you will still run into some missing modules like _ssl and _hashlib unless you supply copies of third-party libraries that macOS does not or no longer provides; the Python Developer's Guide has some suggestions (https://devguide.python.org/setup/#macos-and-os-x).
msg372607 - (view) Author: Andrew (andrewfg1992) Date: 2020-06-29 17:48
Thanks for your reply Ned.

I think you are right about mac OS sandboxes messing something up. Python builds most of the extension modules fine if I do a straightforward build in the /tmp directory like you said.

Do you know of any easy ways to disable mac OS sandboxes from being used in the python build from the command line? Alternatively, how do I open the python project in xcode11 GUI? I think I might be able to disable sandboxes from there. If you are not very familiar with mac OS sandboxes like me, feel free to leave that up to me to figure out.

Regarding you question about my command line tools version:

I upgraded the my xcode command line tools installation to 11.4 - I believe this is the latest version that's supported on mac 10.15.1

I still get the same error behavior though when building outside of /tmp.

In this case, here is the compiler that's getting used.

% gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.29)
Target: x86_64-apple-darwin19.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
msg372619 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 19:25
> Do you know of any easy ways to disable mac OS sandboxes from being used in the python build from the command line?

Please show the exact steps you used to build this, in particular, the full ./configure statement and any relevant env variable settings. I don't know how to disable the sandboxes because I've never seen something like this before and I don't know for sure how to reproduce. If I had to take a guess, I suspect it has to do with the location of your source tree which looks like it might be at a unconventional location at the root level: /mathworks/devel.  As you may know, macOS 10.15 Catalina has made a lot of under the cover changes to file system structure including the splitting of / into immutable and mutable subvolumes (/System/Volumes/Data). You may want to try moving /mathworks to another location rather than directly under /, perhaps under /opt or under /Users.
msg372621 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 19:31
PS. Or try doing an out-of-tree build. Perhaps the problem is just due to trying to do the build under that unconventional location under /.  

cd ~/build
./path/to/sourcetree/configure ...
make ...
msg372689 - (view) Author: Andrew (andrewfg1992) Date: 2020-06-30 13:38
The build steps were, unpack source, ./configure, make, using default settings on a company machine.

The configure_and_make_output.txt I attached should show the whole ./configure output. The only difference is that I was using xcode 11.0 for that output- the errors and behavior are identical though.

The error only seems to occur when building to a network location. This is most of our user space because we oftentimes do builds across platforms with the same source. Buliding at locations or out of source builds to /tmp or /Users/Shared works fine though. /mathworks is a network location which was originally mounted at root, but now we have it mounted at /System/Volumes/Data/mathworks. From there, all other network locations are available through other mount points. We also have some related symlinks at /System/Volumes/Data. I think I need to see how the network access move to /Systems/Volumes/Data might have changed and broken things for the python build. Likely, something deep in the python build is not working well with our new network setup. I will see if I can debug deeper into setup.py and PyBuildExt.build_extensions() to see if it's not working right with our network setup and to get you a way to reproduce this behavior. It's hard to see exactly how the paths get passed to xcode and the compiler though. I'll see what I can get you.
History
Date User Action Args
2020-06-30 13:38:54andrewfg1992setmessages: + msg372689
2020-06-29 19:31:38ned.deilysetmessages: + msg372621
2020-06-29 19:25:59ned.deilysetmessages: + msg372619
2020-06-29 17:48:44andrewfg1992setstatus: pending -> open

messages: + msg372607
2020-06-27 07:19:58ned.deilysetstatus: open -> pending
resolution: works for me
messages: + msg372453
2020-06-26 18:11:08andrewfg1992create