classification
Title: python 3.9.1 failed to create .so files as universal2 (arm64, x86_64) on Mac
Type: compile error Stage: patch review
Components: macOS Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, ronaldoussoren, xz_sophos
Priority: normal Keywords: patch

Created on 2020-12-11 10:34 by xz_sophos, last changed 2021-01-10 10:34 by ronaldoussoren.

Pull Requests
URL Status Linked Edit
PR 24187 open ronaldoussoren, 2021-01-10 10:07
Messages (4)
msg382861 - (view) Author: (xz_sophos) Date: 2020-12-11 10:34
With 3.9.1 source code, I am trying to make univerisal2 build of python.framework from the source code. The problem I am seeing is the main python binary is universal, but all the .so files in lib-dynload (Python.framework/Versions/3.9/lib/python3.9/lib-dynload/) are only x86_64.

The Mac is MacOS 10.15, with Xcode 12.2 installed. Xcode12.2 has SDKs to support universal build for both arm64 and x86_64 architecture. I have no problems making other universal applications and frameworks on the machine. 

Following the documentation. I ran the following build command:
./configure --enable-universalsdk --enable-framework=./tmp --with-universal-archs=universal2 --without-ensurepip
make
make install

The resulting python was universal but the .so files are not.
msg382862 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-12-11 11:38
Currently macOS 11 is needed to build a universal2 framework. That because we haven't spent time yet on detecting if the current compiler can compile for arm64.

An easy workaround: Patch _osx_support.py, in particular the function "_supports_arm64_builds". This checks that the macOS version is 11.0 or later.



A PR that detects if Xcode 12 or the corresponding command line tools are used  would be welcome :-)
msg384757 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-01-10 10:10
I've added a (draft) PR for this. The PR is a this time not tested on a 10.15 system.

Another change that might be needed: ensure that sysconfig.get_platform() does not return "universal2" as the binary architecture when the compiler does not support arm64.
msg384758 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-01-10 10:34
> Another change that might be needed: ensure that sysconfig.get_platform() 
> does not return "universal2" as the binary architecture when the compiler 
> does not support arm64.

Hmmm... _osx_support.get_platform_osx() contains the following comment:

        # Use the original CFLAGS value, if available, so that we
        # return the same machine type for the platform string.
        # Otherwise, distutils may consider this a cross-compiling
        # case and disallow installs.

That indicates that fixing the value of sysconfig.get_platform() when using a "universal2" python on a system that doesn't support arm64 might break distils. 

I haven't done further research on this. In general I'd prefer to create the correct platform tag, but not if that results in not being able to install extensions.
History
Date User Action Args
2021-01-10 10:34:42ronaldoussorensetmessages: + msg384758
versions: + Python 3.10
2021-01-10 10:10:17ronaldoussorensetmessages: + msg384757
2021-01-10 10:07:19ronaldoussorensetkeywords: + patch
stage: patch review
pull_requests: + pull_request23015
2020-12-11 11:38:37ronaldoussorensetmessages: + msg382862
2020-12-11 10:34:20xz_sophoscreate