classification
Title: Provide "python3.x-intel64" command with macOS universal2 framework builds
Type: Stage: resolved
Components: macOS Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: miss-islington, ned.deily, ronaldoussoren
Priority: normal Keywords: patch

Created on 2021-05-02 07:23 by ned.deily, last changed 2021-05-02 10:56 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25804 merged ned.deily, 2021-05-02 07:50
PR 25807 merged miss-islington, 2021-05-02 08:48
PR 25810 merged ned.deily, 2021-05-02 09:52
Messages (4)
msg392669 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-05-02 07:23
On certain macOS configurations, it is possible to have a universal build (one that contains executables and libraries with binaries for more than one CPU architecture combined into "fat" files) where running with a choice of archs. Examples in the past included Intel-64 (x86_64) vs Intel-32 (i386) vs (in macOS 10.5 and 10.6) ppc (using Rosetta emulation. While it is possible to use the macOS arch utility to select a non-default arch for execution:
   arch -i386 /usr/local/bin/python3
this method does not handle the common case of a Python program spawning another Python subprocess (to run a test, say) by calling the subprocess module to execute the interpreter binary recorded in sys.executable, which only contains the interpreter executable file name, not the arch command.  Thus the subprocess attempts to launch in the default arch for the configuration rather than the overriden arch, which may cause program failures or incorrect results.  To get around this where appropriate, framework universal builds on macOS provided an additional 32-bit-only python3.x executable file, python3.x-32, that solves this problem: by invoking Python with python3.x-32, the main interpreter process runs in 32-bit mode and any spawned subprocesses using sys.executable will also run in 32-bit mode.

With the introduction of Apple Silicon Macs and their support for running x86_64 Intel-64 binaries with Rosetta 2 emulation, the need to be able to force Python to run a non-default arch reliably is again important for the transition period while third-party packages are being ported or tested on the new arm64-based Macs. For example, there are some popular packages on PyPI that do not yet provide universal2 or just arm64 wheels but, by forcing Python to run in Intel mode, existing wheels can be tested and used.

To that end, the PR for this issue adds a "python3.x-intel64" and "python3-intel64" executable or symlink, as needed, when installing a macOS "universal2" framework build, similar to the "python3.x-32" and "python3-32" links installed for macOS "intel" framework builds.

An example:

$ sw_vers 
ProductName:	macOS
ProductVersion:	11.3
BuildVersion:	20E232
$ uname -a
Darwin pyb20 20.4.0 Darwin Kernel Version 20.4.0: Fri Mar  5 01:14:02 PST 2021; root:xnu-7195.101.1~3/RELEASE_ARM64_T8101 arm64
$ python3.8 -m pip install --only-binary ':all:' numpy
ERROR: Could not find a version that satisfies the requirement numpy (from versions: none)
ERROR: No matching distribution found for numpy
$ python3.8-intel64 -m pip install --only-binary ':all:' numpy 
Collecting numpy
  Using cached numpy-1.20.2-cp38-cp38-macosx_10_9_x86_64.whl (16.0 MB)
Installing collected packages: numpy
Successfully installed numpy-1.20.2

Of course, for this to be useful assumes that running under Rosetta 2 emulation provides the correct results. Testing is advised!
msg392676 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-05-02 08:48
New changeset 0cb33da1cc9cebb9b2d67d446feb1cfd36fe7f55 by Ned Deily in branch 'master':
bpo-44009: Provide "python3.x-intel64" for Apple Silicon Macs (GH-25804)
https://github.com/python/cpython/commit/0cb33da1cc9cebb9b2d67d446feb1cfd36fe7f55
msg392677 - (view) Author: miss-islington (miss-islington) Date: 2021-05-02 09:10
New changeset de0e3f8c16c13221dbca66b9496a302fd5749c37 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-44009: Provide "python3.x-intel64" for Apple Silicon Macs (GH-25804) (GH-25807)
https://github.com/python/cpython/commit/de0e3f8c16c13221dbca66b9496a302fd5749c37
msg392687 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-05-02 10:55
New changeset 78e55888ab379cc61c0b5db67f0b881e526628d2 by Ned Deily in branch '3.8':
bpo-44009: Provide "python3.x-intel64" for Apple Silicon Macs (GH-25810)
https://github.com/python/cpython/commit/78e55888ab379cc61c0b5db67f0b881e526628d2
History
Date User Action Args
2021-05-02 10:56:39ned.deilysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-05-02 10:55:07ned.deilysetmessages: + msg392687
2021-05-02 09:52:19ned.deilysetpull_requests: + pull_request24497
2021-05-02 09:10:24miss-islingtonsetmessages: + msg392677
2021-05-02 08:48:54ned.deilysetmessages: + msg392676
2021-05-02 08:48:39miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24494
2021-05-02 07:50:00ned.deilysetkeywords: + patch
stage: patch review
pull_requests: + pull_request24489
2021-05-02 07:23:23ned.deilycreate