Support macOS 11 and Apple Silicon Macs #85272
Comments
|
macOS 11 is darwin 20.0.0. This confuses the configure script, resulting in defining _POSIX_C_SOURCE and friends. |
ronaldoussoren
added
3.8
ronaldoussoren
added
3.8
|
A first testrun on the arm-mac resulted in test failures in test_distutils (in particular test_deployment_target_default in the distutils test). That's something I have to investigate further. |
|
I've created 3 patches for ctypes
With those, test_ctypes should all pass on arm64 |
|
PR21564 is my attempt to merge all changes needed to support macOS 11 and arm64. To be honest I had lost track of what's needed. With this the patch the tests pass on my DTK system. TODO:
PR 21583 is a separate attempt at doing the code changes that are needed to build with the latest SDK and deploy on macOS 10.9. This patch is still in a fairly rough state, and needs more testing (I haven't done any testing on older platforms yet) TODO:
TDB:
|
|
Hi, Hope this hhelps... Regards, |
|
#22855 is currently the most complete PR, and should be almost ready for merging (although there hasn't been any code review at this point). |
ned-deily
changed the title
|
Are there plans to backport PR 22855 to any branches older than 3.9? |
The plan is to also support 3.8 on Big Sur and Apple Silicon as 3.8 is still in bugfix mode. There are no plans to backport support to 3.7 and 3.6 which are in the security-fix-only phase of their release cycles. |
|
Is there an ETA for having Big Sur support backported to the 3.8 branch, or is it already there and I'm just not seeing it here? |
|
There is no ETA. I'm spending more time away from the computer and on other projects when I'm not at work. I also have no personal need for 3.8 support. If anyone else is up to the job: the back port to 3.9 should be a good starting point, but I don't trust the output of the cherry_picker tool to 3.8. |
|
I ran into a minor issue with the ctypes extension and the _dyld_shared_cache_contains_path dependency that was added as part of this work (commit e8b1c03). When building on Intel macOS 10.15 using an 11.x SDK, the configure check for this symbol passes (#define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH 1) and the extension compiles (albeit with a lot of deprecation warnings spewing from the Apple SDK headers). However, at run-time, we blow up due to a missing symbol error: (This build output is from https://github.com/indygreg/python-build-standalone/runs/1947611547?check_suite_focus=true) The missing symbol is provided by libdyld.dylib, which doesn't appear to be linked directly by CPython's build system. I'm only able to reproduce this on a GitHub Actions 10.5 machine using the 11.1 SDK: it seems to work fine on both an 11.1 Intel and M1 device. I reckon the correct fix is to have _ctypes link libdyld is this build configuration. I was also able to work around it in python-build-standalone by unsetting HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH (hacky solution but this feature isn't needed by the build that was failing). And of course linking a non-Apple libffi should also work, which is potentially why pyenv/Homebrew/etc users aren't noticing. python-build-standalone uses its own Clang built from source, so it is entirely possible there's something differing between the Apple Clang and my Clang causing this to fail. However, at least 1 other person ran into this in the wild (https://stackoverflow.com/questions/65853539/symbol-not-found-dyld-shared-cache-contains-path-on-osx-after-installl-from), so it might be generic. While I'm here, I've taught python-build-standalone to cross-compile CPython 3.9 from both Intel and M1 Apple devices (without using universal binaries). I also have some (hacky) support for targeting iPhone/iWatch/iTV. If there's interest, I could start sending patches upstream. Let me know! |
|
I had some time today, so I took a crack at merging this into Python 3.8. I've filed #69461 with my proposal. To summarize, I'm suggesting back porting all changes from bpo-41100 (as well as the subsequent bpo-42688) but without the weak-linking changes to This should unblock Homebrew (which currently manually patches based on some obsolete PRs) as well as folks who build from source and have a particular need (as we would for example at Dropbox). |
|
Is there any hope of a backport to 3.7? Or has that ship sailed? |
Python 3.7 doesn't accept bugfixes anymore, nor new features, nor new platofrm support (macOS 11). The 3.7 branch now only accept security fixes: |
|
Trying to build from source to Apple M1 Big Sur (macOS 11), and it's failing to compile the standard library extension modules. It seems to be that the .c files are not being located properly, but looking in Modules/ I can see all the required source files are there. attached is a typical build log, have tried a variety of options to no avail. Similar build configurations have worked for previous MacOS releases, which leads me to believe this is something else. I've searched fairly exhaustively for any information related to this and come up with nothing. |
|
@harlantc, in general, please open new issues for new problems. That said, I took a quick look at the build.log you provided and it was not obvious exactly why things went wrong. The most obvious difference is that when I do a 3.9.4 build, the extension module source paths (the -c arguments to the failing compiles) have been normalized by setup.py to absolute paths, whereas in the failing build.log they are unprefixed relative paths. I suggest you look at the top level setup.py file to figure out why that is the case in your build. I also note that the build appears to be on a very early release of macOS 11 Big Sur; it would be best to upgrade to the latest version of it and ensure you have the latest version of the Command Line Tools installed. Also, your configure arguments appear to be unnecessary complex: the -m64 and the --build and --target arguments shouldn't be necessary and we rarely test with --enable-shared on macOS. It's possible they are contributing factors. If you do find something that appears to be a Python build issue, please open a new issue with the details. |
|
@ned.deily sorry about that, will do that in the future. Thanks for replying. in regard to the extra args, those were all added in an attempt to make things work, I should probably have shown an earlier log that excluded them. I wasn't sure setup.py would be the right place to look for more, I'll take a look there. Thanks |
|
No problem. Also you should not need to specify sysroot if you are; the Apple compiler front-end takes care of that for you (see man xcrun for more details). |
|
Thanks, Max, for doing the backport to 3.8 hard work! While PR 25274 covered most of the changes needed, there were some additional fixes and changes needed and I felt it was easier to cherry-pick all of the changes into a new PR. I have done extensive building and testing on Big Sur on both Apple Silicon and Intel Macs and regression building and testing on some older macOS versions including macOS 10.9 and python.org macOS installers. So far, it looks good to me. Assuming the 3.8 release manager has no objections, I will merge this in time for 3.8.10 and we can finally close this issue. |
|
Thank you to everyone who contributed to this major undertaking! A particular thank you to Lawrence for doing much of the initial work and paving the way. Now that 3.8 also supports Big Sur and Apple Silicon Macs as of the imminent 3.8.10 release, it's time to close this issue. If new concerns arise, pleasa open or use other issues. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: