classification
Title: Add distutils mvsccompiler support for Windows ARM64 build
Type: compile error Stage: resolved
Components: Windows Versions: Python 3.10
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: avladu, jaraco, miss-islington, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-11-19 12:04 by avladu, last changed 2021-03-04 18:41 by jaraco. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23399 merged avladu, 2020-11-19 12:13
PR 24753 merged jaraco, 2021-03-04 18:06
Messages (12)
msg381407 - (view) Author: Adrian Vladu (avladu) * Date: 2020-11-19 12:04
To add support for building packages that have C extensions on Windows ARM64, some fixes are required in the integrated distutils wrapper for Visual Studio compiler (Lib/distutils/msvc9compiler.py)

This is a hardcoded fix that needs to be improved so that it applies only on windows arm64:
https://github.com/ader1990/cpython/commit/b8c59c9b96a7ad11094224b5631aae3b89323f7a

Any suggestions are welcome.
msg381447 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-19 17:41
Firstly, msvc9compiler.py is very deprecated, we don't touch that one at all or use it in distutils.

Second, _msvccompiler.py is weakly deprecated (and likely soon fully deprecated), so you'd be better to make the fix in the setuptools project instead.

But we'd very much like to see that fix made. So if you're prepared to work on it, that'd be great! I've added Jason to this issue so he can point you at the right place in setuptools to look.
msg381457 - (view) Author: Adrian Vladu (avladu) * Date: 2020-11-19 21:45
This fix is __required__ to build a lot of important packages in the python ecosystem, like numpy, pandas, pywin32 and probably a lot more, as most of these important packages have not migrated to setuptools and usually maintain support for multiple python versions.

I know that there is a way to change all the packages to use the canonical approach to compile things, but most of them have a tweaked compiler on top of the msvc9compiler.py compiler. I do not see a change in all the python packages to happen as fast as a small backwards compatible commit in the main cpython code. I see this commit as a fix for the win-arm64 package build and not an "extra" feature.

As I can see it, not adding this fix here will require a split of trees that are going to be used just for this reason.

I will add the fix to setuptools too, but that will not change the above (maintaining separate branches for people who want to use those packages).
msg381461 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-20 01:27
There is no ARM64 release of Python for Windows right now, so unfortunately this doesn't constitute any kind of bugfix. We don't have a buildbot, and nobody out there has CI, so it's not a supported platform.

I get that it's easier to centrally fix all the third-party packages by changing the core library, but we've been saying for nearly 10 years that distutils shouldn't be used and shouldn't be assumed to be up to date. Anyone building on msvc9compiler.py has been broken since Python 3.5, because they've been using the wrong compiler.

A change to _msvccompiler.py for 3.10 can be considered, though you will need to get it ready before PEP 632 is accepted, at which point we close all distutils-related issues and only reopen for critical bugs.
msg381468 - (view) Author: Adrian Vladu (avladu) * Date: 2020-11-20 08:49
Thank you for the suggestion, I will update the PR accordingly to change the __msvccompiler.py. I just need to find a good candidate that uses that implementation to check if the compilation gets fixed.
msg381487 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-11-20 15:59
If you wish for the functionality to be available in setuptools (backported), please contribute the change at https://github.com/pypa/distutils. At some point, contributions to CPython will also be synced there as well, and any changes there get synced into Setuptools. Unfortunately, at this stage, that functionality is only exposed through a feature flag when `SETUPTOOLS_USE_DISTUTILS=local` is set... until downstream implementations in Linux distributions and monkeypatches such as with Numpy can be updated not to rely on the implementation details as found in the stdlib.
msg381600 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-11-22 08:06
As I reviewed the PR, I do realize how tricky this change is going to be. In addition to the three MSVC implementations in distutils, Setuptools has its own (https://github.com/pypa/setuptools/blob/master/setuptools/msvc.py).

Interestingly, that file indicates support for ARM (when describing support for MSVC 14+), so at least someone has previously embarked on this territory.

If it turns out that supporting ARM in msvc9compiler also implies adding support for MSVC 14+, I think you'll be hard-pressed to find a satisfying solution that doesn't break compatibility for many use-cases.

Thanks for investigating this issue and good luck.
msg387355 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-19 21:00
Closed due to PEP 632 (distutils is now deprecated).

Changes to support this should go into setuptools or another build backend. I'm already trying to get some resources together to help these other projects build/test on ARM64, so that should help. If someone wants to make a start on developing the cross-compilation support (though I'm not 100% sure we'll need it...), that would be great.
msg388105 - (view) Author: miss-islington (miss-islington) Date: 2021-03-04 16:59
New changeset cb7bc7640935f6b05e9d2acfe4b33d496e8f8666 by Adrian Vladu in branch 'master':
bpo-42405: fix C extensions build on Windows ARM64 (GH-23399)
https://github.com/python/cpython/commit/cb7bc7640935f6b05e9d2acfe4b33d496e8f8666
msg388112 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-03-04 18:00
Jason, this issue and the associated PR are out of date and should not be merged.

The PR also only affects completely unused code, so it has no value whatsoever being in CPython. I don't really care too much about the unused code, but I will *insist* on the NEWS entry being reverted, to avoid causing confusion.

(As an aside, I'd love setuptools to gain this functionality, and I'm trying to get access to hardware for you so you can validate it in CI.)
msg388114 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2021-03-04 18:04
Hey Steve. Thanks for the clarification. I'd forgotten the context, specifically that any patches needed to be merged before the PEP was accepted, and was working from a memory that we were willing to accept this change. Happy to revert it, as that also saves me the time of making sure the change is backported to pypa/distutils. Adrian, please submit the change to pypa/distutils.
msg388120 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2021-03-04 18:41
New changeset fbf75b9997e280b1220755d0a17dbed71240d42e by Jason R. Coombs in branch 'master':
Revert "bpo-42405: fix C extensions build on Windows ARM64 (GH-23399)" (#24753)
https://github.com/python/cpython/commit/fbf75b9997e280b1220755d0a17dbed71240d42e
History
Date User Action Args
2021-03-04 18:41:58jaracosetmessages: + msg388120
2021-03-04 18:06:21jaracosetpull_requests: + pull_request23524
2021-03-04 18:04:16jaracosetmessages: + msg388114
2021-03-04 18:00:04steve.dowersetmessages: + msg388112
2021-03-04 16:59:16miss-islingtonsetnosy: + miss-islington
messages: + msg388105
2021-02-19 21:00:14steve.dowersetstatus: open -> closed
resolution: out of date
messages: + msg387355

stage: patch review -> resolved
2020-11-22 08:06:01jaracosetmessages: + msg381600
2020-11-20 15:59:20jaracosetmessages: + msg381487
2020-11-20 08:49:32avladusetmessages: + msg381468
2020-11-20 01:27:51steve.dowersetmessages: + msg381461
2020-11-19 21:45:57avladusetmessages: + msg381457
2020-11-19 17:41:26steve.dowersetnosy: + jaraco
messages: + msg381447
2020-11-19 12:13:31avladusetkeywords: + patch
stage: patch review
pull_requests: + pull_request22291
2020-11-19 12:04:38avladucreate