classification
Title: Add distutils mvsccompiler support for Windows ARM64 build
Type: compile error Stage: patch review
Components: Windows Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: avladu, jaraco, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-11-19 12:04 by avladu, last changed 2020-11-22 08:06 by jaraco.

Pull Requests
URL Status Linked Edit
PR 23399 open avladu, 2020-11-19 12:13
Messages (7)
msg381407 - (view) Author: Adrian Vladu (avladu) * Date: 2020-11-19 12:04
To add support for building packages that have C extensions on Windows ARM64, some fixes are required in the integrated distutils wrapper for Visual Studio compiler (Lib/distutils/msvc9compiler.py)

This is a hardcoded fix that needs to be improved so that it applies only on windows arm64:
https://github.com/ader1990/cpython/commit/b8c59c9b96a7ad11094224b5631aae3b89323f7a

Any suggestions are welcome.
msg381447 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-19 17:41
Firstly, msvc9compiler.py is very deprecated, we don't touch that one at all or use it in distutils.

Second, _msvccompiler.py is weakly deprecated (and likely soon fully deprecated), so you'd be better to make the fix in the setuptools project instead.

But we'd very much like to see that fix made. So if you're prepared to work on it, that'd be great! I've added Jason to this issue so he can point you at the right place in setuptools to look.
msg381457 - (view) Author: Adrian Vladu (avladu) * Date: 2020-11-19 21:45
This fix is __required__ to build a lot of important packages in the python ecosystem, like numpy, pandas, pywin32 and probably a lot more, as most of these important packages have not migrated to setuptools and usually maintain support for multiple python versions.

I know that there is a way to change all the packages to use the canonical approach to compile things, but most of them have a tweaked compiler on top of the msvc9compiler.py compiler. I do not see a change in all the python packages to happen as fast as a small backwards compatible commit in the main cpython code. I see this commit as a fix for the win-arm64 package build and not an "extra" feature.

As I can see it, not adding this fix here will require a split of trees that are going to be used just for this reason.

I will add the fix to setuptools too, but that will not change the above (maintaining separate branches for people who want to use those packages).
msg381461 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-11-20 01:27
There is no ARM64 release of Python for Windows right now, so unfortunately this doesn't constitute any kind of bugfix. We don't have a buildbot, and nobody out there has CI, so it's not a supported platform.

I get that it's easier to centrally fix all the third-party packages by changing the core library, but we've been saying for nearly 10 years that distutils shouldn't be used and shouldn't be assumed to be up to date. Anyone building on msvc9compiler.py has been broken since Python 3.5, because they've been using the wrong compiler.

A change to _msvccompiler.py for 3.10 can be considered, though you will need to get it ready before PEP 632 is accepted, at which point we close all distutils-related issues and only reopen for critical bugs.
msg381468 - (view) Author: Adrian Vladu (avladu) * Date: 2020-11-20 08:49
Thank you for the suggestion, I will update the PR accordingly to change the __msvccompiler.py. I just need to find a good candidate that uses that implementation to check if the compilation gets fixed.
msg381487 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-11-20 15:59
If you wish for the functionality to be available in setuptools (backported), please contribute the change at https://github.com/pypa/distutils. At some point, contributions to CPython will also be synced there as well, and any changes there get synced into Setuptools. Unfortunately, at this stage, that functionality is only exposed through a feature flag when `SETUPTOOLS_USE_DISTUTILS=local` is set... until downstream implementations in Linux distributions and monkeypatches such as with Numpy can be updated not to rely on the implementation details as found in the stdlib.
msg381600 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-11-22 08:06
As I reviewed the PR, I do realize how tricky this change is going to be. In addition to the three MSVC implementations in distutils, Setuptools has its own (https://github.com/pypa/setuptools/blob/master/setuptools/msvc.py).

Interestingly, that file indicates support for ARM (when describing support for MSVC 14+), so at least someone has previously embarked on this territory.

If it turns out that supporting ARM in msvc9compiler also implies adding support for MSVC 14+, I think you'll be hard-pressed to find a satisfying solution that doesn't break compatibility for many use-cases.

Thanks for investigating this issue and good luck.
History
Date User Action Args
2020-11-22 08:06:01jaracosetmessages: + msg381600
2020-11-20 15:59:20jaracosetmessages: + msg381487
2020-11-20 08:49:32avladusetmessages: + msg381468
2020-11-20 01:27:51steve.dowersetmessages: + msg381461
2020-11-19 21:45:57avladusetmessages: + msg381457
2020-11-19 17:41:26steve.dowersetnosy: + jaraco
messages: + msg381447
2020-11-19 12:13:31avladusetkeywords: + patch
stage: patch review
pull_requests: + pull_request22291
2020-11-19 12:04:38avladucreate