classification
Title: pip install selects 32 bit wheels for 64 bit python if vcvarsall.bat amd64_x86 in environment
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.8
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: Ryan Thornton, altendky, dstufft, eric.araujo, puetzk, steve.dower, vinay.sajip
Priority: normal Keywords: 3.5regression

Created on 2019-12-06 18:49 by Ryan Thornton, last changed 2020-09-29 11:13 by vinay.sajip. This issue is now closed.

Messages (6)
msg357936 - (view) Author: Ryan Thornton (Ryan Thornton) Date: 2019-12-06 18:49
## Expected Behavior

pip install should download dependencies matching the architecture of the python executable being used.

## Actual Behavior

When calling pip install from a Visual Studio command prompt configured to cross compile from x64 to x86, pip installs wheels matching the architecture of Visual Studio's cross compile target (i.e. `VSCMD_ARG_TGT_ARCH=x86`) and not the architecture of python itself (x64).

This results in a broken installation of core libraries.

## Steps to Reproduce

System Details:
Windows 10 x64
Python 3.8 x64
Visual Studio 2017 15.9.14

Environment Details:
vcvarsall.bat amd64_x86

1. "C:\Program Files\Python38\python.exe" -mvenv "test"
2. cd test\Scripts
3. pip install cffi==1.13.2

Results in the following:

> Collecting cffi
>  Using cached https://files.pythonhosted.org/packages/f8/26/5da5cafef77586e4f7a136b8a24bc81fd2cf1ecb71b6ec3998ffe78ea2cf/cffi-1.13.2-cp38-cp38-win32.whl

## Context

I think the regression was introduced here:
62dfd7d6fe11bfa0cd1d7376382c8e7b1275e38c

https://github.com/python/cpython/commit/62dfd7d6fe11bfa0cd1d7376382c8e7b1275e38c
msg357945 - (view) Author: Kevin Puetz (puetzk) Date: 2019-12-06 21:31
Besides the fact the MSVC's target platform isn't really related to the architecture for dependencies being installed, I'm not sure VSCMD_ARG_TGT_ARCH is an appropriate variable to look at in the first place. It doesn't seem to be at all documented, and (from looking at the implemenation) it seems to be meant as how parse_cmd.bat communicates with vcvars.bat, dotnet.bat, winsdk.bat, etc (these are all sub-programms that make up vcvarsall.bat)

I think %PLATFORM% (which generally ends up set equal to %VSCMD_ARG_TGT_ARCH% once vsdevcmd\ext\vcvars.bat has done its thing) is might be a better variable that cross-compiling should actually look at when it wants to know what architecture MSVC will compile to.
msg357946 - (view) Author: Kevin Puetz (puetzk) Date: 2019-12-06 21:42
Just to link the various pieces together: I think 

https://bitbucket.org/pypa/distlib/src/2d145da7cb42590039fbd56a9ab764d5d4716a98/distlib/wheel.py#lines-53

is the place in distlib that's being breaking pip due to redefining distutils.util.get_platform() as the cross-compile target architecture  rather than the build architecture.

https://docs.python.org/3/distutils/apiref.html doesn't show it as a documented public function, but the commit Ryan linked did introduce a new get_host_plaform() that matches the old behavior; if that became public and distlib used itinstead (conditional on the python version?) that would also be a way to fix the symptom Ryan noted (pip now choosing the wrong wheel to install).

I don't know how the pieces here fit together well enough to now which side should change.

@zooba: adding you to the Cc since it was your comments at https://github.com/python/cpython/pull/11774#discussion_r254461961 that seem to have led to the decision to re-define get_platform, maybe you have some relevant context?
msg358129 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-12-09 19:01
> maybe you have some relevant context

Nothing you haven't figured out yourself. As you say, they're both private functions, and as we investigated adding a "get_target_platform" function, it turned out that nearly all users were expecting the target platform, so rather than updating everything we added one for the less-common case.

get_host_platform was not defined before, so I'd say use it if it's there and use get_platform if it's not.

I doubt we're going to add any new public/supported APIs to distutils ever again, to be honest.
msg377627 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-09-28 20:54
I'm closing this as external, as the canonical source for platform tags is now the packaging library.

Either pip and/or distlib should switch to using that library or the logic included (which I'll note is considerably more complex - but also more well defined and tested - than what is in distutils).
msg377662 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-09-29 11:13
FYI I'm in the process of updating distlib to add the get_platform() / get_host_platform() distinction. The next release should have it.
History
Date User Action Args
2020-09-29 11:13:15vinay.sajipsetkeywords: + 3.5regression

messages: + msg377662
2020-09-28 20:54:35steve.dowersetstatus: open -> closed
resolution: third party
messages: + msg377627

stage: resolved
2020-09-24 03:12:12altendkysetnosy: + altendky
2019-12-09 19:01:12steve.dowersetmessages: + msg358129
2019-12-06 23:10:31puetzksetnosy: + vinay.sajip
2019-12-06 21:42:19puetzksetnosy: + steve.dower
messages: + msg357946
2019-12-06 21:31:20puetzksetnosy: + puetzk
messages: + msg357945
2019-12-06 18:49:45Ryan Thorntoncreate