classification
Title: distutils cannot find Build Tools 2017 since 3.7.2
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, miss-islington, ned.deily, pablogsal, schlamar, steve.dower, uranusjr
Priority: normal Keywords: patch, patch, patch

Created on 2019-01-09 12:53 by schlamar, last changed 2019-01-21 21:58 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11495 merged schlamar, 2019-01-10 07:25
PR 11495 merged schlamar, 2019-01-10 07:25
PR 11495 merged schlamar, 2019-01-10 07:26
PR 11626 merged miss-islington, 2019-01-20 18:48
PR 11626 merged miss-islington, 2019-01-20 18:48
PR 11626 merged miss-islington, 2019-01-20 18:48
Messages (10)
msg333312 - (view) Author: Marc Schlaich (schlamar) * Date: 2019-01-09 12:53
vshwere.exe doesn't return Build Tools 2017 per default. This means Build Tools 2017 are not detected by distutils in 3.7.2 and you get the famous "Microsoft Visual C++ 14.0 is required" error.

Please see https://github.com/Microsoft/vswhere/issues/125 for more details.

The solution is to add 

    "-products", "*",

to the vswhere.exe call.

This is a regression of https://bugs.python.org/issue35067.
msg333351 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-09 22:43
Excellent catch! Are you interested in creating a PR?

Also, I don't recall whether we are handling multiple installs or not, but if we are not I would rather explicitly list the products that may include compilers than using "*" (since there are and will be more products that do not). If we're checking for installed packages as well and iterating over all results then "*" will be fine.
msg333365 - (view) Author: Marc Schlaich (schlamar) * Date: 2019-01-10 07:18
We are passing arguments

 -latest        (Return only the newest version and last installed.)

and

 "-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",

So this should handle the cases multiple installs and different products without compilers.

I try to get a PR ready.
msg334079 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-20 07:42
Reminder to myself (or permission for anyone else) to merge this. My phone won't let me actually click the github merge button, unfortunately, but this seems good to go.
msg334097 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-01-20 18:47
New changeset b2dc4a3313c236fedbd6df664722cd47f3d91a72 by Ned Deily (Marc Schlaich) in branch 'master':
bpo-35699: fix distuils cannot detect Build Tools 2017 anymore (GH-11495)
https://github.com/python/cpython/commit/b2dc4a3313c236fedbd6df664722cd47f3d91a72
msg334098 - (view) Author: miss-islington (miss-islington) Date: 2019-01-20 19:06
New changeset 2fa53cfa8960f4bcb36718da4424e980fc305ef5 by Miss Islington (bot) in branch '3.7':
bpo-35699: fix distuils cannot detect Build Tools 2017 anymore (GH-11495)
https://github.com/python/cpython/commit/2fa53cfa8960f4bcb36718da4424e980fc305ef5
msg334111 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-01-21 05:30
Merged on behalf of Steve.  Thanks, Marc!
msg334150 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-01-21 16:54
There is one buildbot (https://buildbot.python.org/all/#builders/40/builds/1524) that started randomly failing unrelated tests with this change - usually one of zipapp, pkgutil, or zipimport. I don't have any idea what the relationship here is, unless the distutils tests have stopped skipping on those machines and are affecting some sort of global state (which they shouldn't, and the other tests shouldn't be relying on, but here we are).

I haven't looked at the actual tests (from my phone) yet. Anyone have any better ideas?
msg334166 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-01-21 18:50
I read a few of the logs, and all errors roots in the same place, when the test case tries to remove the tempdir during teardown/cleanup. The Windows (and other platforms not supporting directory fds) implementation of rmtree can have race conditions if files are added while the directory being walked, causing the error.

I don’t know the case of CPython, but Pipenv has a similar problem when running concurrent tests with subprocesses. We never solved it (but simply wrap the rmtree call inside a try block and look away).
msg334186 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-21 21:58
@Steve, we are investigating and tracking this issue here:

https://bugs.python.org/issue35795

According to the buildbot owner "directory tree removal needs to go through support.rmtree, not any of the stdlib methods for doing so.  In a nutshell, the tests need to be changed to do this".

Not sure why this has happened now
History
Date User Action Args
2019-01-21 21:58:10pablogsalsetkeywords: patch, patch, patch

messages: + msg334186
2019-01-21 18:50:45uranusjrsetmessages: + msg334166
2019-01-21 16:54:00steve.dowersetkeywords: patch, patch, patch
nosy: + pablogsal
messages: + msg334150

2019-01-21 05:30:55ned.deilysetstatus: open -> closed
messages: + msg334111

keywords: patch, patch, patch
resolution: fixed
stage: patch review -> resolved
2019-01-20 19:06:10miss-islingtonsetnosy: + miss-islington
messages: + msg334098
2019-01-20 18:48:25miss-islingtonsetpull_requests: + pull_request11391
2019-01-20 18:48:13miss-islingtonsetpull_requests: + pull_request11390
2019-01-20 18:48:01miss-islingtonsetpull_requests: + pull_request11389
2019-01-20 18:47:44ned.deilysetnosy: + ned.deily
messages: + msg334097
2019-01-20 07:42:05steve.dowersetkeywords: patch, patch, patch

messages: + msg334079
2019-01-19 08:01:45uranusjrsetnosy: + uranusjr
2019-01-10 07:26:11schlamarsetkeywords: + patch
stage: patch review
pull_requests: + pull_request11018
2019-01-10 07:26:04schlamarsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11017
2019-01-10 07:25:57schlamarsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11016
2019-01-10 07:18:48schlamarsetmessages: + msg333365
2019-01-09 22:43:05steve.dowersetmessages: + msg333351
2019-01-09 12:53:53schlamarcreate