classification
Title: Use VS 2017 compiler for build
Type: enhancement Stage: resolved
Components: Build, Windows Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: cgohlke, inada.naoki, jkloth, ned.deily, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-09-04 20:17 by steve.dower, last changed 2018-01-06 18:49 by cgohlke. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3311 merged steve.dower, 2017-09-04 21:25
PR 3386 merged steve.dower, 2017-09-06 17:12
Messages (14)
msg301253 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-09-04 20:17
The newer MSVC (v141) is available and reliable, and theoretically binary compatible with v140. This means we can update both Python 3.6 and 3.7 to build with it.

Testing for this should include:
* pythoncore with v141 and remainder with v140
* Python with v141 and wheels with v140
msg301486 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-09-06 17:01
New changeset 5fcd5e64eec9ed67613b8fe7356fb8288151ceba by Steve Dower in branch 'master':
bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (#3311)
https://github.com/python/cpython/commit/5fcd5e64eec9ed67613b8fe7356fb8288151ceba
msg301537 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-09-06 22:55
New changeset fd645ec6f5696e841e6d49075f9fd81e54e74d91 by Steve Dower in branch '3.6':
[3.6] bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (GH-3311) (#3386)
https://github.com/python/cpython/commit/fd645ec6f5696e841e6d49075f9fd81e54e74d91
msg301538 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-09-06 22:57
Note that this should not change anything for 3.6, but enables using VS 2017 to build. If you only have VS 2015 then it will use that, and I've enabled AppVeyor for both to make sure we keep it that way.

3.7 will also build with either, but there isn't as much importance in verifying VS 2015 compatibility.

All of that said, there should be no difference in the built binaries.
msg301562 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2017-09-07 02:28
Nice! VS2017 is much lighter than VS2015 for install size!
https://blogs.msdn.microsoft.com/vcblog/2016/08/22/the-lightweight-visual-studio-15-installer/
msg303862 - (view) Author: Christoph Gohlke (cgohlke) Date: 2017-10-06 22:02
I have Visual Studio 2015 and Visual Studio 2017 Community editions with latest patches installed.

Building a minimal C extension on Python 3.6.3 with distutils now uses MSVC 14.11.25503. That is unexpected.

When building complex extensions that link to static libraries that were built with MSVC 14.0, linking fails, e.g. when building Pillow:

'jpeg.lib' was created with an older compiler than other objects; rebuild old objects and libraries

It looks like practically v141 is not binary compatible with v140.
msg303863 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-10-06 23:07
Build artifacts (static libraries, in this case) are not compatible, but the built binaries are.

This may be a reasonable argument for not updating distutils's support in minor versions (and I'm totally happy to just stop updating distutils period - see the recent distutils-sig discussion for more context). If it had been raised before 3.6.3 was released then I'd have switched it back (or at least swapped the order, so if v140 was found then it would use that first and fall back on v141). Now it's done, I'm not sure whether it hurts more to change it again in 3.6.4 or not...

The immediate workaround is to set DISTUTILS_USE_SDK and run in a VS 2015 command prompt, which will bypass all the search logic. Better to rebuild the static libraries with v141 as well, but I know that's not trivial or always feasible. Alternatively, build with Python 3.6.2 or earlier - until we get the ABI stability properly figured out, that's still the best way to ensure compatibility with older releases.

I'll give the change back some more thought.
msg303864 - (view) Author: Christoph Gohlke (cgohlke) Date: 2017-10-07 00:01
I build most of my binaries after calling the correct vcvarsall.bat so I did not notice until today, when I was trying to build outside that environment. 

Also, I misread your comment (https://bugs.python.org/issue31340#msg301538) earlier and did not worry at that time.

I guess this might become a bigger issue once Appveyor switches to Python 3.6.3. Projects using Anaconda's prebuilt static libraries for v140 will fail to link.

How about giving priority to VS 2015, i.e. "If you have VS 2015 then it will use that"?
msg303867 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-10-07 05:00
> How about giving priority to VS 2015

That would be the fix, but it's not worth releasing an immediate 3.6.4 IMHO and by the time 3.6.4 comes about it could be more pain to change back than to leave it. That's the balance I'm thinking about for a while.

Including Ned so he's aware. If we do a snap 3.6.4 for some other reason then it's worth swapping the order, but after a few months out there I think the damage, small as it is, will have been dealt with.
msg303868 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-10-07 05:03
If you think we should do a 3.6.4 to address this, we can do that.  I'll leave it up to your judgement.
msg303884 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-10-07 15:43
No, but if we need to do 3.6.4 quickly for another reason, let me know and I'll make this change.

It only affects people who are trying to share intermediate build files (which for MSVC purposes, includes static libraries) between their 3.6.2 and 3.6.3 builds. Full builds are going to be fine, and incremental builds should detect the changed tools and be able to rebuild everything. 

So it's solely cases where people are trying to manage the build around distutils rather than letting distutils handle it all that are problematic (and hey, both of these are terrible options! This is why we're discussing how to get people to just stop depending on distutils). I don't think it's worth churning the entire ecosystem just for that.
msg309399 - (view) Author: Christoph Gohlke (cgohlke) Date: 2018-01-03 07:32
Since this has not been reverted/changed for 3.6.4 or 3.7.0a3, another potential issue arises: C extensions that are compiled with Visual Studio 2017 are linked to a newer version of vcruntime140.dll (latest is 14.12.25810, VS2017 15.5) than the DLL shipped with the CPython 3.6.4/3.7.0a3 installer (14.0.24210, VS2015 Update 3). There is no standard way C extensions that are compiled with VS2017 can load the correct/newer vcruntime140.dll. That said, I have not noticed any problems yet while testing with Python 3.7.0a3.
msg309547 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-01-06 11:23
Have you confirmed that's a problem? There are new compatibility promises that did not exist for earlier versions of the runtime, but that does not mean there are no bugs.
msg309568 - (view) Author: Christoph Gohlke (cgohlke) Date: 2018-01-06 18:49
> Have you confirmed that's a problem?

No, I have not noticed any problems yet with with Python 3.7.0a3 and many extensions built with VS2017.5.
History
Date User Action Args
2018-01-06 18:49:22cgohlkesetmessages: + msg309568
2018-01-06 11:23:51steve.dowersetmessages: + msg309547
2018-01-03 07:32:11cgohlkesetmessages: + msg309399
2017-10-07 15:43:52steve.dowersetmessages: + msg303884
2017-10-07 05:03:14ned.deilysetmessages: + msg303868
2017-10-07 05:00:07steve.dowersetnosy: + ned.deily
messages: + msg303867
2017-10-07 00:01:07cgohlkesetmessages: + msg303864
2017-10-06 23:07:09steve.dowersetmessages: + msg303863
2017-10-06 22:02:05cgohlkesetnosy: + cgohlke
messages: + msg303862
2017-09-07 02:28:14inada.naokisetnosy: + inada.naoki
messages: + msg301562
2017-09-06 22:57:09steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg301538

stage: backport needed -> resolved
2017-09-06 22:55:27steve.dowersetmessages: + msg301537
2017-09-06 22:49:58steve.dowersetstage: needs patch -> backport needed
2017-09-06 17:12:15steve.dowersetpull_requests: + pull_request3393
2017-09-06 17:01:40steve.dowersetmessages: + msg301486
2017-09-04 21:25:14steve.dowersetpull_requests: + pull_request3339
2017-09-04 21:03:54jklothsetnosy: + jkloth
2017-09-04 20:17:15steve.dowercreate