Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use VS 2017 compiler for build #75521

Closed
zooba opened this issue Sep 4, 2017 · 14 comments
Closed

Use VS 2017 compiler for build #75521

zooba opened this issue Sep 4, 2017 · 14 comments
Assignees
@zooba
Labels
3.7 (EOL) end of life build The build process and cross-build OS-windows type-feature A feature request or enhancement

Comments

@zooba
Copy link
Member

zooba commented Sep 4, 2017

BPO 31340
Nosy @pfmoore, @tjguk, @jkloth, @ned-deily, @methane, @zware, @zooba
PRs
  • bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) #3311
  • [3.6] bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (GH-3311) #3386
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
closed_at = <Date 2017-09-06.22:57:09.278>
created_at = <Date 2017-09-04.20:17:14.961>
labels = ['type-feature', '3.7', 'OS-windows', 'build']
title = 'Use VS 2017 compiler for build'
updated_at = <Date 2018-01-06.18:49:22.620>
user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2018-01-06.18:49:22.620>
actor = 'cgohlke'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2017-09-06.22:57:09.278>
closer = 'steve.dower'
components = ['Build', 'Windows']
creation = <Date 2017-09-04.20:17:14.961>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 31340
keywords = []
message_count = 14.0
messages = ['301253', '301486', '301537', '301538', '301562', '303862', '303863', '303864', '303867', '303868', '303884', '309399', '309547', '309568']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'jkloth', 'ned.deily', 'methane', 'cgohlke', 'zach.ware', 'steve.dower']
pr_nums = ['3311', '3386']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31340'
versions = ['Python 3.6', 'Python 3.7']
    @zooba
    Copy link
    Member Author

    zooba commented Sep 4, 2017

    The newer MSVC (v141) is available and reliable, and theoretically binary compatible with v140. This means we can update both Python 3.6 and 3.7 to build with it.

    Testing for this should include:

    • pythoncore with v141 and remainder with v140
    • Python with v141 and wheels with v140

    @zooba zooba added the 3.7 (EOL) end of life label Sep 4, 2017
    @zooba zooba self-assigned this Sep 4, 2017
    @zooba zooba added build The build process and cross-build OS-windows type-feature A feature request or enhancement labels Sep 4, 2017
    @zooba
    Copy link
    Member Author

    zooba commented Sep 6, 2017

    New changeset 5fcd5e6 by Steve Dower in branch 'master':
    bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (bpo-3311)
    5fcd5e6

    @zooba
    Copy link
    Member Author

    zooba commented Sep 6, 2017

    New changeset fd645ec by Steve Dower in branch '3.6':
    [3.6] bpo-31340: Change to building with MSVC v141 (included with Visual Studio 2017) (GH-3311) (bpo-3386)
    fd645ec

    @zooba
    Copy link
    Member Author

    zooba commented Sep 6, 2017

    Note that this should not change anything for 3.6, but enables using VS 2017 to build. If you only have VS 2015 then it will use that, and I've enabled AppVeyor for both to make sure we keep it that way.

    3.7 will also build with either, but there isn't as much importance in verifying VS 2015 compatibility.

    All of that said, there should be no difference in the built binaries.

    @zooba zooba closed this as completed Sep 6, 2017
    @methane
    Copy link
    Member

    methane commented Sep 7, 2017

    Nice! VS2017 is much lighter than VS2015 for install size!
    https://blogs.msdn.microsoft.com/vcblog/2016/08/22/the-lightweight-visual-studio-15-installer/

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 6, 2017

    I have Visual Studio 2015 and Visual Studio 2017 Community editions with latest patches installed.

    Building a minimal C extension on Python 3.6.3 with distutils now uses MSVC 14.11.25503. That is unexpected.

    When building complex extensions that link to static libraries that were built with MSVC 14.0, linking fails, e.g. when building Pillow:

    'jpeg.lib' was created with an older compiler than other objects; rebuild old objects and libraries

    It looks like practically v141 is not binary compatible with v140.

    @zooba
    Copy link
    Member Author

    zooba commented Oct 6, 2017

    Build artifacts (static libraries, in this case) are not compatible, but the built binaries are.

    This may be a reasonable argument for not updating distutils's support in minor versions (and I'm totally happy to just stop updating distutils period - see the recent distutils-sig discussion for more context). If it had been raised before 3.6.3 was released then I'd have switched it back (or at least swapped the order, so if v140 was found then it would use that first and fall back on v141). Now it's done, I'm not sure whether it hurts more to change it again in 3.6.4 or not...

    The immediate workaround is to set DISTUTILS_USE_SDK and run in a VS 2015 command prompt, which will bypass all the search logic. Better to rebuild the static libraries with v141 as well, but I know that's not trivial or always feasible. Alternatively, build with Python 3.6.2 or earlier - until we get the ABI stability properly figured out, that's still the best way to ensure compatibility with older releases.

    I'll give the change back some more thought.

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Oct 7, 2017

    I build most of my binaries after calling the correct vcvarsall.bat so I did not notice until today, when I was trying to build outside that environment.

    Also, I misread your comment (https://bugs.python.org/issue31340#msg301538) earlier and did not worry at that time.

    I guess this might become a bigger issue once Appveyor switches to Python 3.6.3. Projects using Anaconda's prebuilt static libraries for v140 will fail to link.

    How about giving priority to VS 2015, i.e. "If you have VS 2015 then it will use that"?

    @zooba
    Copy link
    Member Author

    zooba commented Oct 7, 2017

    How about giving priority to VS 2015

    That would be the fix, but it's not worth releasing an immediate 3.6.4 IMHO and by the time 3.6.4 comes about it could be more pain to change back than to leave it. That's the balance I'm thinking about for a while.

    Including Ned so he's aware. If we do a snap 3.6.4 for some other reason then it's worth swapping the order, but after a few months out there I think the damage, small as it is, will have been dealt with.

    @ned-deily
    Copy link
    Member

    If you think we should do a 3.6.4 to address this, we can do that. I'll leave it up to your judgement.

    @zooba
    Copy link
    Member Author

    zooba commented Oct 7, 2017

    No, but if we need to do 3.6.4 quickly for another reason, let me know and I'll make this change.

    It only affects people who are trying to share intermediate build files (which for MSVC purposes, includes static libraries) between their 3.6.2 and 3.6.3 builds. Full builds are going to be fine, and incremental builds should detect the changed tools and be able to rebuild everything.

    So it's solely cases where people are trying to manage the build around distutils rather than letting distutils handle it all that are problematic (and hey, both of these are terrible options! This is why we're discussing how to get people to just stop depending on distutils). I don't think it's worth churning the entire ecosystem just for that.

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Jan 3, 2018

    Since this has not been reverted/changed for 3.6.4 or 3.7.0a3, another potential issue arises: C extensions that are compiled with Visual Studio 2017 are linked to a newer version of vcruntime140.dll (latest is 14.12.25810, VS2017 15.5) than the DLL shipped with the CPython 3.6.4/3.7.0a3 installer (14.0.24210, VS2015 Update 3). There is no standard way C extensions that are compiled with VS2017 can load the correct/newer vcruntime140.dll. That said, I have not noticed any problems yet while testing with Python 3.7.0a3.

    @zooba
    Copy link
    Member Author

    zooba commented Jan 6, 2018

    Have you confirmed that's a problem? There are new compatibility promises that did not exist for earlier versions of the runtime, but that does not mean there are no bugs.

    @cgohlke
    Copy link
    Mannequin

    cgohlke mannequin commented Jan 6, 2018

    Have you confirmed that's a problem?

    No, I have not noticed any problems yet with with Python 3.7.0a3 and many extensions built with VS2017.5.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @zooba zooba

    Labels
    3.7 (EOL) end of life build The build process and cross-build OS-windows type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @methane @zooba @ned-deily