Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update PCBuild for VS 2015 #67108

Closed
zooba opened this issue Nov 22, 2014 · 31 comments
Closed

Update PCBuild for VS 2015 #67108

zooba opened this issue Nov 22, 2014 · 31 comments
Assignees
Labels
build The build process and cross-build OS-windows type-feature A feature request or enhancement

Comments

@zooba
Copy link
Member

zooba commented Nov 22, 2014

BPO 22919
Nosy @pitrou, @vstinner, @tjguk, @jkloth, @tpn, @zware, @zooba
Files
  • projects_highlights.diff
  • projects_full.diff
  • fixups.diff: tcl/tk/OpenSSL fixups
  • cpython_93607_to_93616.diff
  • python3.diff
  • round4.diff
  • round5complete.patch
  • round6incremental.patch
  • round7complete.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2015-01-17.23:11:30.791>
    created_at = <Date 2014-11-22.21:40:07.139>
    labels = ['type-feature', 'OS-windows', 'build']
    title = 'Update PCBuild for VS 2015'
    updated_at = <Date 2015-11-06.18:08:00.514>
    user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2015-11-06.18:08:00.514>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2015-01-17.23:11:30.791>
    closer = 'steve.dower'
    components = ['Build', 'Windows']
    creation = <Date 2014-11-22.21:40:07.139>
    creator = 'steve.dower'
    dependencies = []
    files = ['37246', '37247', '37248', '37251', '37265', '37270', '37374', '37392', '37405']
    hgrepos = []
    issue_num = 22919
    keywords = ['patch', 'needs review']
    message_count = 31.0
    messages = ['231531', '231533', '231541', '231547', '231593', '231604', '231631', '231634', '231635', '231878', '232237', '232239', '232248', '232249', '232344', '232410', '232570', '232571', '233637', '233641', '233643', '233644', '233648', '233652', '233671', '233945', '233946', '233969', '234202', '254205', '254206']
    nosy_count = 11.0
    nosy_names = ['pitrou', 'vstinner', 'tim.golden', 'jkloth', 'trent', 'BreamoreBoy', 'python-dev', 'zach.ware', 'mdengler', 'steve.dower', 'Kevin.Phillips']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue22919'
    versions = ['Python 3.5']

    @zooba
    Copy link
    Member Author

    zooba commented Nov 22, 2014

    I've basically finished the core of the work to refresh the PCBuild project files and support building with VS 2015, and I believe it's ready to merge in.

    Though the title says VS 2015, builds will work fine with VS 2010 and VS 2013 (and probably VS 2012, but I didn't try it). Even so, I've copied the old project files into PC/VS10.0, though I'd be happy to just forget them.

    I suspect Tools/msi/msi.py will stop working with this change. My replacement installer is not ready yet. If anyone's particularly concerned about msi.py then I can try and restore it before checking this in, but I'm 99.9% certain it won't be used for 3.5, so I don't see the point.

    I've attached two patches, one with all the changes and the other with the "highlights" - the diff between the diffs are most of the project files and the added/deleted files. I imagine practically everyone will have a better time viewing the changes in their own hg tools, so the changes can be pulled from my sandbox with "hg pull https://hg.python.org/sandbox/steve.dower -b Projects" or viewed at https://hg.python.org/sandbox/steve.dower/shortlog/d08b456124e5. (I'll rebase these as a single commit in default when it goes in.)

    A few changes that may deserve some more discussion:

    • Builds for 32-bit now go to PCBuild\win32 instead of PCBuild (the batch file updates should make this transparent)

    • The project that used to trigger the OpenSSL build is now two projects (libeay, ssleay) that do the build directly, so they may need updating whenever we update the OpenSSL version or change _ssl/_hashlib. I'm obviously happy to track these. (The advantages are proper incremental builds, better debugging, better optimisation - definitely PGO if we turn that back on.)

    • The posixmodule.c update for Py_Verify_fd may become redundant - the VC14 CRT will include a per-thread function to change the invalid parameter handling, so when that's available we should be able to switch away from this trickery completely.

    • I changed some GetVersion calls (which no longer behave properly as of Windows 8.1) to use functions from VersionHelpers.h in VS 2013 and later.

    • build.bat now builds in parallel by default, with '-M' to disable.

    • PCBuild/readme.txt has been updated as if VS 2015 has already been released. This is not yet true, but I doubt anyone who notices will be particularly confused or upset.

    @zooba zooba self-assigned this Nov 22, 2014
    @zooba zooba added build The build process and cross-build OS-windows type-feature A feature request or enhancement labels Nov 22, 2014
    @zooba
    Copy link
    Member Author

    zooba commented Nov 22, 2014

    Oh, one other thing I just thought of: the current release of nmake (in VS 2015) has a regression that will make TCL and TK fail to build, but I have a workaround for their makefile. OpenSSL also had a problem with VC14 but their fix isn't in the build we've got on svn.python.org.

    If possible, I'd like to get the workarounds into our sources for those before this is checked in, so that the buildbots (currently there's one that I know of with VS 2015 Preview installed) can pick up those fixes and won't fail because of these.

    The nmake issue should be fixed for VS 2015 RC and the OpenSSL issue is already fixed (in their 1.1 branch, IIRC). I've attached a patch listing the changes needed.

    @zooba
    Copy link
    Member Author

    zooba commented Nov 23, 2014

    Fixed and pushed a minor issue with debug builds - I wasn't setting the 'g' suffix on the tcltk libs correctly.

    @zooba
    Copy link
    Member Author

    zooba commented Nov 23, 2014

    Pushed some changes based on Zach's feedback and added a new complete patch (minus the deletion of the PC/VS9.0 and PC/VS10.0 folders).

    @zooba
    Copy link
    Member Author

    zooba commented Nov 24, 2014

    Added some fixes to the python3.dll build (and xxlimited test project) which I noticed thanks to Antoine's feedback.

    @zware
    Copy link
    Member

    zware commented Nov 24, 2014

    Another round of review posted on Rietveld. I'll try to get the VS2015 preview downloaded today and give some more feedback after using it.

    Just one inconsequential note on python3.diff: the 'lib' command in python3dll.vcxproj could use a '/nologo' flag.

    @zware
    Copy link
    Member

    zware commented Nov 25, 2014

    After a debug build with VS2015, my biggest concern is this:

    ..\PC\bdist_wininst\install.rc(19): fatal error RC1015: cannot open include file 'afxres.h'. [P:\ath\to\cpython\PCbuild\bdist_wininst.vcxproj]

    There's also a slew of new warnings, mostly C4456 and C4457 with a few C4459s thrown in. I think we can safely disable those three (which are all about name shadowing), perhaps with a XXX comment alongside to the effect of 'it might be nice to not disable these warnings someday'.

    Interestingly, there's also this warning:

    C:\Program Files (x86)\Windows Kits\8.1\Include\um\shlobj.h(1054): warning C4091: 'typedef ': ignored on left of 'tagGPFIDL_FLAGS' when no variable is declared [P:\ath\to\cpython\PCbuild\pylauncher.vcxproj]

    And a matching one for pywlauncher.vcxproj.

    @zooba
    Copy link
    Member Author

    zooba commented Nov 25, 2014

    After a debug build with VS2015, my biggest concern is this:

    ..\PC\bdist_wininst\install.rc(19): fatal error RC1015: cannot open
    include file 'afxres.h'. [P:\ath\to\cpython\PCbuild\bdist_wininst.vcxproj]

    I thought I had a workaround for this, but I'll have to check it out. Oddly enough, all of my builds have been fine (I think I'm mostly doing release builds - does that make a difference?)

    There's also a slew of new warnings, mostly C4456 and C4457 with a
    few C4459s thrown in. I think we can safely disable those three
    (which are all about name shadowing), perhaps with a XXX comment
    alongside to the effect of 'it might be nice to not disable these
    warnings someday'.

    I'm torn about that. Ideally I'd like to suppress the existing ones and still have new ones appear, but that probably requires going through and fixing them. Maybe leaving the warnings there will encourage someone to do it? :)

    Interestingly, there's also this warning:

    C:\Program Files (x86)\Windows Kits\8.1\Include\um\shlobj.h(1054):
    warning C4091: 'typedef ': ignored on left of 'tagGPFIDL_FLAGS' when
    no variable is declared [P:\ath\to\cpython\PCbuild\pylauncher.vcxproj]

    And a matching one for pywlauncher.vcxproj.

    This is a known issue with the Windows SDK - they need to release a new version to handle the changes in the compiler. It doesn't affect the Python build, though it is currently breaking some of my installer work :( I'm confident it will be fixed soon - they're getting a lot of reports about it.

    @zooba
    Copy link
    Member Author

    zooba commented Nov 25, 2014

    Added changes from Zach's last review.

    Highlight: I deleted the make_versioninfo project and build step :)

    @zooba
    Copy link
    Member Author

    zooba commented Nov 30, 2014

    > After a debug build with VS2015, my biggest concern is this:
    >
    > ..\PC\bdist_wininst\install.rc(19): fatal error RC1015: cannot open
    > include file 'afxres.h'. [P:\ath\to\cpython\PCbuild\bdist_wininst.vcxproj]

    I thought I had a workaround for this, but I'll have to check it out.
    Oddly enough, all of my builds have been fine (I think I'm mostly
    doing release builds - does that make a difference?)

    I haven't been able to reproduce this. bdist_wininst isn't selected to build for debug in VS, but it builds fine for me anyway. Was your repo dirty? (It didn't make a difference for me, but I have had files occasionally not update properly when switching branches.)

    @zooba
    Copy link
    Member Author

    zooba commented Dec 6, 2014

    Does anyone else want to provide any feedback? Or should I go ahead and get it in (after rebasing/retesting/etc.)? The feedback so far has been really helpful.

    @zware
    Copy link
    Member

    zware commented Dec 6, 2014

    Sorry Steve, I haven't been on Windows since my last review to give it
    another try. And, of course, somewhere in the 5 seconds between shutting
    down from Gentoo and turning back on for Windows last night, my laptop
    decided that POSTing was just too much effort, and it would rather just sit
    around spinning the fan and not doing anything else at all.

    I should be able to get VS2015 set up on my backup (wife's) machine
    Tuesday; until then, if you'd like to post another full patch to appease
    Rietveld, I can try to look through it again.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 6, 2014

    Rebased everything onto default and pushed it to my sandbox.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 6, 2014

    Having trouble getting the patch uploaded... next attempt.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 9, 2014

    Small incremental patch from Zach's feedback and a few sysconfig changes I missed.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 10, 2014

    Rebased and updated NEWS, so here's a complete patch (and it's all in the latest change in https://hg.python.org/sandbox/steve.dower#Projects)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Dec 12, 2014

    New changeset 5754f069b123 by Steve Dower in branch 'default':
    Issue bpo-22919: Windows build updated to support VC 14.0 (Visual Studio 2015), which will be used for the official 3.5 release.
    https://hg.python.org/cpython/rev/5754f069b123

    @zooba
    Copy link
    Member Author

    zooba commented Dec 12, 2014

    I'll be closely tracking any issues that arise out of this throughout the next couple of days, but it should be fairly smooth (especially for people who don't upgrade VS immediately). There are certainly a few things that are broken, but I'll make separate issues for those.

    @zooba zooba closed this as completed Dec 12, 2014
    @vstinner
    Copy link
    Member

    vstinner commented Jan 8, 2015

    Hi, I'm no more able to compile Python 3.5 on Windows. I have Visual Studio 2008 Express, Visual Studio 2010 Express. VS 2010 is unable to open PCbuild/pcbuild.sln: it says that the file was created on a more recent Visual Studio version and it is unable to open it.

    I tried to install Visual Studio 2013 community but it told me that my Windows Seven is too old: it looks like Windows 8.1 or newer is required :-(

    I found a Visual Studio 2015 Ultime preview, but I don't want to install an heavy "Ultime" IDE, I just want a simple C compiler...

    Though the title says VS 2015, builds will work fine with VS 2010 and VS 2013 (and probably VS 2012, but I didn't try it). Even so, I've copied the old project files into PC/VS10.0, though I'd be happy to just forget them.

    There is no PC/VS10.0 file or directory in the Python source code. I only found two .sln files: PCbuild/pcbuild.sln and PC/example_nt/example.sln.

    The devguide looks outdated: it only mentions Visual Studio 2010 (and 2008 for Python 2.7). Can you please update it?
    https://docs.python.org/devguide/setup.html#compiling

    I am no more able to develop on Windows, the issue becomes very annoying for me. I hate having to install a huge IDE to compile Python.

    @vstinner vstinner reopened this Jan 8, 2015
    @vstinner
    Copy link
    Member

    vstinner commented Jan 8, 2015

    Oh by the way, 2 Windows buildbot slaves are no more able to compile Python. I don't know if it's related to this issue or not.

    Builder "AMD64 Windows7 SP1 3.x", owned by Jeremy Kloth:
    http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x

    2>LINK : fatal error LNK1104: cannot open file 'C:\buildbot.python.org\3.x.kloth-win64\build\PCBuild\amd64\python35_d.dll' [C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]

    Builder "x86 Windows Server 2003 [SB] 3.x" managed by Snakebite.org (trent):
    http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%203.x

    E:\Data\buildslave\cpython\3.x.snakebite-win2k3r2sp2-x86\build\Tools\buildbot\..\..\build\TEEB4C~1 - The process cannot access the file because it is being used by another process.
    (...)
    2>E:\Data\buildslave\cpython\3.x.snakebite-win2k3r2sp2-x86\build\PCbuild\pythoncore.vcxproj(415,5): warning : Toolset v100 is not used for official builds. Your build may have errors or incompatibilities.
    (...)
    C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\Microsoft.Cpp.Win32.Targets(147,5): error MSB6006: "CL.exe" exited with code 128. [E:\Data\buildslave\cpython\3.x.snakebite-win2k3r2sp2-x86\build\PCbuild\pythoncore.vcxproj]

    Both slaves are part of the group of "3.x stable" buildbots :-(

    @pitrou
    Copy link
    Member

    pitrou commented Jan 8, 2015

    Visual Studio 2013 Professional works fine under Windows 7 SP1 here.

    @vstinner
    Copy link
    Member

    vstinner commented Jan 8, 2015

    Visual Studio 2013 Professional works fine under Windows 7 SP1 here.

    Ok, good to know. But is it correct that the free version of VS 2013 (community) requires Windows 8.1 or newer? It's not cool to require to upgrade Windows to being able to freely compile Python.

    My MSDN account is probably outdated, I don't think that I have access to the Professional version.

    Would it be possible to build Python with the light Windows SDK which basically only includes the C compiler and the CRT?

    @BreamoreBoy
    Copy link
    Mannequin

    BreamoreBoy mannequin commented Jan 8, 2015

    @victor I don't know what version you need for Windows 7 or earlier but I can tell you that VS 2013 Community edition is *NOT* free, I fell into that trap myself, you need the Express edition.

    @zooba
    Copy link
    Member Author

    zooba commented Jan 8, 2015

    You will need Windows 7 *SP1*, as I don't think VS will run without the updates. There is also a service pack for VS 2010 that may enable opening the newer solution - it certainly worked for me.

    We decided not to keep the old project files as they weren't being maintained.

    Community edition is free, but you need a Microsoft account (also free) to register it. I don't know why this is necessary, but that's the way they set it up.

    I'll check out that buildbot when I get a chance (Trent's one is known to be busted). The stable buildbots and the one with VS 2015 were doing fine last time I looked.

    I haven't tried with the SDK compilers, but it should be possible to build with them now. VS is no longer required to build from the command line.

    @zooba
    Copy link
    Member Author

    zooba commented Jan 8, 2015

    Just emailed Jeremy about the buildbot. It looks like the last time tests were run something didn't clean up properly and left the build output locked. There's nothing wrong with the project files.

    @vstinner
    Copy link
    Member

    You will need Windows 7 *SP1*

    I do have Windows 7 SP1.

    There is also a service pack for VS 2010 that may enable opening the newer solution - it certainly worked for me.

    Oh, I also worked for me. I have VS 2010 *Express* and I'm now able to open the project after installing the Service Pack 1 of VS 2010.

    Note: The binary (in debug mode) moved from PCbuild\python_d.exe to PCbuild\win32\python_d.exe

    @zooba
    Copy link
    Member Author

    zooba commented Jan 13, 2015

    I do have Windows 7 SP1.

    I expected so, but didn't want to assume.

    I have VS 2010 *Express* and I'm now able to open the project after installing the Service Pack 1 of VS 2010.

    Glad to hear it. You made me a little nervous there :) (I don't feel like requiring VS 2010 SP1 is unreasonable, but I could be wrong... not sure there's any sensible way to support VS 2010 RTM though.)

    Note: The binary (in debug mode) moved from PCbuild\python_d.exe to PCbuild\win32\python_d.exe

    Yes, that was always a pet peeve of mine, but I got big thanks from the other Windows nosy list for making that move. At this point, everything should be okay with the new location, but if there are any more issues just let me know and I'll check them out.

    @zware
    Copy link
    Member

    zware commented Jan 13, 2015

    STINNER Victor added the comment:

    Note: The binary (in debug mode) moved from PCbuild\python_d.exe to PCbuild\win32\python_d.exe

    There ought to be a 'python.bat' in the root of the source tree that
    will always point to the last-built python[_d].exe, which may ease
    things for you.

    @zooba
    Copy link
    Member Author

    zooba commented Jan 17, 2015

    Closing again, as Victor's issue was resolved (VS 2010 SP1 is needed, and I'm updating the devguide to specify that on bpo-23257).

    @zooba zooba closed this as completed Jan 17, 2015
    @KevinPhillips
    Copy link
    Mannequin

    KevinPhillips mannequin commented Nov 6, 2015

    This change has the adverse side effect of requiring users of Python 3.5 to use the Visual Studio 2015 runtime library. Further, as is documented on Microsoft's website, the minimum system requirements for installing and running this runtime is Windows 7 SP1 or newer.

    As a result, adopting this new compiler for the Python runtime prevents users of operating systems older than this from using this version. Even users with Windows 7 pre service pack are unable to use it. Based on a quick review of the comment thread here, the choice for adopting the 2015 compiler was relatively arbitrary and thus an older version, say 2013 for example, could have been used instead and older operation systems would still be supported with little to no impact on Python.

    While this is unfortunate to say the least, what makes matters worse is there appears to be little to no information on the Python.org website indicating that the minimum system requirements have now been affected in this way. Also, for users attempting to install v3.5 on a system older than this are presented with a cryptic error message basically just saying that "some unknown error has occurred", leaving them scratching their head trying to figure out what's wrong.

    Seeing as how you can't go back in time and fix this in 3.5.0, I am hoping that at the very least you could add some information about this to some obvious location on the main Python.org website, and maybe consider updating the installation for future 3.5.x releases such that a check is done to see if the operating system meets these new minimum requirements and present the user with some helpful information that would lead them to a resolution.

    @zooba
    Copy link
    Member Author

    zooba commented Nov 6, 2015

    The check is already added and will be in 3.5.1.

    Changes to the python.org website should be suggested on https://github.com/python/pythondotorg (but I agree it could be more clear on the main site - the rule that we only support the same versions of Windows that Microsoft supports is not very well known, unfortunately).

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build OS-windows type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants