Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x #77795

Closed
vstinner opened this issue May 23, 2018 · 22 comments
Closed

Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x #77795

vstinner opened this issue May 23, 2018 · 22 comments
Assignees
Labels
3.8 only security fixes build The build process and cross-build OS-windows

Comments

@vstinner
Copy link
Member

BPO 33614
Nosy @pfmoore, @vstinner, @tjguk, @ned-deily, @zware, @serhiy-storchaka, @zooba, @miss-islington
PRs
  • bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated #7165
  • [3.7] bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165) #7167
  • [3.6] bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165) #7168
  • bpo-33614: Override exit code in find_msbuild.bat #7169
  • [3.7] bpo-33614: Override exit code in find_msbuild.bat (GH-7169) #7176
  • [3.6] bpo-33614: Override exit code in find_msbuild.bat (GH-7169) #7177
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2018-05-28.23:17:58.099>
    created_at = <Date 2018-05-23.12:20:55.236>
    labels = ['build', '3.8', 'OS-windows']
    title = 'Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x'
    updated_at = <Date 2018-05-28.23:17:58.097>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2018-05-28.23:17:58.097>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2018-05-28.23:17:58.099>
    closer = 'steve.dower'
    components = ['Build', 'Windows']
    creation = <Date 2018-05-23.12:20:55.236>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 33614
    keywords = ['patch']
    message_count = 22.0
    messages = ['317392', '317398', '317406', '317420', '317426', '317457', '317459', '317463', '317464', '317465', '317518', '317594', '317597', '317598', '317903', '317904', '317908', '317909', '317913', '317939', '317950', '317954']
    nosy_count = 8.0
    nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ned.deily', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'miss-islington']
    pr_nums = ['7165', '7167', '7168', '7169', '7176', '7177']
    priority = None
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue33614'
    versions = ['Python 3.8']

    @vstinner
    Copy link
    Member Author

    http://buildbot.python.org/all/#/builders/80/builds/244

       (Link target) -> 
         python3_d.def : error LNK2001: unresolved external symbol PyImport_GetModule [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
         python3_d.def : error LNK2001: unresolved external symbol Py_UTF8Mode [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
         D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\amd64\python3_d.lib : fatal error LNK1120: 2 unresolved externals [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
    

    Maybe it's related to this recent change:

    commit 4e29f56
    Author: Serhiy Storchaka <storchaka@gmail.com>
    Date: Tue May 22 20:59:42 2018 +0300

    Add missed details of the C API introduced in 3.7. (GH-7047)
    
    * Set the limited API version for PyImport_GetModule and PyOS_*Fork
      functions.
    * Add PyImport_GetModule and Py_UTF8Mode in [PC/python3.def](https://github.com/python/cpython/blob/main/PC/python3.def).
    * Add several functions in [Doc/data/refcounts.dat](https://github.com/python/cpython/blob/main/Doc/data/refcounts.dat).
    

    The two missing symbols are defined by:

    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
    PyAPI_FUNC(PyObject *) PyImport_GetModule(PyObject *name);
    #endif

    and

    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
    PyAPI_DATA(int) Py_UTF8Mode;
    #endif

    They are not part of the stable ABI.

    @vstinner vstinner added build The build process and cross-build OS-windows 3.8 only security fixes labels May 23, 2018
    @serhiy-storchaka
    Copy link
    Member

    My bad, I didn't test that change on Windows myself, and CI seems was broken at that moment.

    But I have no ideas why the build is failed. These names are part of stable ABI since version 3.7. I don't see any difference between PyImport_GetModule and say PyInterpreterState_GetID.

    @serhiy-storchaka
    Copy link
    Member

    It succeeds after removing directories amd64 and obj in PCbuild. Seems they are left from old builds and the clean up script did nor remove them.

    This may be a cause of other weird bugs on Windows.

    @zooba
    Copy link
    Member

    zooba commented May 23, 2018

    I guess that means there is an invalid date comparison going on to detect when the file has changed. This file doesn't change often, so it's easy to miss :)

    All the critical builds should forcibly rebuild it, so this isn't urgent, but I'll take a look next time I get a chance to work on the build files.

    @zooba zooba self-assigned this May 23, 2018
    @serhiy-storchaka
    Copy link
    Member

    If "PCbuild\build.bat -p x64" builds Python, "PCbuild\clean.bat -p x64" removes all files in PCbuild\amd64 but python3stub.exp and python3stub.lib.

    @ned-deily
    Copy link
    Member

    All the critical builds should forcibly rebuild it, so this isn't urgent, but I'll take a look next time I get a chance to work on the build files.

    It's more than a *bit* urgent since it is causing one of the 3.7 stable buildbots to fail to build and not run tests and we normally would not allow a release with stable buildbot failures.

    @ned-deily
    Copy link
    Member

    @zooba
    Copy link
    Member

    zooba commented May 23, 2018

    Okay, I'll take a look now.

    @ned-deily
    Copy link
    Member

    Steve: just a minute ago, zware out the win8.1 bot's repo and restarted a 3.7 build.

    @ned-deily
    Copy link
    Member

    er, "cleaned out"

    @ned-deily
    Copy link
    Member

    That Windows8.1 buildbot sure keeps busy with all those builders assigned to it :) Those refleaks builds take a *long* time!

    http://buildbot.python.org/all/#/workers/11

    But I see at least one builder has made it through successfully and no longer fails in this manner:

    http://buildbot.python.org/all/#/builders/12/builds/892

    So I think we can close this now. Thanks, everyone.

    @serhiy-storchaka
    Copy link
    Member

    I think the underlying issue is not fixed still. Adding new symbols in python3.def will break this buildbot (as well as users worktrees) again.

    @zooba
    Copy link
    Member

    zooba commented May 24, 2018

    Considering we didn't change anything in the sources, I wouldn't expect it to be fixed.

    It doesn't have to block 3.7 if Ned's okay with it, but I'd like to keep this open and fix the build scripts.

    @zooba zooba removed the 3.7 (EOL) end of life label May 24, 2018
    @zooba zooba reopened this May 24, 2018
    @ned-deily
    Copy link
    Member

    Sorry, I *did* close the issue prematurely; thanks for noticing!

    @zooba
    Copy link
    Member

    zooba commented May 28, 2018

    New changeset e97ba4c by Steve Dower in branch 'master':
    bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165)
    e97ba4c

    @ned-deily
    Copy link
    Member

    Steve - PR 7165 appears to have caused a buildbot failure:

    http://buildbot.python.org/all/#/builders/12/builds/910

    @zooba
    Copy link
    Member

    zooba commented May 28, 2018

    Thanks! I guess we were getting lucky in the past, as that error could have shown up for any number of reasons...

    @miss-islington
    Copy link
    Contributor

    New changeset 3757939 by Miss Islington (bot) in branch '3.7':
    bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165)
    3757939

    @miss-islington
    Copy link
    Contributor

    New changeset c489a76 by Miss Islington (bot) in branch '3.6':
    bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165)
    c489a76

    @zooba
    Copy link
    Member

    zooba commented May 28, 2018

    I *believe* this change will prevent the issue in the future, so marking this as fixed. But I wasn't able to reproduce it locally, so there may be something funny about either the file system or the version of MSBuild on the buildbots that caused it to not rebuild properly.

    In any case, my change causes it to rebuild based on contents rather than timestamp, so it should be more reliable.

    @zooba zooba closed this as completed May 28, 2018
    @vstinner
    Copy link
    Member Author

    It seems like the commit c489a76
    broke the AMD64 Windows10 buildbot: bpo-33675. I reopen this issue.

    @vstinner vstinner reopened this May 28, 2018
    @zooba
    Copy link
    Member

    zooba commented May 28, 2018

    Already fixed in #51418, #51425 and #51426 (which I apparently typo'd "bpo" in the commit message, so they didn't link up properly here).

    @zooba zooba closed this as completed May 28, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes build The build process and cross-build OS-windows
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants