classification
Title: Compilation of Python fails on AMD64 Windows8.1 Refleaks 3.x
Type: Stage: resolved
Components: Build, Windows Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: miss-islington, ned.deily, paul.moore, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: Keywords: patch

Created on 2018-05-23 12:20 by vstinner, last changed 2018-05-28 23:17 by steve.dower. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7165 merged steve.dower, 2018-05-28 18:43
PR 7167 merged miss-islington, 2018-05-28 19:33
PR 7168 merged miss-islington, 2018-05-28 19:34
PR 7169 merged steve.dower, 2018-05-28 19:47
PR 7176 merged miss-islington, 2018-05-28 21:14
PR 7177 merged miss-islington, 2018-05-28 21:14
Messages (22)
msg317392 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-23 12:20
http://buildbot.python.org/all/#/builders/80/builds/244

       (Link target) -> 
         python3_d.def : error LNK2001: unresolved external symbol PyImport_GetModule [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
         python3_d.def : error LNK2001: unresolved external symbol Py_UTF8Mode [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]
         D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\amd64\python3_d.lib : fatal error LNK1120: 2 unresolved externals [D:\buildarea\3.x.ware-win81-release.refleak\build\PCbuild\python3dll.vcxproj]

Maybe it's related to this recent change:

commit 4e29f566e8821c09d8274eadcdd355e8b1284b8b
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Tue May 22 20:59:42 2018 +0300

    Add missed details of the C API introduced in 3.7. (GH-7047)
    
    * Set the limited API version for PyImport_GetModule and PyOS_*Fork
      functions.
    * Add PyImport_GetModule and Py_UTF8Mode in PC/python3.def.
    * Add several functions in Doc/data/refcounts.dat.

The two missing symbols are defined by:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
PyAPI_FUNC(PyObject *) PyImport_GetModule(PyObject *name);
#endif

and

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03070000
PyAPI_DATA(int) Py_UTF8Mode;
#endif

They are not part of the stable ABI.
msg317398 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-23 12:49
My bad, I didn't test that change on Windows myself, and CI seems was broken at that moment.

But I have no ideas why the build is failed. These names are part of stable ABI since version 3.7. I don't see any difference between PyImport_GetModule and say PyInterpreterState_GetID.
msg317406 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-23 13:53
It succeeds after removing directories amd64 and obj in PCbuild. Seems they are left from old builds and the clean up script did nor remove them.

This may be a cause of other weird bugs on Windows.
msg317420 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-23 16:13
I guess that means there is an invalid date comparison going on to detect when the file has changed. This file doesn't change often, so it's easy to miss :)

All the critical builds should forcibly rebuild it, so this isn't urgent, but I'll take a look next time I get a chance to work on the build files.
msg317426 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-23 16:49
If "PCbuild\build.bat -p x64" builds Python, "PCbuild\clean.bat -p x64" removes all files in PCbuild\amd64 but python3stub.exp and python3stub.lib.
msg317457 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-23 21:06
> All the critical builds should forcibly rebuild it, so this isn't urgent, but I'll take a look next time I get a chance to work on the build files.

It's more than a *bit* urgent since it is causing one of the 3.7 stable buildbots to fail to build and not run tests and we normally would not allow a release with stable buildbot failures.
msg317459 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-23 21:07
(for example, http://buildbot.python.org/all/#/builders/133/builds/251 )
msg317463 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-23 22:06
Okay, I'll take a look now.
msg317464 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-23 22:07
Steve: just a minute ago, zware out the win8.1 bot's repo and restarted a 3.7 build.
msg317465 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-23 22:08
er, "cleaned out"
msg317518 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 03:03
That Windows8.1 buildbot sure keeps busy with all those builders assigned to it :)  Those refleaks builds take a *long* time!

http://buildbot.python.org/all/#/workers/11

But I see at least one builder has made it through successfully and no longer fails in this manner:

http://buildbot.python.org/all/#/builders/12/builds/892

So I think we can close this now.  Thanks, everyone.
msg317594 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-24 17:52
I think the underlying issue is not fixed still. Adding new symbols in python3.def will break this buildbot (as well as users worktrees) again.
msg317597 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-24 18:04
Considering we didn't change anything in the sources, I wouldn't expect it to be fixed.

It doesn't have to block 3.7 if Ned's okay with it, but I'd like to keep this open and fix the build scripts.
msg317598 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 18:16
Sorry, I *did* close the issue prematurely; thanks for noticing!
msg317903 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-28 19:32
New changeset e97ba4c690613d734843db218aeedce2f0e5937f by Steve Dower in branch 'master':
bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165)
https://github.com/python/cpython/commit/e97ba4c690613d734843db218aeedce2f0e5937f
msg317904 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-28 19:35
Steve - PR 7165 appears to have caused a buildbot failure:

http://buildbot.python.org/all/#/builders/12/builds/910
msg317908 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-28 19:49
Thanks! I guess we were getting lucky in the past, as that error could have shown up for any number of reasons...
msg317909 - (view) Author: miss-islington (miss-islington) Date: 2018-05-28 19:51
New changeset 3757939e9c00a36f939d774ec5c79e5d6b9a77bb by Miss Islington (bot) in branch '3.7':
bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165)
https://github.com/python/cpython/commit/3757939e9c00a36f939d774ec5c79e5d6b9a77bb
msg317913 - (view) Author: miss-islington (miss-islington) Date: 2018-05-28 20:16
New changeset c489a767af46f33e73d96a5746e46a7365814db2 by Miss Islington (bot) in branch '3.6':
bpo-33614: Ensures module definition files for the stable ABI on Windows are correctly regenerated. (GH-7165)
https://github.com/python/cpython/commit/c489a767af46f33e73d96a5746e46a7365814db2
msg317939 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-28 22:44
I *believe* this change will prevent the issue in the future, so marking this as fixed. But I wasn't able to reproduce it locally, so there may be something funny about either the file system or the version of MSBuild on the buildbots that caused it to not rebuild properly.

In any case, my change causes it to rebuild based on contents rather than timestamp, so it should be more reliable.
msg317950 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-28 23:09
It seems like the commit c489a767af46f33e73d96a5746e46a7365814db2
 broke the AMD64 Windows10 buildbot: bpo-33675. I reopen this issue.
msg317954 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-05-28 23:17
Already fixed in GH-7169, GH-7176 and GH-7177 (which I apparently typo'd "bpo" in the commit message, so they didn't link up properly here).
History
Date User Action Args
2018-05-28 23:18:38steve.dowerlinkissue33675 superseder
2018-05-28 23:17:58steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg317954
2018-05-28 23:09:55vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg317950
2018-05-28 22:44:21steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg317939

stage: patch review -> resolved
2018-05-28 21:14:43miss-islingtonsetpull_requests: + pull_request6814
2018-05-28 21:14:36miss-islingtonsetpull_requests: + pull_request6813
2018-05-28 20:16:10miss-islingtonsetmessages: + msg317913
2018-05-28 19:51:18miss-islingtonsetnosy: + miss-islington
messages: + msg317909
2018-05-28 19:49:01steve.dowersetmessages: + msg317908
2018-05-28 19:47:45steve.dowersetpull_requests: + pull_request6804
2018-05-28 19:35:58ned.deilysetmessages: + msg317904
2018-05-28 19:34:16miss-islingtonsetpull_requests: + pull_request6803
2018-05-28 19:33:30miss-islingtonsetpull_requests: + pull_request6802
2018-05-28 19:32:09steve.dowersetmessages: + msg317903
2018-05-28 18:43:13steve.dowersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request6800
2018-05-24 18:16:40ned.deilysetmessages: + msg317598
2018-05-24 18:04:33steve.dowersetstatus: closed -> open
versions: - Python 3.7
messages: + msg317597

resolution: fixed -> (no value)
stage: resolved -> needs patch
2018-05-24 17:52:39serhiy.storchakasetmessages: + msg317594
2018-05-24 03:03:39ned.deilysetstatus: open -> closed
priority: release blocker ->
messages: + msg317518

resolution: fixed
stage: resolved
2018-05-23 22:08:07ned.deilysetmessages: + msg317465
2018-05-23 22:07:44ned.deilysetmessages: + msg317464
2018-05-23 22:06:36steve.dowersetmessages: + msg317463
2018-05-23 21:07:51ned.deilysetmessages: + msg317459
2018-05-23 21:07:02ned.deilysetversions: + Python 3.7
2018-05-23 21:06:22ned.deilysetpriority: normal -> release blocker
nosy: + ned.deily
messages: + msg317457

2018-05-23 16:49:02serhiy.storchakasetmessages: + msg317426
2018-05-23 16:13:26steve.dowersetassignee: steve.dower
messages: + msg317420
2018-05-23 13:53:38serhiy.storchakasetmessages: + msg317406
2018-05-23 12:49:53serhiy.storchakasetmessages: + msg317398
2018-05-23 12:21:17vstinnersetversions: + Python 3.8
2018-05-23 12:21:12vstinnersetnosy: + paul.moore, tim.golden, serhiy.storchaka, zach.ware, steve.dower
components: + Build, Windows
2018-05-23 12:20:55vstinnercreate