classification
Title: distutils msvccompiler fails due to mspdb140.dll error on debug builds
Type: Stage: resolved
Components: Tests, Windows Versions:
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: jkloth, paul.moore, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: buildbot

Created on 2016-01-26 13:09 by vstinner, last changed 2016-07-19 02:26 by steve.dower. This issue is now closed.

Messages (14)
msg258954 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-26 13:09
It looks like the buildbot started to fail with the build http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7166/

The change 37dc870175be43cdcb6920cc642a99fc10cd4bb4 of issue #25850 changed Lib/distutils/_msvccompiler.py: "Use cross-compilation by default for 64-bit Windows.". It's an interesting suspect.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7214/steps/test/logs/stdio

test test_distutils failed

======================================================================
ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\_msvccompiler.py", line 480, in link
    self.spawn([self.linker] + ld_args)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\_msvccompiler.py", line 503, in spawn
    return super().spawn(cmd)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\spawn.py", line 38, in spawn
    _spawn_nt(cmd, search_path, dry_run=dry_run)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\spawn.py", line 81, in _spawn_nt
    "command %r failed with exit status %d" % (cmd, rc))
distutils.errors.DistutilsExecError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1101

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\tests\test_install.py", line 215, in test_record_extensions
    cmd.run()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\install.py", line 539, in run
    self.run_command('build')
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 338, in run
    self.build_extensions()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 447, in build_extensions
    self._build_extensions_serial()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 472, in _build_extensions_serial
    self.build_extension(ext)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 557, in build_extension
    target_lang=language)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\ccompiler.py", line 717, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\_msvccompiler.py", line 483, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1101
msg258956 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-26 13:11
Same issue on Windows 8 and Windows 10 buildbots:

* http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/1716/steps/test/logs/stdio
* http://buildbot.python.org/all/builders/AMD64%20Windows10%203.x/builds/598/steps/test/logs/stdio
msg258961 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-26 17:11
So the more interesting error message is higher up:

xxmodule.c
xxmodule.obj : warning LNK4197: export 'PyInit_xx' specified multiple times; using first specification
   Creating library d:\temp\tmptzty591d\Debug\temp\tmptzty591d\xx_d.cp36-win_amd64.lib and object d:\temp\tmptzty591d\Debug\temp\tmptzty591d\xx_d.cp36-win_amd64.exp
LINK : /LTCG specified but no code generation required; remove /LTCG from the link command line to improve linker performance
LINK : fatal error LNK1101: incorrect MSPDB140.DLL version; recheck installation of this product

It's entirely possible that this was an issue with VS 2015 that has been resolved in VS 2015 Update 1. Let me check with some people who may know.

In the meantime, it would be handy if the owners could confirm that the buildbots are running VS 2015 and not Update 1 (released in November, so if you haven't installed anything that recently then it's unrelated). Also the full version numbers of the following files:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\mspdb140.dll
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\link.exe
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_arm\link.exe
msg258963 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-01-26 17:32
Steve, I'm assuming you meant "x86_amd64" rather than "x86_arm" on that last filename?

Note that this issue is restricted to Debug builds (as suggested by the issue being with mspdb140.dll), the Non-Debug buildbot is not failing.
msg258966 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-26 19:11
Yep, that's what I meant. So many cross compilers these days...
msg258967 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-26 19:22
> It's entirely possible that this was an issue with VS 2015 that has been resolved in VS 2015 Update 1. Let me check with some people who may know.

If a fix cannot be found quickly, would it be possible to revert the Python change until VS is fixed and the fix is deployed to all buildbots? I'm trying to keep green buildbots. It helps me to validate my work on multiple platforms ;)
msg258969 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-26 19:29
It would be easier to just temporary skip some tests on particular platforms.
msg258974 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-26 21:36
Serhiy Storchaka added the comment:
> It would be easier to just temporary skip some tests on particular platforms.

Ah yes, I also like this option. Well, it's up to you Steve :-)
msg259160 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-28 21:31
This has been confirmed as a bug in VS 2015 RTM that was fixed in Update 1.

It only affects debug builds, which are not common and I expect most people using them to be keeping VS up to date.

I'd rather not add extra logic to distutils just to detect this case. Ideally, the buildbots should be updated to VS 2015 Update 1 at some point.
msg259173 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-28 23:09
> I'd rather not add extra logic to distutils just to detect this case.

Hum, does it make sense to skip some tests on VS 2015 without the
update? (Is it difficult to detect VS version?)
msg259176 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-28 23:54
For this particular issue, the way to detect it is to read HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\DevDiv\vc\Servicing\14.0\Tools.X86.X64@UpdateVersion and see if it is at least "14.0.23506".

This doesn't extend to other VS versions, but it should be sufficient to skip the tests for this bug. I really don't want that check in the actual stdlib.
msg259180 - (view) Author: Jeremy Kloth (jkloth) * Date: 2016-01-29 00:19
I'm upgrading the VS installation on my buildbot currently so this should be a non-issue going forward.

However, I think it would be good to add/change the bug title to include something about the /LTCG error so as to enable better searching when developers do encounter this error in their own projects.
msg259183 - (view) Author: Jeremy Kloth (jkloth) * Date: 2016-01-29 01:33
My buildbot updated to VS2015.1 successfully.  test_distutils is passing again.

As to the bug title, it should mention the incorrect DLL version error, not the /LTCG *warning*.  That's what I get for just doing a quick scan of the comments.
msg270803 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-07-19 02:26
We're a couple more updates along and it all seems good.
History
Date User Action Args
2016-07-19 02:26:35steve.dowersetstatus: open -> closed
resolution: out of date
messages: + msg270803

stage: resolved
2016-01-29 16:31:16steve.dowersettitle: test_distutils fails on "AMD64 Windows7 SP1 3.x" buildbot -> distutils msvccompiler fails due to mspdb140.dll error on debug builds
2016-01-29 01:33:29jklothsetmessages: + msg259183
2016-01-29 00:19:28jklothsetmessages: + msg259180
2016-01-28 23:54:30steve.dowersetmessages: + msg259176
2016-01-28 23:09:54vstinnersetmessages: + msg259173
2016-01-28 21:31:53steve.dowersetmessages: + msg259160
2016-01-26 21:36:27vstinnersetmessages: + msg258974
2016-01-26 19:29:11serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg258969
2016-01-26 19:22:30vstinnersetmessages: + msg258967
2016-01-26 19:11:53steve.dowersetmessages: + msg258966
2016-01-26 17:32:17zach.waresetmessages: + msg258963
2016-01-26 17:11:03steve.dowersetassignee: steve.dower
messages: + msg258961
2016-01-26 13:52:00jklothsetnosy: + jkloth
2016-01-26 13:11:06vstinnersetmessages: + msg258956
2016-01-26 13:09:42vstinnercreate