It looks like the buildbot started to fail with the build http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7166/

The change 37dc870175be43cdcb6920cc642a99fc10cd4bb4 of issue #25850 changed Lib/distutils/_msvccompiler.py: "Use cross-compilation by default for 64-bit Windows.". It's an interesting suspect.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7214/steps/test/logs/stdio

test test_distutils failed

======================================================================
ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\_msvccompiler.py", line 480, in link
    self.spawn([self.linker] + ld_args)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\_msvccompiler.py", line 503, in spawn
    return super().spawn(cmd)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\spawn.py", line 38, in spawn
    _spawn_nt(cmd, search_path, dry_run=dry_run)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\spawn.py", line 81, in _spawn_nt
    "command %r failed with exit status %d" % (cmd, rc))
distutils.errors.DistutilsExecError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1101

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\tests\test_install.py", line 215, in test_record_extensions
    cmd.run()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\install.py", line 539, in run
    self.run_command('build')
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\dist.py", line 974, in run_command
    cmd_obj.run()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 338, in run
    self.build_extensions()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 447, in build_extensions
    self._build_extensions_serial()
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 472, in _build_extensions_serial
    self.build_extension(ext)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\command\build_ext.py", line 557, in build_extension
    target_lang=language)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\ccompiler.py", line 717, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\distutils\_msvccompiler.py", line 483, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1101

Same issue on Windows 8 and Windows 10 buildbots:

* http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/1716/steps/test/logs/stdio
* http://buildbot.python.org/all/builders/AMD64%20Windows10%203.x/builds/598/steps/test/logs/stdio

So the more interesting error message is higher up:

xxmodule.c
xxmodule.obj : warning LNK4197: export 'PyInit_xx' specified multiple times; using first specification
   Creating library d:\temp\tmptzty591d\Debug\temp\tmptzty591d\xx_d.cp36-win_amd64.lib and object d:\temp\tmptzty591d\Debug\temp\tmptzty591d\xx_d.cp36-win_amd64.exp
LINK : /LTCG specified but no code generation required; remove /LTCG from the link command line to improve linker performance
LINK : fatal error LNK1101: incorrect MSPDB140.DLL version; recheck installation of this product

It's entirely possible that this was an issue with VS 2015 that has been resolved in VS 2015 Update 1. Let me check with some people who may know.

In the meantime, it would be handy if the owners could confirm that the buildbots are running VS 2015 and not Update 1 (released in November, so if you haven't installed anything that recently then it's unrelated). Also the full version numbers of the following files:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\mspdb140.dll
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\link.exe
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_arm\link.exe

Steve, I'm assuming you meant "x86_amd64" rather than "x86_arm" on that last filename?

Note that this issue is restricted to Debug builds (as suggested by the issue being with mspdb140.dll), the Non-Debug buildbot is not failing.

Yep, that's what I meant. So many cross compilers these days...

> It's entirely possible that this was an issue with VS 2015 that has been resolved in VS 2015 Update 1. Let me check with some people who may know.

If a fix cannot be found quickly, would it be possible to revert the Python change until VS is fixed and the fix is deployed to all buildbots? I'm trying to keep green buildbots. It helps me to validate my work on multiple platforms ;)

It would be easier to just temporary skip some tests on particular platforms.

Serhiy Storchaka added the comment:
> It would be easier to just temporary skip some tests on particular platforms.

Ah yes, I also like this option. Well, it's up to you Steve :-)

This has been confirmed as a bug in VS 2015 RTM that was fixed in Update 1.

It only affects debug builds, which are not common and I expect most people using them to be keeping VS up to date.

I'd rather not add extra logic to distutils just to detect this case. Ideally, the buildbots should be updated to VS 2015 Update 1 at some point.

> I'd rather not add extra logic to distutils just to detect this case.

Hum, does it make sense to skip some tests on VS 2015 without the
update? (Is it difficult to detect VS version?)

For this particular issue, the way to detect it is to read HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\DevDiv\vc\Servicing\14.0\Tools.X86.X64@UpdateVersion and see if it is at least "14.0.23506".

This doesn't extend to other VS versions, but it should be sufficient to skip the tests for this bug. I really don't want that check in the actual stdlib.

I'm upgrading the VS installation on my buildbot currently so this should be a non-issue going forward.

However, I think it would be good to add/change the bug title to include something about the /LTCG error so as to enable better searching when developers do encounter this error in their own projects.

My buildbot updated to VS2015.1 successfully.  test_distutils is passing again.

As to the bug title, it should mention the incorrect DLL version error, not the /LTCG *warning*.  That's what I get for just doing a quick scan of the comments.

We're a couple more updates along and it all seems good.