On the AMD64 Windows10 2.7 buildbot, the Python compilation fails on mt.exe crashing with error code C0000005 (code -1073741819). It is crashing since the build 279 (Mon Sep 4 2017).

This build was trigged by a single change: the commit 986b7ff (bpo-30450):

"[2.7] bpo-30450: Pull Windows dependencies from GitHub rather than SVN (GH-1783) (GH-3306)"

mt.exe: "The Mt.exe file is a tool that generates signed files and catalogs. It is available in the Microsoft Windows Software Development Kit (SDK). Mt.exe requires that the file referenced in the manifest be present in the same directory as the manifest."

Error C0000005: "Windows System error (...) Exception Error that occurs when a process (application, driver, etc.) tries to execute a processor command that references memory outside the space allocated to that process by the Operating System."

http://buildbot.python.org/all/builders/AMD64%20Windows10%202.7/builds/279/steps/compile/logs/stdio

(...)
"D:\buildarea\2.7.bolen-windows10\build\PCbuild\obj\amd64_Debug\pythoncore\symtable.obj"
"D:\buildarea\2.7.bolen-windows10\build\PCbuild\obj\amd64_Debug\pythoncore\sysmodule.obj"
"D:\buildarea\2.7.bolen-windows10\build\PCbuild\obj\amd64_Debug\pythoncore\thread.obj"
"D:\buildarea\2.7.bolen-windows10\build\PCbuild\obj\amd64_Debug\pythoncore\traceback.obj"
"D:\buildarea\2.7.bolen-windows10\build\PCbuild\obj\amd64_Debug\pythoncore\getbuildinfo.obj"
Creating library D:\buildarea\2.7.bolen-windows10\build\PCBuild\\amd64\python27_d.lib and object D:\buildarea\2.7.bolen-windows10\build\PCBuild\\amd64\python27_d.exp
pythoncore.vcxproj -> D:\buildarea\2.7.bolen-windows10\build\PCBuild\\amd64\python27_d.dll
Manifest:
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\mt.exe /nologo /verbose /outputresource:"D:\buildarea\2.7.bolen-windows10\build\PCBuild\\amd64\python27_d.dll;#2" /manifest "D:\buildarea\2.7.bolen-windows10\build\PCbuild\obj\amd64_Debug\pythoncore\python27_d.dll.intermediate.manifest"
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppCommon.targets(574,5): error MSB6006: "mt.exe" exited with code -1073741819. [D:\buildarea\2.7.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
Done Building Project "D:\buildarea\2.7.bolen-windows10\build\PCbuild\pythoncore.vcxproj" (Build target(s)) -- FAILED.
The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the project, and will be ignored.
The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the project, and will be ignored.
Done Building Project "D:\buildarea\2.7.bolen-windows10\build\PCbuild\pcbuild.proj" (Build target(s)) -- FAILED.
(...)

Any update on this bug?

The "AMD64 Windows10 2.7" buildbot is still broken:
http://buildbot.python.org/all/#/builders/96

I believe I've identified the issue, and put a workaround in place.

The 986b7ff commit somehow affected which binary of mt.exe is being run (for version 5.2.3790.2076, 2005), changing from:
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\x64\mt.exe
to
C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin\mt.exe
The 32-bit version is what is crashing, at least when trying to add the manifest to a 64-bit library (it does work for other manifest operations).

I thought the binary was just incompatible, but I located a later version of mt (6.3.9600.17336, 2012) under C:\Program Files (x86)\Windows Kits\8.1 for which both the 32-bit and 64-bit versions worked. Of course, there was also the issue of why Win8 was working with the 5.2 32-bit version.

Then I noticed a .NET config file for the newer 6.3 mt to handle running on systems with .NET 4 but not 1.1 or 2. As it turns out, the Win10 buildbot is .NET 4+ only, while the older 8.1 buildbot has frameworks back to .NET 2. So that's probably what kept the Win8 buildbot from failing just like Win10.

What I did for now is borrow the .NET exe config files from mt 6.3 for use with the SDK mt 5.2 binaries, which looks like it's done the trick.

It might be best to restore the old behavior if possible so building works without the workaround, but I'm not sure how the binary is being determined. None of the mt binaries are in the default path so must be found by the build process somehow. Reverting to using env.bat with the latest source (so msbuild is found on the path) still ends up calling the 32-bit mt. So something else in the changes must be influencing the binary choice. And given that this only affects using the scripts to build 2.7 on a recently installed machine, hard to say if it's worth much effort.

Are you able to install .NET 3.5 on the Windows 10 machine? This is basically a requirement for the VC9 toolset (it's compatible with .NET 2.0 so should fulfil that requirement).

Sure, I can certainly do that.

Does "basically a requirement" mean it should have been there all along (with the VC9 installation), or just that trying to use VC9 on a recent system like Win 10 safely requires it installed separately? I guess Win 8 was the first version to default to only having 4+ already installed.

Oh, any preference between using the OS component setup (windows features) to install vs. downloading an installer? I was thinking of using the features approach since it's easy to document as a requirement for anyone else. I doubt there's much difference in the end result.

I don't remember exactly how VS 2008 handled the installation, but it likely wasn't designed with Windows 8 in mind :)

I believe since Windows 8, the Windows Features approach is the correct way to install it, and I'm not even sure you can download installers any more. Certainly the docs [1] do not link anywhere.

1: https://docs.microsoft.com/en-us/dotnet/framework/install/dotnet-35-windows-10

Sounds good. I must admit I hadn't actually gone looking for a standalone installer yet, but just assumed it would exist.

The Win10 worker now has the .NET 3.5 feature installed (which also installed 2.0 and 3.0). Builds seem fine even after removing the temporary exe config file I had put in place.

So I think we should be good for this issue.

The worker is happy, closing the issue.

Thank you!