classification
Title: Windows builds sometimes fail on Azure and GitHub Action: fatal error RC1116: RC terminating after preprocessor errors
Type: compile error Stage: resolved
Components: Build, Windows Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, lukasz.langa, miss-islington, paul.moore, serhiy.storchaka, sobolevn, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2021-09-16 14:45 by sobolevn, last changed 2021-09-29 13:54 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28393 merged steve.dower, 2021-09-16 17:01
PR 28394 merged miss-islington, 2021-09-16 18:29
PR 28395 merged miss-islington, 2021-09-16 18:29
PR 28622 merged miss-islington, 2021-09-29 13:37
Messages (14)
msg401948 - (view) Author: Nikita Sobolev (sobolevn) * Date: 2021-09-16 14:45
I've started to notice that CPython's builds on Windows now simetimes fail with something like this:

(both Azure and Github Actions are affected)

```
Using "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe" (found in the Visual Studio installation)
Using py -3.9 (found 3.9 with py.exe)

D:\a\cpython\cpython>"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe" "D:\a\cpython\cpython\PCbuild\pcbuild.proj" /t:Build /m /nologo /v:m /clp:summary /p:Configuration=Release /p:Platform=Win32 /p:IncludeExternals=true /p:IncludeCTypes=true /p:IncludeSSL=true /p:IncludeTkinter=true /p:UseTestMarker= /p:GIT="C:\Program Files\Git\bin\git.exe"          
  _freeze_module.c
  config_minimal.c
  atexitmodule.c
  faulthandler.c
  gcmodule.c
  getbuildinfo.c
  posixmodule.c
  signalmodule.c
  _tracemalloc.c
  _iomodule.c
  bufferedio.c
  bytesio.c
  fileio.c
  iobase.c
  stringio.c
  textio.c
  winconsoleio.c
  abstract.c
  accu.c
  boolobject.c
  Compiling...
  bytearrayobject.c
  bytes_methods.c
  bytesobject.c
  call.c
  capsule.c
  cellobject.c
  classobject.c
  codeobject.c
  complexobject.c
  descrobject.c
  dictobject.c
  enumobject.c
  exceptions.c
  fileobject.c
  floatobject.c
  frameobject.c
  funcobject.c
  genericaliasobject.c
  genobject.c
  interpreteridobject.c
  Compiling...
  iterobject.c
  listobject.c
  longobject.c
  memoryobject.c
  methodobject.c
  moduleobject.c
  namespaceobject.c
  object.c
  obmalloc.c
  odictobject.c
  picklebufobject.c
  rangeobject.c
  setobject.c
  sliceobject.c
  structseq.c
  tupleobject.c
  typeobject.c
  unicodectype.c
  unicodeobject.c
  unionobject.c
  Compiling...
  weakrefobject.c
  myreadline.c
  parser.c
  peg_api.c
  pegen.c
  string_parser.c
  token.c
  tokenizer.c
  getpathp.c
  invalid_parameter_handler.c
  msvcrtmodule.c
  winreg.c
  _warnings.c
  asdl.c
  ast.c
  ast_opt.c
  ast_unparse.c
  bltinmodule.c
  bootstrap_hash.c
  ceval.c
D:\a\cpython\cpython\Python\ceval.c(3669,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
D:\a\cpython\cpython\Python\ceval.c(3777,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
  Compiling...
  codecs.c
  compile.c
  context.c
  dtoa.c
  dynamic_annotations.c
  dynload_win.c
  errors.c
  fileutils.c
  formatter_unicode.c
  frame.c
  future.c
  getargs.c
  getcompiler.c
  getcopyright.c
  getopt.c
  getplatform.c
  getversion.c
  hamt.c
  hashtable.c
  import.c
  Compiling...
  importdl.c
  initconfig.c
  marshal.c
  modsupport.c
  mysnprintf.c
  mystrtoul.c
  pathconfig.c
  preconfig.c
  pyarena.c
  pyctype.c
  pyfpe.c
  pyhash.c
  pylifecycle.c
  pymath.c
  pystate.c
  pystrcmp.c
  pystrhex.c
  pystrtod.c
  Python-ast.c
  pythonrun.c
  Compiling...
  Python-tokenize.c
  pytime.c
  specialize.c
  structmember.c
  suggestions.c
  symtable.c
  sysmodule.c
  thread.c
  traceback.c
     Creating library D:\a\cpython\cpython\PCbuild\win32\_freeze_module.lib and object D:\a\cpython\cpython\PCbuild\win32\_freeze_module.exp
  Generating code
  Finished generating code
  _freeze_module.vcxproj -> D:\a\cpython\cpython\PCbuild\win32\_freeze_module.exe
  Updated files: __hello__.h
  Killing any running python.exe instances...
  Regenerate pycore_ast.h pycore_ast_state.h Python-ast.c
  D:\a\cpython\cpython\Python\Python-ast.c, D:\a\cpython\cpython\Include\internal\pycore_ast.h, D:\a\cpython\cpython\Include\internal\pycore_ast_state.h regenerated.
  Regenerate opcode.h opcode_targets.h
  Include\opcode.h regenerated from Lib\opcode.py
  Jump table written into Python\opcode_targets.h
  Regenerate token-list.inc token.h token.c token.py
  Generated sources are up to date
  Getting build info from "C:\Program Files\Git\bin\git.exe"
  Building heads/main-dirty:7dacb70 main
  _abc.c
  _bisectmodule.c
  blake2module.c
  blake2b_impl.c
  blake2s_impl.c
  _codecsmodule.c
  _collectionsmodule.c
  _contextvarsmodule.c
  _csv.c
  _functoolsmodule.c
  _heapqmodule.c
  _json.c
  _localemodule.c
  _lsprof.c
  _math.c
  _pickle.c
  _randommodule.c
  sha3module.c
  _sre.c
  _stat.c
  Compiling...
  _struct.c
  _weakref.c
  arraymodule.c
  atexitmodule.c
  audioop.c
  binascii.c
  cmathmodule.c
  _datetimemodule.c
  errnomodule.c
  faulthandler.c
  gcmodule.c
  itertoolsmodule.c
  main.c
  mathmodule.c
  md5module.c
  mmapmodule.c
  _opcode.c
  _operator.c
  posixmodule.c
  rotatingtree.c
  Compiling...
  sha1module.c
  sha256module.c
  sha512module.c
  signalmodule.c
  _statisticsmodule.c
  symtablemodule.c
  _threadmodule.c
  _tracemalloc.c
  _typingmodule.c
  timemodule.c
  xxsubtype.c
  _xxsubinterpretersmodule.c
  fileio.c
  bytesio.c
  stringio.c
  bufferedio.c
  iobase.c
  textio.c
  winconsoleio.c
  _iomodule.c
  Compiling...
  _codecs_cn.c
  _codecs_hk.c
  _codecs_iso2022.c
  _codecs_jp.c
  _codecs_kr.c
  _codecs_tw.c
  multibytecodec.c
  _winapi.c
  abstract.c
  accu.c
  boolobject.c
  bytearrayobject.c
  bytes_methods.c
  bytesobject.c
  call.c
  capsule.c
  cellobject.c
  classobject.c
  codeobject.c
  complexobject.c
  Compiling...
  descrobject.c
  dictobject.c
  enumobject.c
  exceptions.c
  fileobject.c
  floatobject.c
  frameobject.c
  funcobject.c
  genericaliasobject.c
  genobject.c
  interpreteridobject.c
  iterobject.c
  listobject.c
  longobject.c
  memoryobject.c
  methodobject.c
  moduleobject.c
  namespaceobject.c
  object.c
  obmalloc.c
  Compiling...
  odictobject.c
  picklebufobject.c
  rangeobject.c
  setobject.c
  sliceobject.c
  structseq.c
  tupleobject.c
  typeobject.c
  unicodectype.c
  unicodeobject.c
  unionobject.c
  weakrefobject.c
  myreadline.c
  tokenizer.c
  token.c
  pegen.c
  parser.c
  string_parser.c
  peg_api.c
  invalid_parameter_handler.c
  Compiling...
  winreg.c
  config.c
  getpathp.c
  msvcrtmodule.c
  pyhash.c
  _warnings.c
  asdl.c
  ast.c
  ast_opt.c
  ast_unparse.c
  bltinmodule.c
  bootstrap_hash.c
  ceval.c
D:\a\cpython\cpython\Python\ceval.c(3669,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
D:\a\cpython\cpython\Python\ceval.c(3777,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
  codecs.c
  compile.c
  context.c
  dynamic_annotations.c
  dynload_win.c
  errors.c
  fileutils.c
  Compiling...
  formatter_unicode.c
  frame.c
  frozen.c
  future.c
  getargs.c
  getcompiler.c
  getcopyright.c
  getopt.c
  getplatform.c
  getversion.c
  hamt.c
  hashtable.c
  import.c
  importdl.c
  initconfig.c
  marshal.c
  modsupport.c
  mysnprintf.c
  mystrtoul.c
  pathconfig.c
  Compiling...
  preconfig.c
  pyarena.c
  pyctype.c
  pyfpe.c
  pylifecycle.c
  pymath.c
  pytime.c
  pystate.c
  pystrcmp.c
  pystrhex.c
  pystrtod.c
  dtoa.c
  Python-ast.c
  Python-tokenize.c
  pythonrun.c
  specialize.c
  suggestions.c
  structmember.c
  symtable.c
  sysmodule.c
  Compiling...
  thread.c
  traceback.c
  zlibmodule.c
  adler32.c
  compress.c
  crc32.c
  deflate.c
  infback.c
  inffast.c
  inflate.c
  inftrees.c
  trees.c
  uncompr.c
  zutil.c
  dl_nt.c
  getbuildinfo.c
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winnt.h(253): error RC2188: D:\a\cpython\cpython\PCbuild\obj\311win32_Release\pythoncore\RCa05056(47) : fatal error RC1116: RC terminating after preprocessor errors [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
  

Build FAILED.

D:\a\cpython\cpython\Python\ceval.c(3669,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
D:\a\cpython\cpython\Python\ceval.c(3777,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
D:\a\cpython\cpython\Python\ceval.c(3669,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
D:\a\cpython\cpython\Python\ceval.c(3777,13): warning C4018: '>=': signed/unsigned mismatch [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winnt.h(253): error RC2188: D:\a\cpython\cpython\PCbuild\obj\311win32_Release\pythoncore\RCa05056(47) : fatal error RC1116: RC terminating after preprocessor errors [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
    4 Warning(s)
    1 Error(s)
```

Links:
- https://github.com/python/cpython/runs/3620202822
- https://dev.azure.com/Python/cpython/_build/results?buildId=87889&view=logs&j=91c152bd-7320-5194-b252-1404e56e2478&t=c7e99cd8-4756-5292-d34b-246ff5fc615f
msg401952 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-16 15:06
It also fails on the 3.9 and 3.10 on the Windows jobs of GitHub Action.
msg401953 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-16 15:10
It may be related to bpo-45020 which recently got a change:

New changeset 9fd87a5fe5c468cf94265365091267838b004b7f by Eric Snow in branch 'main':
bpo-45020: Revert "Drop the frozen .h files from the repo." (gh-28380)
https://github.com/python/cpython/commit/9fd87a5fe5c468cf94265365091267838b004b7f
msg401962 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-09-16 15:44
This is more likely to relate to bpo-45188, "De-couple the Windows builds from freezing modules.", for which the PR was merged yesterday:

New changeset 09b4ad11f323f8702cde795e345b75e0fbb1a9a5 by Steve Dower in branch 'main':
bpo-45188: Windows now regenerates frozen modules at the start of build instead of late (GH-28322)
https://github.com/python/cpython/commit/09b4ad11f323f8702cde795e345b75e0fbb1a9a5

(https://github.com/python/cpython/pull/28322)
msg401963 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-09-16 15:44
Of course, it might also be unrelated.
msg401965 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-16 15:45
> C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winnt.h(253): error RC2188: D:\a\cpython\cpython\PCbuild\obj\311win32_Release\pythoncore\RCa05056(47) : fatal error RC1116: RC terminating after preprocessor errors [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]

Let's split it into sub-parts:

* C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winnt.h(253): error RC2188
* D:\a\cpython\cpython\PCbuild\obj\311win32_Release\pythoncore\RCa05056(47)
* fatal error RC1116: RC terminating after preprocessor errors
* [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]

RCxxxx errors are generated by the resource compiler

> C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winnt.h(253)

What is the code at line 253?

> error RC2188

What is this error?

> 311win32_Release\pythoncore\RCa05056(47)

Is this a directory with a generated name, or is it an error code?

> fatal error RC1116

What is this error?

--

Python contains the following .rc files:

PC/pylauncher.rc
PC/pyshellext.rc
PC/python_exe.rc
PC/python_nt.rc
PC/pythonw_exe.rc
PC/sqlite3.rc

None of these files include winnt.h.

In fact, "winnt.h" cannot be found in any file of the Python source code.

PC/python_nt.rc includes PC/python_ver_rc.h which contains 3 includes:

* #include "winver.h" => Windows SDK header file
* #include "modsupport.h" => Include/modsupport.h
  => it includes #include <stdarg.h>
* #include "patchlevel.h" => Include/patchlevel.h

I'm not sure why modsupport.h is included.

--

I found one page mentioning RC2188 *and* RC1116 error codes together:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/7ba8eb72-12e9-4c78-af68-7f50c170040f/warning-rc4011-and-error-rc2188?forum=vclanguage

"RCxxxx errors are generated by the resource compiler. One of your .rc files includes directly or indirectly headers that RC can't understand so you'll have to check that and remove the #includes."
msg401967 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-16 15:46
> bpo-45188: Windows now regenerates frozen modules at the start of build instead of late (GH-28322)

Maybe the resource compiler needs a header file which is not generated yet, and so the build fails.

In msg401948 logs, I don't see when header files are generated. This build step seems to be fully quiet.
msg401968 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-16 15:47
This issue is serious: it prevents to merge many pull requests, it blocks the Python development workflow.
msg401973 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-16 16:44
The build order change was not backported (and should not be), so it's unrelated to the failures on 3.9 and 3.10.

More likely this is either a problematic compiler update, or some additional data leaking into the build information (e.g. I could see this happening if someone added a UTF-8 copyright symbol somewhere instead of an escaped version).
msg401974 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-16 16:49
> C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\um\winnt.h

This appears to be the Windows 11 preview SDK, so most likely it's a bug in that. We'll obviously need some logic to not automatically upgrade to this SDK on build.
msg401978 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-16 18:29
New changeset f4b94b1f57827083990272b5f282aa1493ae2bf4 by Steve Dower in branch 'main':
bpo-45220: Avoid automatically selecting the Windows 11 SDK preview when building (GH-28393)
https://github.com/python/cpython/commit/f4b94b1f57827083990272b5f282aa1493ae2bf4
msg401980 - (view) Author: miss-islington (miss-islington) Date: 2021-09-16 18:54
New changeset 7ad07eed885ae333d5dfc470b9c11ff2f7b229aa by Miss Islington (bot) in branch '3.9':
bpo-45220: Avoid automatically selecting the Windows 11 SDK preview when building (GH-28393)
https://github.com/python/cpython/commit/7ad07eed885ae333d5dfc470b9c11ff2f7b229aa
msg401984 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-16 19:47
New changeset f798cef8aaaa7faeb428bdb9ebf73df1a6146304 by Miss Islington (bot) in branch '3.10':
bpo-45220: Avoid automatically selecting the Windows 11 SDK preview when building (GH-28393) (GH-28394)
https://github.com/python/cpython/commit/f798cef8aaaa7faeb428bdb9ebf73df1a6146304
msg402874 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-29 13:54
New changeset 456d6d9fefac828c19fc1b562f6301f00965fe59 by Miss Islington (bot) in branch '3.8':
bpo-45220: Avoid automatically selecting the Windows 11 SDK preview when building (GH-28393) (GH-28622)
https://github.com/python/cpython/commit/456d6d9fefac828c19fc1b562f6301f00965fe59
History
Date User Action Args
2021-09-29 13:54:54lukasz.langasetnosy: + lukasz.langa
messages: + msg402874
2021-09-29 13:37:21miss-islingtonsetpull_requests: + pull_request26995
2021-09-16 20:30:10steve.dowersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-09-16 19:47:13serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg401984
2021-09-16 18:54:29miss-islingtonsetmessages: + msg401980
2021-09-16 18:29:46steve.dowersetmessages: + msg401978
2021-09-16 18:29:45miss-islingtonsetpull_requests: + pull_request26807
2021-09-16 18:29:41miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26806
2021-09-16 17:01:49steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request26805
2021-09-16 16:49:31steve.dowersetmessages: + msg401974
2021-09-16 16:45:00steve.dowersetmessages: + msg401973
2021-09-16 15:47:16vstinnersetmessages: + msg401968
2021-09-16 15:46:52vstinnersetmessages: + msg401967
2021-09-16 15:45:06vstinnersetmessages: + msg401965
2021-09-16 15:44:43eric.snowsetmessages: + msg401963
2021-09-16 15:44:02eric.snowsetnosy: + eric.snow
messages: + msg401962
2021-09-16 15:10:35vstinnersetmessages: + msg401953
2021-09-16 15:06:58vstinnersettitle: Windows builds sometimes fail on main branch: fatal error RC1116: RC terminating after preprocessor errors -> Windows builds sometimes fail on Azure and GitHub Action: fatal error RC1116: RC terminating after preprocessor errors
2021-09-16 15:06:39vstinnersettitle: Windows builds sometimes fail on main branch -> Windows builds sometimes fail on main branch: fatal error RC1116: RC terminating after preprocessor errors
2021-09-16 15:06:16vstinnersetnosy: + vstinner

messages: + msg401952
versions: + Python 3.9, Python 3.10
2021-09-16 15:01:19zach.waresetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2021-09-16 14:45:16sobolevncreate