classification
Title: Master build failure on Windows getting file system encoding
Type: compile error Stage: resolved
Components: Build, Windows Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Dennis Sweeney, db3l, pablogsal, paul.moore, steve.dower, terry.reedy, tim.golden, vstinner, zach.ware
Priority: critical Keywords: 3.10regression

Created on 2021-04-03 20:15 by terry.reedy, last changed 2021-04-04 12:28 by pablogsal. This issue is now closed.

Messages (10)
msg390140 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-03 20:15
After a fresh update from upstream
> PCbuild\build.bat -D (or without -D)
on my machine ends with

  python.vcxproj -> f:\dev\3x\PCbuild\amd64\python_d.pdb (Full PDB)
  Python path configuration:
    PYTHONHOME = (not set)
    PYTHONPATH = 'f:\dev\3x\Lib'
    program name = 'f:\dev\3x\PCbuild\amd64\python_d.exe'
    isolated = 0
    environment = 1
    user site = 1
    import site = 1
    sys._base_executable = 'f:\\dev\\3x\\PCbuild\\amd64\\python_d.exe'
    sys.base_prefix = 'f:\\dev\\3x'
    sys.base_exec_prefix = 'f:\\dev\\3x'
    sys.platlibdir = 'lib'
    sys.executable = 'f:\\dev\\3x\\PCbuild\\amd64\\python_d.exe'
    sys.prefix = 'f:\\dev\\3x'
    sys.exec_prefix = 'f:\\dev\\3x'
    sys.path = [
      'f:\\dev\\3x\\Lib',
      'f:\\dev\\3x\\PCbuild\\amd64\\python310_d.zip',
      'f:\\dev\\3x\\DLLs',
      'f:\\dev\\3x\\lib',
      'f:\\dev\\3x\\PCbuild\\amd64',
    ]
EXEC : Fatal Python warning : init_fs_encoding: failed to get the Python codec
of the filesystem encoding [f:\dev\3x\PCbuild\python.vcxproj]
  Python runtime state: core initialized
  Traceback (most recent call last):
    File "f:\dev\3x\Lib\encodings\__init__.py", line 31, in <module>
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: The command "setlocal
\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: set PYTHONPATH=f:\dev
\3x\Lib\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: "f:\dev\3x\PCbuild\am
d64\python_d.exe" "f:\dev\3x\PC\validate_ucrtbase.py" ucrtbased" exited with co
de -1073741819.
  pythonw.vcxproj -> f:\dev\3x\PCbuild\amd64\pythonw_d.exe
  pythonw.vcxproj -> f:\dev\3x\PCbuild\amd64\pythonw_d.pdb (Full PDB)

EXEC : Fatal Python warning : init_fs_encoding: failed to get the Python codec
of the filesystem encoding [f:\dev\3x\PCbuild\python.vcxproj]
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: The command "setlocal
\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: set PYTHONPATH=f:\dev
\3x\Lib\r
f:\dev\3x\PCbuild\python.vcxproj(123,5): warning MSB3073: "f:\dev\3x\PCbuild\am
d64\python_d.exe" "f:\dev\3x\PC\validate_ucrtbase.py" ucrtbased" exited with co
de -1073741819.

I believe I pulled and rebuilt fine just a few days ago. Subsequently running
> python
ends with

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
Traceback (most recent call last):
  File "f:\dev\3x\\lib\encodings\__init__.py", line 31, in <module>

Running > chkdsk F: /scan revealed no obvious disk error, but there could be corrupted bytes in some file, but I don't know what file(s) to check (if text) or delete and have git redownload.  Anyone else have a similar problem?
msg390144 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-03 21:46
Re ""f:\dev\3x\PC\validate_ucrtbase.py" ucrtbased" exited with co
de -1073741819."

f:\dev\3x>py -3.10 PC\validate_ucrtbase.py ucrtbased
C:\WINDOWS\SYSTEM32\ucrtbased.DLL is version 10.0.14393.33

so the failure is due to running the file with the nonfunctional python_d.exe.

In a text editor, encodings/__init__.py looks fine.
f:\dev\3x>py -3.10 f:\dev\3x\\lib\encodings\__init__.py
(to check text further) fails at line 33
  from . import alias

In any case, line 31 is 'import codecs'.  codecs.py looks fine and
f:\dev\3x>py -3.10 f:\dev\3x\\lib\codecs.py
runs without error, so its text appears to be intact.  Perhaps one of the imports of compiled C fail drastically.
msg390150 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-03 23:14
IMO it's https://bugs.python.org/issue27129#msg389991
msg390153 - (view) Author: David Bolen (db3l) * Date: 2021-04-03 23:30
Terry, it's not clear to me if this is the same issue (at first blush it appears different) but could you see if you have any pyc files in __pycache__ folders within the Tools or Parser directories in your tree that could be from prior builds?  If so, remove them and try a new build?

The build.bat script's clean process is not currently cleaning them up, which can lead to various oddities if you reuse a source tree across the commit Victor references, as that invalidates the older files but they aren't regenerated.
msg390155 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-04-03 23:46
I ran into the same issue on Windows 10 and git bisected it to the same first bad commit, fcb55c0037baab6f98f91ee38ce84b6f874f034a

The issue persists after rm .\Parser\__pycache__\*
My .\Tools directory has no __pycache__.
msg390156 - (view) Author: David Bolen (db3l) * Date: 2021-04-03 23:51
Dennis, just to make sure I wasn't too literal, some of the __pycache__ folders are in subdirectories (at least in Tools), so just to double check, you did a recursive search beneath Tools and Parser right?
msg390162 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-04-04 00:33
I could not successfully build even with deletion of __pycache__ in subfolders. I finally got the build to succeed after changing the magic number, so I opened GH-25069.
msg390163 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-04-04 00:34
Correction: I opened GH-25172
msg390167 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-04 02:14
Thanks everyone for the rapid responses.  The PR fixes the build for me, with a clean test suite pass, so I added reviewers who could merge, a blurb, and my approval.  This can be closed as out-of-date as soon as the (or a) fix is merged.
msg390185 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-04-04 12:28
Fixed by https://github.com/python/cpython/pull/25172
History
Date User Action Args
2021-04-04 12:28:02pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg390185

stage: resolved
2021-04-04 02:14:27terry.reedysetpriority: normal -> critical
2021-04-04 02:14:08terry.reedysetmessages: + msg390167
2021-04-04 00:34:06Dennis Sweeneysetmessages: + msg390163
2021-04-04 00:33:20Dennis Sweeneysetmessages: + msg390162
2021-04-03 23:51:05db3lsetmessages: + msg390156
2021-04-03 23:46:01Dennis Sweeneysetnosy: + Dennis Sweeney
messages: + msg390155
2021-04-03 23:30:09db3lsetnosy: + db3l
messages: + msg390153
2021-04-03 23:14:21vstinnersetmessages: + msg390150
2021-04-03 21:46:15terry.reedysetmessages: + msg390144
2021-04-03 20:15:48terry.reedycreate