classification
Title: Dump the Python path configuration at the first import error
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, matrixise, ncoghlan, pablogsal, steve.dower, vstinner
Priority: normal Keywords: patch

Created on 2019-09-20 16:44 by vstinner, last changed 2019-09-23 14:01 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 16300 merged vstinner, 2019-09-20 17:07
PR 16332 merged vstinner, 2019-09-23 12:57
PR 16333 merged vstinner, 2019-09-23 13:13
Messages (8)
msg352869 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-20 16:44
When Python path configuration is not properly configured, Python fails with an error looking like:
---
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff0065b2740 (most recent call first):
<no Python frame>
---

This error is very common and hard to debug. It can happen in various cases for various reasons:

* Python is embedded in an application
* PYTHONPATH and/or PYTHONHOME environment variable is set
* pyenv.cfg configuration for virtual environment
* Local .py files which override the standard library

A few examples:

* bpo-38234: Py_SetPath() regression in Python 3.8 which impacts PyInstaller
* https://bugs.python.org/issue35978#msg335363 virtual env
* https://bugs.python.org/issue35969#msg335249 "encodings.py" file in the current directory
* https://stackoverflow.com/questions/55657593/python-3-8-0a3-failed-to-get-the-python-codec-of-the-filesystem-encoding embedded Python
* https://askubuntu.com/questions/640010/fatal-python-error-py-initialize-unable-to-get-the-locale-encoding-syntaxe Python 3 installed by Anaconda which uses Python 2.7 stdlib because of PYTHONPATH
* https://github.com/GNS3/gns3-gui/issues/2267 Python embedded in "gns3"
* https://github.com/JuliaPy/PyCall.jl/issues/410 venv
* https://techglimpse.com/error-executing-python3-5-command-solution/ another random error

Attached PR dumps the Python path configuration if importing the Python codec of the filesystem encoding fails. In the Python initialization, it's the first import which is done.
msg352873 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-20 17:08
Without my change:

---
vstinner@apu$ PYTHONHOME=/xxx ./python 
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fa9c32d6740 (most recent call first):
<no Python frame>
---


With my PR 16300:

---
$ PYTHONHOME=/xxx ./python 
Python path configuration:
  PYTHONHOME = '/xxx'
  PYTHONPATH = '/usr/share/qa-tools/python-modules'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys.executable = '/home/vstinner/python/master/python'
  sys.prefix = '/xxx'
  sys.exec_prefix = '/xxx'
  sys.path = [
    '/usr/share/qa-tools/python-modules',
    '/xxx/lib/python39.zip',
    '/xxx/lib/python3.9',
    '/xxx/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f9cef949740 (most recent call first):
<no Python frame>
---
msg352874 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-20 17:13
>   user site = 1
>   import site = 1

I'm not sure that dumping these configuration variables are useful to debug this very specific error:

"Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding"

They only have an effect on sys.path later.
msg352925 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-09-21 05:39
+1
msg353013 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-23 12:46
New changeset fcdb027234566c4d506d6d753c7d5638490fb088 by Victor Stinner in branch 'master':
bpo-38236: Dump path config at first import error (GH-16300)
https://github.com/python/cpython/commit/fcdb027234566c4d506d6d753c7d5638490fb088
msg353015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-23 13:35
New changeset 88e6447451fb5525e83e802c66c3e51b4a45bf86 by Victor Stinner in branch 'master':
bpo-38236: Fix init_dump_ascii_wstr() (GH-16333)
https://github.com/python/cpython/commit/88e6447451fb5525e83e802c66c3e51b4a45bf86
msg353016 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-23 13:59
New changeset c5c642565e260477ae2fb29d0c86a91e19702ae3 by Victor Stinner in branch '3.8':
bpo-38236: Dump path config at first import error (GH-16300) (GH-16332)
https://github.com/python/cpython/commit/c5c642565e260477ae2fb29d0c86a91e19702ae3
msg353017 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-23 14:01
I backported my change to Python 3.8.

Example with PYTHONPATH set to Python 2.7 with Python 3.8:

$ PYTHONPATH=/usr/lib64/python2.7/ ./python -c pass
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/usr/lib64/python2.7/'
  program name = './python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/vstinner/python/3.8/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.executable = '/home/vstinner/python/3.8/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/lib64/python2.7/',
    '/usr/local/lib/python38.zip',
    '/home/vstinner/python/3.8/Lib',
    '/home/vstinner/python/3.8/build/lib.linux-x86_64-3.8-pydebug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
  File "/usr/lib64/python2.7/encodings/__init__.py", line 123
    raise CodecRegistryError,\
                            ^
SyntaxError: invalid syntax

Current thread 0x00007fe3ade4d740 (most recent call first):
<no Python frame>
History
Date User Action Args
2019-09-23 14:01:37vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-23 14:01:26vstinnersetmessages: + msg353017
versions: + Python 3.8
2019-09-23 13:59:08vstinnersetmessages: + msg353016
2019-09-23 13:36:00vstinnersetmessages: + msg353015
2019-09-23 13:13:40vstinnersetpull_requests: + pull_request15910
2019-09-23 12:57:52vstinnersetpull_requests: + pull_request15909
2019-09-23 12:46:03vstinnersetmessages: + msg353013
2019-09-21 05:39:55matrixisesetnosy: + matrixise
messages: + msg352925
2019-09-20 17:13:10vstinnersetmessages: + msg352874
2019-09-20 17:08:39vstinnersetmessages: + msg352873
2019-09-20 17:07:19vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request15886
2019-09-20 16:44:55vstinnercreate