This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: bytes do not work on sys.path
Type: Stage: patch review
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.smith, eryksun, graingert, jaraco
Priority: normal Keywords: patch

Created on 2022-03-15 10:32 by graingert, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
demo.py graingert, 2022-03-15 10:32 reproducer of the failure
zipfile_demo.py graingert, 2022-03-15 11:24
Pull Requests
URL Status Linked Edit
PR 31897 open graingert, 2022-03-15 10:58
PR 31934 open graingert, 2022-03-16 14:18
Messages (10)
msg415233 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-15 10:32
importing a module with bytes in `sys.path` fails with:

  File "<frozen importlib._bootstrap_external>", line 182, in _path_isabs
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

(see reproducer in attached demo.py)

however `sys.path` is documented as supporting bytes "Only strings and bytes should be added to sys.path; all other data types are ignored during import." https://docs.python.org/3/library/sys.html?highlight=Only%20strings%20and%20bytes#sys.path

bytes are allowed in PathFinder._find_spec https://github.com/python/cpython/blob/2cf7f865f099db11cc6903b334d9c376610313e8/Lib/importlib/_bootstrap_external.py#L1460-L1462 but perhaps they should be ignored or explicitly fsdecoded ? 


see also:
https://bugs.python.org/issue32642
https://github.com/python/importlib_metadata/issues/372#issuecomment-1067799424
msg415234 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2022-03-15 11:03
In case it helps anyone:

On Windows 3.11.0a5+ the full traceback is:

$ ./python.bat demo.py
Running Debug|x64 interpreter...
Traceback (most recent call last):
  File "...\demo.py", line 23, in <module>
    sys.exit(main())
             ^^^^^^
  File "...\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\demo.py", line 11, in _tmp_path
    yield pathlib.Path(tmp_dir)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\demo.py", line 18, in main
    import module
    ^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1080, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1487, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1459, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1596, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1656, in _fill_cache
TypeError: a bytes-like object is required, not 'str'

And on cygwin 3.8.12:

$ python demo.py
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 1346, in _path_importer_cache
KeyError: b'/tmp/tmprpymgive'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "demo.py", line 23, in <module>
    sys.exit(main())
  File "demo.py", line 18, in main
    import module
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1407, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1376, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1348, in _path_importer_cache
  File "<frozen importlib._bootstrap_external>", line 1324, in _path_hooks
  File "<frozen importlib._bootstrap_external>", line 1594, in path_hook_for_FileFinder
  File "<frozen importlib._bootstrap_external>", line 1469, in __init__
  File "<frozen importlib._bootstrap_external>", line 177, in _path_isabs
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
msg415235 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-15 11:11
this is a regression from 3.2:

```
Python 3.2.6 (default, Jan 18 2016, 19:21:14) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tempfile
>>> tempfile.TemporaryDirectory()
<TemporaryDirectory '/tmp/tmpd4jzut'>
>>> v = _
>>> tmp_dir = str(v.__enter__())
>>> tmp_dir
'/tmp/tmpd4jzut'
>>> f = open(tmp_dir + "/module.py", "w")
>>> f.write("def function():\n    return 1\n")
29
>>> f.close()
>>> import sys
>>> sys.path.append(tmp_dir.encode())
>>> import module
>>> module
<module 'module' from '/tmp/tmpd4jzut/module.py'>
>>> 
```
msg415236 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-15 11:14
https://docs.python.org/3/reference/import.html#path-entry-finders says "The encoding of bytes entries is determined by the individual path entry finders." see https://github.com/python/cpython/commit/82c1c781c7ee6496bd4c404b7ba972eed5dbcb12
msg415237 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-15 11:24
interestingly bytes filenames pointing to zip files on sys.path do support bytes (see zipfile_demo.py)
msg415238 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2022-03-15 11:29
> this is a regression from 3.2

In Windows, bytes paths in sys.path do not work in 3.2+. I didn't test 3.0 and 3.1, but practically one can say that bytes paths were never supported in Python 3 on Windows.
msg415239 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-15 11:35
zipimporter.zipimporter handles non-bytes paths here: https://github.com/python/cpython/blob/2cf7f865f099db11cc6903b334d9c376610313e8/Lib/zipimport.py#L65-L67 I think FileFinder should do the same
msg415323 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2022-03-16 02:33
I'd advocate for not supporting bytes paths and instead updating the documentation to require strings.
msg415340 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-16 14:18
> I'd advocate for not supporting bytes paths and instead updating the documentation to require strings.

I've got PR GH-31934 started to do this
msg415443 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2022-03-17 19:22
I think it depends on whether we want to say the standard/included/built-in import mechanisms don't support byte paths, or byte paths are entirely not supported even for 3rd-party code? I definitely think we should at least do the former, but I'm hesitant to close the door for those that need it by doing the latter.
History
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91181
2022-03-21 23:00:18brett.cannonlinkissue47026 dependencies
2022-03-17 19:22:31brett.cannonsetmessages: + msg415443
2022-03-17 19:19:36brett.cannonsetnosy: + brett.cannon
2022-03-16 14:18:28graingertsetmessages: + msg415340
pull_requests: + pull_request30026
2022-03-16 02:33:13jaracosetnosy: + jaraco
messages: + msg415323
2022-03-15 11:35:15graingertsetmessages: + msg415239
2022-03-15 11:29:08eryksunsetnosy: + eryksun
messages: + msg415238
2022-03-15 11:24:43graingertsetfiles: + zipfile_demo.py

messages: + msg415237
2022-03-15 11:14:32graingertsetmessages: + msg415236
2022-03-15 11:11:17graingertsetmessages: + msg415235
2022-03-15 11:03:37eric.smithsetnosy: + eric.smith
messages: + msg415234
2022-03-15 10:58:24graingertsetkeywords: + patch
stage: patch review
pull_requests: + pull_request29993
2022-03-15 10:33:28graingertsetcomponents: + Library (Lib)
versions: + Python 3.9, Python 3.10, Python 3.11
2022-03-15 10:32:43graingertcreate