classification
Title: shutil.which() should support bytes
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, cheryl.sabella, hynek, loewis, python-dev, serhiy.storchaka, tarek, vstinner
Priority: normal Keywords: patch

Created on 2013-06-22 15:29 by Arfrever, last changed 2019-02-13 11:25 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
which_bytes.patch vstinner, 2013-06-23 00:21 review
Pull Requests
URL Status Linked Edit
PR 11818 merged cheryl.sabella, 2019-02-11 14:49
Messages (11)
msg191644 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2013-06-22 15:29
shutil.which() should support bytes. Some other functions in shutil module support bytes.

>>> shutil.which("echo")
'/bin/echo'                                                                                                                                                  
>>> shutil.which(b"echo")                                                                                                                                    
Traceback (most recent call last):                                                                                                                           
  File "<stdin>", line 1, in <module>                                                                                                                        
  File "/usr/lib64/python3.3/shutil.py", line 1126, in which
    name = os.path.join(dir, thefile)
  File "/usr/lib64/python3.3/posixpath.py", line 92, in join
    "components.") from None
TypeError: Can't mix strings and bytes in path components.
>>> shutil.which("echo", path="/bin")
'/bin/echo'
>>> shutil.which("echo", path=b"/bin")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.3/shutil.py", line 1098, in which
    path = path.split(os.pathsep)
TypeError: Type str doesn't support the buffer API
>>> shutil.which(b"echo", path=b"/bin")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.3/shutil.py", line 1098, in which
    path = path.split(os.pathsep)
TypeError: Type str doesn't support the buffer API
msg191684 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-06-23 00:21
Here is an implementation.
msg191686 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-06-23 07:41
What about bytearrays and other byte-like objects?

However I'm not sure this enhancement is worth to be accepted. Many other high-level functions in os and shutil modules do not support bytes paths. For shutil.which() there is no backward compatibility with Python 2 which we should support.
msg191691 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-06-23 11:42
"What about bytearrays and other byte-like objects?

However I'm not sure this enhancement is worth to be accepted. Many other high-level functions in os and shutil modules do not support bytes paths. For shutil.which() there is no backward compatibility with Python 2 which we should support."

Bytes is the native type for path on all platforms... except Windows. So I fixed many functions of the os module to support bytes parameters. For example, functions of os.path support str and bytes types, but not bytearray. I also prefer to reject byte-like objects, to keep it simple.

The bytes type should be accepted for any kind of os function parameter: path, environment variable, command line argument, etc. It's not a question of being compatible with Python 2, an application can make the choice of only used bytes because bytes is the native type for OS data on UNIX.

Python 3 supports also "bytes stored in Unicode" thanks to the PEP 393, but it's not exactly the same: a path stored as str depends on the filesystem locale, whereas a bytes string is well defined (it's just an array of 8-bit unsigned numbers) and doesn't depend on any configuration variable.
msg206370 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-16 21:23
All low-level functions in the os.path module supports string and bytes paths. But not all high-level functions in the shutil module supports bytes paths. Adding support for bytes paths will complicate implementation and tests.
msg206373 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-12-16 21:46
Almost all functions of the shutil module support bytes filenames. I tested shutil.copyfile(), shutil.ignore_patterns(), shutil.copytree(), shutil.rmtree() and shutil.move().

I only found one function which only support Unicode filenames: shutil.make_archive().
msg206374 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-12-16 21:49
New changeset a1a05e2724dd by Victor Stinner in branch 'default':
Issue #18283: shutil.which() now supports bytes argument, not only text argument.
http://hg.python.org/cpython/rev/a1a05e2724dd
msg206386 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-12-16 22:40
which_bytes.patch does not work on Windows.

According to Serhiy, it's a new feature and so should wait for Python 3.5.
msg335141 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python triager) Date: 2019-02-09 22:52
Would it be worthwhile to make a PR of this for 3.8?
msg335209 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-11 11:59
> Would it be worthwhile to make a PR of this for 3.8?

Yes. Would you be interested to convert my old patch to a fresh PR? I can help you to work on that.
msg335415 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-13 11:25
New changeset 5680f6546dcda550ad70eefa0a5ebf1375303307 by Victor Stinner (Cheryl Sabella) in branch 'master':
bpo-18283: Add support for bytes to shutil.which (GH-11818)
https://github.com/python/cpython/commit/5680f6546dcda550ad70eefa0a5ebf1375303307
History
Date User Action Args
2019-02-13 11:25:48vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-02-13 11:25:13vstinnersetmessages: + msg335415
2019-02-11 14:49:39cheryl.sabellasetstage: patch review
pull_requests: + pull_request11840
2019-02-11 11:59:33vstinnersetmessages: + msg335209
2019-02-09 22:52:41cheryl.sabellasetnosy: + cheryl.sabella

messages: + msg335141
versions: + Python 3.8, - Python 3.5
2013-12-17 16:42:02Arfreversetstage: resolved -> (no value)
2013-12-16 22:40:52vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg206386

versions: + Python 3.5, - Python 3.4
2013-12-16 22:22:52Arfreversetstage: resolved
versions: + Python 3.4, - Python 3.5
2013-12-16 21:54:30vstinnersetstatus: open -> closed
resolution: fixed
2013-12-16 21:49:11python-devsetnosy: + python-dev
messages: + msg206374
2013-12-16 21:46:36vstinnersetmessages: + msg206373
2013-12-16 21:23:53serhiy.storchakasetmessages: + msg206370
versions: + Python 3.5, - Python 3.4
2013-06-23 11:44:22vstinnersetnosy: + loewis
2013-06-23 11:42:32vstinnersetmessages: + msg191691
2013-06-23 07:41:24serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg191686
2013-06-23 00:21:36vstinnersetfiles: + which_bytes.patch
keywords: + patch
messages: + msg191684
2013-06-22 22:06:24vstinnersetnosy: + vstinner
2013-06-22 17:47:40serhiy.storchakasettype: enhancement
versions: - Python 3.3
2013-06-22 15:29:11Arfrevercreate