Title: Differences between /usr/bin/which and shutil.which()
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.3, Python 3.4
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: barry Nosy List: barry, brian.curtin, hynek, ned.deily, pitrou, python-dev, r.david.murray, serhiy.storchaka, tarek
Priority: normal Keywords: patch

Created on 2013-01-22 12:14 by serhiy.storchaka, last changed 2013-04-16 15:22 by barry. This issue is now closed.

File name Uploaded Description Edit
shutil_which_empty_path.patch serhiy.storchaka, 2013-01-23 14:23 review
Messages (15)
msg180376 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-22 12:14
$ PATH= /usr/bin/which python
$ PATH=: /usr/bin/which python
$ PATH=/usr: /usr/bin/which python

>>> shutil.which('python', path='')
>>> shutil.which('python', path=':')
>>> shutil.which('python', path='/usr:')

First, I propose interpret path='' as an empty path, not as a default path (we have None for this). However the interpreting of an empty directory in non-empty PATH can be platform-depending.
msg180378 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-22 12:18
I'm not sure reproducing the quirks of /usr/bin/which is a good idea. shutil.which() is meant to be useful and easy to understand, not to be 100% bash-compatible.
And, anyway, what would be the point of passing an empty path, if the return value is guaranteed to be None?
msg180380 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-22 12:36
/usr/bin/which is not a Bash. ;)

The path can be unexpectedly empty. If we got None then we can detect the error, but if we got something out of the path then we can miss our fault.
msg180398 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-22 15:11
What I think it is suppose to do (the user expects it to do) is find the program that would be run if the command were typed at the command prompt.

rdmurray@hey:~>which python
rdmurray@hey:~>export PATH=
rdmurray@hey:~>which python
python not found
zsh: command not found: python

As Serhiy noted, this result may be platform dependent.  Which is unfortunate.
msg180400 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-22 15:25
No, I noted that result of PATH=: or PATH=$PATH: can be platform dependent (I'm not sure).
msg180404 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-22 15:58
I was speaking in general of 'which program would be executed if the command is typed at the prompt' as being system dependent, which it demonstrably is since the behavior on unix and windows differs with regards to the current directory.
msg180405 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-22 15:59
And no, what I wrote wasn't clear :)
msg180413 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-01-22 17:17
FWIW, the POSIX standard gives some guidance on how PATH is to be interpreted for conforming systems, including:

"A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent <colon> characters ( "::" ), as an initial <colon> preceding the rest of the list, or as a trailing <colon> following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH."
msg180464 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-23 14:23
Thank you, Ned, for information.

Here is a patch which remove the first difference (processing an empty path).

The second difference is not semantically significant and I'm not sure whether we need to get rid of it.
msg182285 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-02-17 21:32
msg182303 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-18 08:06
The result of PATH= is also platform dependent. Testing on OS X which has a BSD heritage rather a Linux one:

$ PATH= /usr/bin/which python

# without patch
$ PATH= ./python -c 'import shutil; print(shutil.which("python"))'
$ ./python -c 'import shutil; print(shutil.which("python", path=""))'

# with the patch:
$ PATH= ./python -c 'import shutil; print(shutil.which("python"))'
$ ./python -c 'import shutil; print(shutil.which("python", path=""))'

So, for OS X, shutil.which doesn't match /usr/bin/which behavior for the PATH= case either with or without the patch.  FreeBSD (8.2) /usr/bin/which is the same.  The other cases are the same as Linux.

I suppose the patched behavior is preferable, though.

In any case, the shutil.which docs also need to be updated.
msg187039 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-04-15 22:48
From a documentation standpoint, path='' is not the same as "When no path is specified", so indeed it should return None when path=''.  Serhiy's patch looks good to me.
msg187045 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-04-16 01:39
Serhiy, I'd say go ahead and commit it.  +1 from me.
msg187091 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-04-16 15:19
New changeset eb8c575fa781 by Barry Warsaw in branch '3.3':
- Issue #17012: shutil.which() no longer fallbacks to the PATH environment

New changeset 8f5b37f8f964 by Barry Warsaw in branch 'default':
- Issue #17012: shutil.which() no longer fallbacks to the PATH environment
msg187093 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-04-16 15:21
I couldn't wait. :)
Date User Action Args
2013-04-16 15:22:58barrysetstatus: open -> closed
resolution: fixed
2013-04-16 15:21:11barrysetassignee: serhiy.storchaka -> barry
messages: + msg187093
2013-04-16 15:19:30python-devsetnosy: + python-dev
messages: + msg187091
2013-04-16 01:39:45barrysetmessages: + msg187045
2013-04-16 01:39:26barrysetassignee: barry -> serhiy.storchaka
2013-04-15 22:57:15barrysetassignee: barry
2013-04-15 22:48:58barrysetmessages: + msg187039
2013-04-15 22:47:46barrysetnosy: + barry
2013-02-18 08:06:39ned.deilysetmessages: + msg182303
2013-02-17 21:32:46serhiy.storchakasetmessages: + msg182285
2013-01-23 14:24:02serhiy.storchakasetstage: patch review
2013-01-23 14:23:38serhiy.storchakasetfiles: + shutil_which_empty_path.patch
keywords: + patch
messages: + msg180464
2013-01-22 17:17:52ned.deilysetnosy: + ned.deily
messages: + msg180413
2013-01-22 15:59:13r.david.murraysetmessages: + msg180405
2013-01-22 15:58:47r.david.murraysetmessages: + msg180404
2013-01-22 15:25:26serhiy.storchakasetmessages: + msg180400
2013-01-22 15:12:00r.david.murraysetnosy: + r.david.murray
messages: + msg180398
2013-01-22 12:36:30serhiy.storchakasetmessages: + msg180380
2013-01-22 12:18:10pitrousetmessages: + msg180378
2013-01-22 12:14:00serhiy.storchakacreate