classification
Title: shutil.which doesn't find files without PATHEXT extension on Windows
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, Manjusaka, SpecLad, eryksun, mitchelly, paul.moore, rmccampbell7, serhiy.storchaka, steve.dower, tim.golden, xtreak, zach.ware
Priority: normal Keywords:

Created on 2017-09-09 18:12 by rmccampbell7, last changed 2021-01-25 22:47 by mitchelly.

Messages (12)
msg301785 - (view) Author: Ryan McCampbell (rmccampbell7) Date: 2017-09-09 18:12
On windows, shutil.which does not match the semantics of built-in command lookup. If you pass the name of a script like foo.py and the PATHEXT variable doesn't include .py it will search for foo.py.exe, foo.py.bat, foo.py.cmd, etc. but not foo.py, which should be the first name checked.
msg326497 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-09-26 18:19
I am adding windows as a component during triaging since PATHEXT seems to be windows specific. Unfortunately, I couldn't verify this since I don't have windows system to check this against master so leaving it to 3.6.

Thanks
msg326498 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-26 19:24
Isn't this an expected behavior?
msg326500 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-09-26 19:50
I don't think this is expected behaviour. It's not documented what should happen in this case but the behaviour suggested by the OP (to search for the path as given, followed by [path+e for e in os.environ['PATHEXT'].split(os.pathsep)] seems reasonable to me.

The details and corner cases need thrashing out, which I don't have time to do right now, but the principle seems sound.
msg326501 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-09-26 19:56
On further reflection, I'm less sure that the proposed behaviour is the best option, but I do think this warrants further consideration.
msg326502 - (view) Author: Ryan McCampbell (rmccampbell7) Date: 2018-09-26 20:12
This is how windows looks up commands, as well as the built in "where" command. (Note that windows doesn't actually distinguish between "executable" files and just plain old files, so this could be confusing for UNIX users... a text file for instance will simply open in the default text editor when "executed" by the shell.)
msg326506 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-09-26 20:48
It certainly doesn't match "which" semantics, but given the F_OK and X_OK flags I can see cases where it ought not to. I'm not sure it does what it implies for those either though.

I can see uses for "find files according to 'which'" and "find executable files according to 'which'".

Either way, I don't see an easy way to change the semantics any earlier than 3.8, so changing the target.
msg326514 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-09-26 22:18
In a patch review [1] for issue 24505, I had a discussion with Toby Tobkin about extending the behavior of shutil.which() on Windows. This was to match the behavior of the CMD shell, including securing the search path via NeedCurrentDirectoryForExePath, searching for the exact filename, and implementing a real execute-access check. 

CMD won't directly execute a file that lacks execute access -- even a script or data file. We can implement this with a real implementation of os.access() that checks file security, for which there's an existing issue. (My previous suggestion to use CreateProcess and AssocQueryString to check for execute access was over the top. At the time I was thinking about building out support for a high-level shutil.execute, but it's out of scope here.)

Note that CMD will only try to execute an extensionless file if "." is in PATHEXT. (In the Windows file namespace, "name" and "name." are equivalent.) This obviously works for PE binaries, but we can also define an association for "." in the registy, which is useful for extensionless script files. Just associate "." files with a launcher that tries CreateProcess and falls back on a shebang line.

[1]: https://web.archive.org/web/20170619131224/http://bugs.python.org:80/review/24505/diff/16179/Lib/shutil.py
msg360439 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2020-01-22 00:13
should I open a new issue for this, or is this an appropriate task to add to this discussion

shutil.which also doesn't apply `PATHEXT` when the path contains a directory separator

C:\Users\IEUser\astpretty>venv\Scripts\python --version
Python 3.7.5

C:\Users\IEUser\astpretty>venv\Scripts\python -c "import shutil; print(repr(shutil.which(r'venv\Scripts\python')))"
None

C:\Users\IEUser\astpretty>venv\Scripts\python -c "import shutil; print(repr(shutil.which(r'venv\Scripts\python.exe')))"
'venv\\Scripts\\python.exe'
msg360482 - (view) Author: Manjusaka (Manjusaka) * Date: 2020-01-22 16:07
Hello Anthony

would you mind to execute this command on your PC?

python -c "import os; print(os.environ.get("PATHEXT", "").split(os.pathsep))"

and show the result about this code?
msg360485 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2020-01-22 16:30
sure, it's the standard PATHEXT (basically a fresh vm from modern.ie) -- the only reason I noticed this is I was reading the source: https://github.com/python/cpython/blob/5bbac8cbdf140ebce446ea4e7db2b20a5d7b8402/Lib/shutil.py#L1367-L1373



C:\Users\IEUser>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
msg385676 - (view) Author: Mitchell Young (mitchelly) Date: 2021-01-25 22:47
I am struggling with the same issue as Anthony. To provide a more direct response to Manjusaka's query:

python -c "import os; print(os.environ.get(\"PATHEXT\", \"\").split(os.pathsep))"
['.COM', '.EXE', '.BAT', '.CMD', '.VBS', '.VBE', '.JS', '.JSE', '.WSF', '.WSH', '.MSC']

Like Anthony, wondering if this should be it's own issue. Simply supporting paths with separators shouldn't, I think, lead to any of the ambiguities of matching a file exactly, without PATHEXT treatment.
History
Date User Action Args
2021-01-25 22:47:48mitchellysetnosy: + mitchelly
messages: + msg385676
2020-01-22 16:30:36Anthony Sottilesetmessages: + msg360485
2020-01-22 16:07:23Manjusakasetnosy: + Manjusaka
messages: + msg360482
2020-01-22 00:13:03Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg360439
2019-08-27 20:45:37SpecLadsetnosy: + SpecLad
2018-09-26 22:18:08eryksunsetnosy: + eryksun
messages: + msg326514
2018-09-26 20:48:57steve.dowersetmessages: + msg326506
versions: + Python 3.8, - Python 3.6
2018-09-26 20:12:46rmccampbell7setmessages: + msg326502
2018-09-26 19:56:21paul.mooresetmessages: + msg326501
2018-09-26 19:50:57paul.mooresetmessages: + msg326500
2018-09-26 19:24:06serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg326498
2018-09-26 18:19:13xtreaksetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg326497
components: + Windows
2018-09-26 18:03:15xtreaksetnosy: + xtreak
2017-09-09 18:12:09rmccampbell7create