On windows, shutil.which does not match the semantics of built-in command lookup. If you pass the name of a script like foo.py and the PATHEXT variable doesn't include .py it will search for foo.py.exe, foo.py.bat, foo.py.cmd, etc. but not foo.py, which should be the first name checked.

I am adding windows as a component during triaging since PATHEXT seems to be windows specific. Unfortunately, I couldn't verify this since I don't have windows system to check this against master so leaving it to 3.6.

Thanks

Isn't this an expected behavior?

I don't think this is expected behaviour. It's not documented what should happen in this case but the behaviour suggested by the OP (to search for the path as given, followed by [path+e for e in os.environ['PATHEXT'].split(os.pathsep)] seems reasonable to me.

The details and corner cases need thrashing out, which I don't have time to do right now, but the principle seems sound.

On further reflection, I'm less sure that the proposed behaviour is the best option, but I do think this warrants further consideration.

This is how windows looks up commands, as well as the built in "where" command. (Note that windows doesn't actually distinguish between "executable" files and just plain old files, so this could be confusing for UNIX users... a text file for instance will simply open in the default text editor when "executed" by the shell.)

It certainly doesn't match "which" semantics, but given the F_OK and X_OK flags I can see cases where it ought not to. I'm not sure it does what it implies for those either though.

I can see uses for "find files according to 'which'" and "find executable files according to 'which'".

Either way, I don't see an easy way to change the semantics any earlier than 3.8, so changing the target.

In a patch review [1] for issue 24505, I had a discussion with Toby Tobkin about extending the behavior of shutil.which() on Windows. This was to match the behavior of the CMD shell, including securing the search path via NeedCurrentDirectoryForExePath, searching for the exact filename, and implementing a real execute-access check. 

CMD won't directly execute a file that lacks execute access -- even a script or data file. We can implement this with a real implementation of os.access() that checks file security, for which there's an existing issue. (My previous suggestion to use CreateProcess and AssocQueryString to check for execute access was over the top. At the time I was thinking about building out support for a high-level shutil.execute, but it's out of scope here.)

Note that CMD will only try to execute an extensionless file if "." is in PATHEXT. (In the Windows file namespace, "name" and "name." are equivalent.) This obviously works for PE binaries, but we can also define an association for "." in the registy, which is useful for extensionless script files. Just associate "." files with a launcher that tries CreateProcess and falls back on a shebang line.

[1]: https://web.archive.org/web/20170619131224/http://bugs.python.org:80/review/24505/diff/16179/Lib/shutil.py

should I open a new issue for this, or is this an appropriate task to add to this discussion

shutil.which also doesn't apply `PATHEXT` when the path contains a directory separator

C:\Users\IEUser\astpretty>venv\Scripts\python --version
Python 3.7.5

C:\Users\IEUser\astpretty>venv\Scripts\python -c "import shutil; print(repr(shutil.which(r'venv\Scripts\python')))"
None

C:\Users\IEUser\astpretty>venv\Scripts\python -c "import shutil; print(repr(shutil.which(r'venv\Scripts\python.exe')))"
'venv\\Scripts\\python.exe'

Hello Anthony

would you mind to execute this command on your PC?

python -c "import os; print(os.environ.get("PATHEXT", "").split(os.pathsep))"

and show the result about this code?

sure, it's the standard PATHEXT (basically a fresh vm from modern.ie) -- the only reason I noticed this is I was reading the source: https://github.com/python/cpython/blob/5bbac8cbdf140ebce446ea4e7db2b20a5d7b8402/Lib/shutil.py#L1367-L1373



C:\Users\IEUser>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW

I am struggling with the same issue as Anthony. To provide a more direct response to Manjusaka's query:

python -c "import os; print(os.environ.get(\"PATHEXT\", \"\").split(os.pathsep))"
['.COM', '.EXE', '.BAT', '.CMD', '.VBS', '.VBE', '.JS', '.JSE', '.WSF', '.WSH', '.MSC']

Like Anthony, wondering if this should be it's own issue. Simply supporting paths with separators shouldn't, I think, lead to any of the ambiguities of matching a file exactly, without PATHEXT treatment.