classification
Title: distutils.spawn.find_executable() fails to find .py files on Windows
Type: behavior Stage: resolved
Components: Distutils Versions:
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Alexander.Todorov, dstufft, eric.araujo, eryksun, steve.dower
Priority: normal Keywords:

Created on 2020-10-07 08:39 by Alexander.Todorov, last changed 2020-10-07 22:17 by steve.dower. This issue is now closed.

Messages (5)
msg378150 - (view) Author: Alexander Todorov (Alexander.Todorov) * Date: 2020-10-07 08:39
As part of installing python-bugzilla via pip it searches for `rst2man` or `rst2man.py`, see:
https://github.com/python-bugzilla/python-bugzilla/blob/master/setup.py#L81

on Windows venvs there is venv\Scripts\rst2man.py (no .exe or without suffix) and when you call find_executable('rst2man.py') if doesn't find it.


The trouble is in this code snippet:

```
    _, ext = os.path.splitext(executable)
    if (sys.platform == 'win32') and (ext != '.exe'):
        executable = executable + '.exe'
```

`ext` here is `.py` and the if condition executes its body so the executable to search for becomes `rst2man.py.exe` which doesn't exist.

The extension check has been like that for more than 20 years:
https://github.com/python/cpython/commit/69628b0ad10f89a65902f5b911d1040ed9ae1ca2

but IMO it should be 

```
    if (sys.platform == 'win32') and (ext == ''):
        executable = executable + '.exe'
```

i.e. add `.exe` only if the file we're looking for doesn't already have an extension.


Let me know what you think? I can submit a PR for this.


Related issues:

- https://bugs.python.org/issue2200

- https://bugs.python.org/issue39260
msg378156 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-07 11:00
The linked code runs subprocess.check_output([rstbin, path]), which won't work with "rst2man.py" in Windows. Reliably running a .py script in Windows requires running it explicitly via py[w].exe or python[w].exe, or via sys.executable from an existing Python process. 

subprocess.Popen calls WinAPI CreateProcessW, which supports PE executables, and also CMD/BAT scripts via %ComSpec%. With shell=True the command is run via `cmd.exe /c`, which tries ShellExecuteExW in order to execute the file using the filetype's default registered action. However, the default action for .py files isn't dependable. It's often configured to open the script in an editor or IDE.

Thus script entrypoints in Windows get installed as wrapped launchers, e.g. "script.exe" instead of "script.py". The launcher executes the embedded entrypoint script via the fully-qualified path of python.exe.
msg378157 - (view) Author: Alexander Todorov (Alexander.Todorov) * Date: 2020-10-07 11:14
@Eryk Sun,
you are of course correct but still don't we need to handle this in Python? 

- find_executable() will search for rst2man.py.exe which doesn't exist so no good for the user
- there is also no rst2man.exe which is probably an issue with how docutils distributes this script - e.g. it doesn't distribute the wrapper .exe file
- lastly the caller (in my case python-bugzilla) will try to execute rst2man.py which will probably fail but at least they would see another failure and are more likely to change their own code base

The trouble for me here is that find_executable() will generally not find anything that doesn't have .exe suffix on Windows. This is independent of how the resulting file would be consumed later.
msg378160 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-07 13:12
The existing behavior of find_executable() is unusual compared to a native Windows file search via CreateProcessW or SearchPathW, for which a default ".exe" extension is appended only if the executable name doesn't already have an extension. (CreateProcessW supports searching for a file that has no extension by adding a trailing dot to the name, which in Windows is equivalent to the name without the trailing dot.) 

However, I don't think it's prudent to change the default behavior of this old code, especially since there are plans to deprecate distutils per PEP 632. It's recommended to migrate existing code to use shutil.which() and subprocess.run(). 

There are open issues to improve the behavior of shutil.which() in Windows. For example, currently it's limited to PATHEXT extensions, which is a generalization of how find_executable() works with ".exe". shutil.which() won't find "rst2man.py" if ".PY" isn't in PATHEXT. The proper behavior, if the CMD shell is the standard of correctness, is to include the exact name as the first candidate to check before trying the name plus the PATHEXT extensions. To be more like the CMD shell, a name without an extension should be excluded, but effectively included anyway if PATHEXT contains a "." entry.
msg378188 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-07 22:17
Yep, unless this directly impacts normal distutils usage, it's not going to be fixed (regardless of the formal deprecation proposal, we already have a policy of minimizing changes to distutils).

You could submit a patch to Bugzilla to switch them to shutil.which.
History
Date User Action Args
2020-10-07 22:17:38steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg378188

resolution: wont fix
stage: resolved
2020-10-07 13:12:06eryksunsetmessages: + msg378160
2020-10-07 11:14:55Alexander.Todorovsetmessages: + msg378157
2020-10-07 11:00:44eryksunsetnosy: + eryksun
messages: + msg378156
2020-10-07 08:39:18Alexander.Todorovcreate