This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: subprocess: replacement shell on windows with executable="..." arg
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: anishathalye, eryksun, paul.moore, snoopyjc, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2020-05-01 13:34 by anishathalye, last changed 2022-04-11 14:59 by admin.

Messages (3)
msg367844 - (view) Author: Anish Athalye (anishathalye) Date: 2020-05-01 13:34
On Windows, using subprocess.call() and specifying both shell=True and the
executable='...' keyword arguments produces an undesirable result when the
specified shell is a POSIX-like shell rather than the standard cmd.exe.

I think the documentation is unclear on the semantics of Popen() when both
shell=True and executable= are specified on Windows. It does say the following
about POSIX systems:

> If shell=True, on POSIX the executable argument specifies a replacement shell
> for the default /bin/sh.

But the documentation doesn't say anything about Windows in this scenario, so
I'm not sure if this is a bug, or if it's undefined behavior.

Concretely, here's an example program that fails due to this:

    import subprocess
    bash = 'C:\\Program Files\\Git\\usr\\bin\\bash.exe'
    subprocess.call('f() { echo test; }; f', shell=True, executable=bash)

It prints out this mysterious-looking error:

    /c: /c: Is a directory

Tracing this into subprocess.py, it looks like it's because the executable bash
(as specified) is being called with the argv that's approximately ['cmd.exe',
'/c', 'f() { echo test; }; f'] (and the program being launched is indeed bash).

Bash doesn't expect a '/c' argument, it wants a '-c' there.

The problematic code in subprocess.py is here:
https://github.com/python/cpython/blob/1def7754b7a41fe57efafaf5eff24cfa15353444/Lib/subprocess.py#L1407
If the '/c' is replaced with a '-c', the example program above works (bash
doesn't seem to care that it's called with an argv[0] that doesn't make sense,
though presumably that should be fixed too).

I'm not sure how this could be fixed. It's unclear when '/c' should be used, as
opposed to '-c'. Doing it based on the contents of the executable= argument or
the SHELL environment variable or COMSPEC might be fragile? I couldn't find
much about this online, but I did find one project (in Ruby) that seems to have
run into a similar problem. See
https://github.com/kimmobrunfeldt/chokidar-cli/issues/15 and
https://github.com/kimmobrunfeldt/chokidar-cli/pull/16.

At the very least, even if this isn't fixed / can't be fixed, it might be nice
if it's possible to give some sort of useful warning/error when this happens --
perhaps say that specifying both shell=True and executable="..." isn't
supported on Windows?

I ran into this issue while while debugging an issue in a project of mine. In
case the additional context is useful, here is the discussion:
https://github.com/anishathalye/dotbot/issues/219
msg407644 - (view) Author: Joe Cool (snoopyjc) Date: 2021-12-04 06:55
Proposed solution:

                if comspec.endswith('sh.exe') or comspec.endswith('sh'):    # issue 40467
                    args = '{} -c "{}"'.format (comspec, args)              # issue 40467
                else:                                                       # issue 40467
                    args = '{} /c "{}"'.format (comspec, args)
msg407647 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-12-04 09:08
> it might be nice if it's possible to give some sort of useful 
> warning/error when this happens -- perhaps say that specifying 
> both shell=True and executable="..." isn't supported on Windows?

The `shell` parameter is documented as follows for Windows:

    On Windows with shell=True, the COMSPEC environment variable 
    specifies the default shell. The only time you need to specify 
    shell=True on Windows is when the command you wish to execute is 
    built into the shell (e.g. dir or copy). You do not need
    shell=True to run a batch file or console-based executable.

It wouldn't hurt to clarify that the COMSPEC shell has to support `/c`. This is required by CreateProcessW(), which uses COMSPEC to run BAT and CMD files.

The discussion about using `executable` with `shell` could be extended for Windows. But this would also require new behavior. For example:

Original:

    if shell:
        startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
        startupinfo.wShowWindow = _winapi.SW_HIDE
        comspec = os.environ.get("COMSPEC", "cmd.exe")
        args = '{} /c "{}"'.format (comspec, args)

Proposed:

    if shell:
        startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
        startupinfo.wShowWindow = _winapi.SW_HIDE
        if executable is not None:
            cmd = executable
        else:
            cmd = os.path.normpath(os.environ.get("COMSPEC", "cmd.exe"))
            if "\\" in cmd:
                executable = cmd
        args = '"{}" /c "{}"'.format(cmd, args)

> if comspec.endswith('sh.exe') or comspec.endswith('sh'):
>     args = '{} -c "{}"'.format (comspec, args)         

sh is not a valid COMSPEC shell. To use sh automatically, subprocess would have to support and prefer the SHELL [1] environment variable in Windows -- and in POSIX for that matter.

---
[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_03
History
Date User Action Args
2022-04-11 14:59:30adminsetgithub: 84647
2021-12-04 09:14:59eryksunsetversions: + Python 3.10, Python 3.11, - Python 3.7
2021-12-04 09:08:33eryksunsetnosy: + eryksun
messages: + msg407647
2021-12-04 06:55:34snoopyjcsetnosy: + snoopyjc
messages: + msg407644
2020-05-01 13:34:17anishathalyecreate