classification
Title: subprocess doesn't pass arguments correctly on Linux when shell=True
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: benhoyt, berwyn, davidfraser, miss-islington, r.david.murray, skrah
Priority: normal Keywords: patch

Created on 2009-08-12 08:33 by davidfraser, last changed 2019-06-13 14:13 by miss-islington. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess-shell-args-2.6.patch davidfraser, 2009-08-14 08:21 Patch against Python 2.6.2 without unneccessary args
Pull Requests
URL Status Linked Edit
PR 14045 merged python-dev, 2019-06-13 07:29
PR 14058 merged miss-islington, 2019-06-13 14:00
Messages (14)
msg91492 - (view) Author: David Fraser (davidfraser) Date: 2009-08-12 08:33
(from
http://stackoverflow.com/questions/1253122/why-does-subprocess-popen-with-shelltrue-work-differently-on-linux-vs-windows/1254322)

When using subprocess.Popen(args, shell=True) to run "gcc --version"
(just as an example), on Windows we get this:

>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc (GCC) 3.4.5 (mingw-vista special r3) ...

So it's nicely printing out the version as I expect. But on Linux we get
this:

>>> from subprocess import Popen
>>> Popen(['gcc', '--version'], shell=True)
gcc: no input files

Because gcc hasn't received the --version option.

The docs don't specify exactly what should happen to the args under
Windows, but it does say, on Unix, "If args is a sequence, the first
item specifies the command string, and any additional items will be
treated as additional shell arguments." IMHO the Windows way is better,
because it allows you to treat Popen(arglist) calls the same as
Popen(arglist, shell=True) ones.

The strange implementation is actually the UNIX one, which does the
following (where each space separates a different argument):

/bin/sh -c gcc --version

It looks like the correct implementation (at least on Linux) would be:

/bin/sh -c "gcc --version" gcc --version

Since this would set the command string from the quoted parameters, and
pass the other parameters successfully.

From the sh man page section for -c:

    Read commands from the command_string operand instead of from the
standard input. Special parameter 0 will be set from the command_name
operand and the positional parameters ($1, $2, etc.) set from the
remaining argument operands.

This patch seems to fairly simply do the trick, as well as testing it.
msg91510 - (view) Author: Ben Hoyt (benhoyt) * Date: 2009-08-12 23:18
Oops, didn't intend to change the type, changing back.
msg91511 - (view) Author: Berwyn (berwyn) Date: 2009-08-13 01:17
I agree with the patch, and it works fine:
    /bin/sh -c "gcc --version" gcc --version

But I begin to wonder whether bash has a bug since bash only seems to
observe the string, not the parameters after it.  For example:
    $ bash -c "./printargs.py abc" def ghi jkl
produces:
    ['./printargs.py', 'abc']

where printargs.py is:
    #!/usr/bin/python
    import sys
    print sys.argv

Just something to be aware of.
msg91542 - (view) Author: David Fraser (davidfraser) Date: 2009-08-14 08:21
Ah, upon closer inspection - the special parameters $0 $1 $2 etc
mentioned in the sh docs refer to parameters within the command string,
so that:
   sh -c 'echo $2 $0 $1' run for "the people"
produces:
   the people run for

So the correct patch would be to leave out the extra parameters and just
have the quoted string...

The question then becomes, is simple " ".join(args) sufficient or should
there be some quoting of them... attaching a patch in the meantime
although that needs to be resolved.
msg95329 - (view) Author: David Fraser (davidfraser) Date: 2009-11-16 08:44
This is closely related to http://bugs.python.org/issue2304
msg111092 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-07-21 17:23
This is a request for changing the current behavior on Unix, and I
doubt this is going to happen, since it could break existing code.

On a side note, there are already two ways get the desired output:


>>> Popen("gcc --version", shell=True)
gcc (GCC) 4.1.3 20080623 (prerelease) (Ubuntu 4.1.2-23ubuntu3) ...


>>> p = Popen(["gcc", "--version"], stdout=PIPE)
>>> p.stdout.read()
b'gcc (GCC) 4.1.3 20080623 (prerelease) ...
msg111115 - (view) Author: Berwyn (berwyn) Date: 2010-07-21 20:39
I think this should be documented more clearly rather than just rejected.  It keeps causing various people grief.  David Fraser's post would be good documentation.
msg111121 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-07-21 21:06
I find this particular section one of the most clear sections in the
whole Python documentation:

"On Unix, with shell=True: If args is a string, it specifies the command string to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])"
msg111122 - (view) Author: Berwyn (berwyn) Date: 2010-07-21 21:23
Good point, Stefan.  In hindsight my issue was probably more with understanding how the sh command line works rather than how python works.
msg111123 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-07-21 21:29
I'm glad you agree. Closing this now.
msg111137 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-22 00:30
Note that the "clear wording" was only introduced on 2/4 of this year (issue 6760).
msg111149 - (view) Author: Berwyn (berwyn) Date: 2010-07-22 03:51
Ah.  I don't need to feel so forgetful, then.
msg345519 - (view) Author: miss-islington (miss-islington) Date: 2019-06-13 14:00
New changeset 95492032c48fef20b9c7076a23fe7e46927a4688 by Miss Islington (bot) (Makdon) in branch 'master':
bpo-6689: os.path.commonpath raises ValueError for different drives isn't documented (GH-14045)
https://github.com/python/cpython/commit/95492032c48fef20b9c7076a23fe7e46927a4688
msg345520 - (view) Author: miss-islington (miss-islington) Date: 2019-06-13 14:13
New changeset ec3839a215a68cf35ff1f90cb6823f67a5abdce3 by Miss Islington (bot) in branch '3.8':
bpo-6689: os.path.commonpath raises ValueError for different drives isn't documented (GH-14045)
https://github.com/python/cpython/commit/ec3839a215a68cf35ff1f90cb6823f67a5abdce3
History
Date User Action Args
2019-06-13 14:13:15miss-islingtonsetmessages: + msg345520
2019-06-13 14:00:37miss-islingtonsetpull_requests: + pull_request13920
2019-06-13 14:00:18miss-islingtonsetnosy: + miss-islington
messages: + msg345519
2019-06-13 07:29:59python-devsetpull_requests: + pull_request13907
2010-07-22 03:51:06berwynsetmessages: + msg111149
title: subprocess doesn't pass arguments correctly on Linux when shell=True -> subprocess doesn't pass arguments correctly on Linux when shell=True
2010-07-22 00:30:56r.david.murraysetmessages: + msg111137
2010-07-22 00:30:44r.david.murraysetmessages: - msg111136
2010-07-22 00:29:42r.david.murraysetnosy: + r.david.murray
messages: + msg111136
2010-07-21 21:29:28skrahsetstatus: open -> closed

messages: + msg111123
2010-07-21 21:23:53berwynsetstatus: pending -> open

messages: + msg111122
2010-07-21 21:07:00skrahsetstatus: open -> pending

messages: + msg111121
2010-07-21 20:39:13berwynsetstatus: pending -> open

messages: + msg111115
2010-07-21 17:23:48skrahsetstatus: open -> pending

type: behavior -> enhancement

nosy: + skrah
messages: + msg111092
resolution: rejected
stage: patch review -> resolved
2010-07-21 16:13:52BreamoreBoysetstage: patch review
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2009-11-16 08:44:43davidfrasersetmessages: + msg95329
2009-08-14 08:21:31davidfrasersetfiles: - current-3.patch
2009-08-14 08:21:21davidfrasersetfiles: - current-2.6.patch
2009-08-14 08:21:02davidfrasersetfiles: + subprocess-shell-args-2.6.patch

messages: + msg91542
2009-08-13 01:17:07berwynsetnosy: + berwyn
messages: + msg91511
2009-08-12 23:18:02benhoytsettype: enhancement -> behavior
messages: + msg91510
2009-08-12 23:16:38benhoytsetnosy: + benhoyt
type: behavior -> enhancement
2009-08-12 08:34:45davidfrasersetfiles: + current-3.patch
2009-08-12 08:33:56davidfrasercreate