classification
Title: subprocess: Arguments to .bat scripts get interpreted by shell
Type: behavior Stage: committed/rejected
Components: Library (Lib) Versions: Python 3.3, Python 3.2, Python 3.1, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder: subprocess.list2cmdline doesn't quote the & character
View: 8972
Assigned To: Nosy List: Arve.Knudsen, SilentGhost, r.david.murray
Priority: normal Keywords:

Created on 2011-02-07 14:49 by Arve.Knudsen, last changed 2011-02-07 19:02 by Arve.Knudsen. This issue is now closed.

Files
File name Uploaded Description Edit
case.zip Arve.Knudsen, 2011-02-07 14:49 Case demonstrating bug
unnamed Arve.Knudsen, 2011-02-07 18:30
unnamed Arve.Knudsen, 2011-02-07 19:02
Messages (10)
msg128119 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2011-02-07 14:49
As a regression from Python 2.6, when running .bat scripts on Windows with f.ex. subprocess.call under Python 2.7.1, arguments get interpreted by the shell. This bit me in the way that I could no longer pass arguments containing the character '|' to .bat scripts, something that works fine under Python 2.6.

To see the bug in action, run tst.py in the attached case.zip with Python 2.7.1. The case also demonstrates that the '|' character only gets interpreted when launching cmd scripts, not e.g. python.exe.
msg128125 - (view) Author: SilentGhost (SilentGhost) Date: 2011-02-07 15:33
This seem to have been discussed in issue8972: closing as a duplicate.
msg128129 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-02-07 15:38
Well, it's not really a duplicate, but rather invalid, since the 2.7 behavior is more correct than the 2.6 behavior.  Not that it matters all that much what particular resolution is attached to a bug.

Arve: It is unfortunate that the bug fix in issue 8972 caused a regression for you, but at this point what is done is done.

Also, for future reference, when attaching files to an issue please attach text files only.  They are much easier to review.
msg128135 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2011-02-07 16:39
Would you mind defining why the 2.7 behaviour is correct in this particular
case? The list2cmdline behaviour discussed in issue 8972 has to do with
shell commandlines; in this case, in my eyes any shell invocation (in order
to run a .bat I guess) is only an implementation detail. Is the 2.7
behaviour correct since the Win32 function CreateProcess happens to start a
shell to run .bat files, which in turn interprets the arguments? I'm
guessing this is what happens...

There certainly can't be any universal definition in subprocess saying that
arguments to shell scripts shall be interpreted by the shell; when trying
now *on Linux* to launch a corresponding .sh script through subprocess,
arguments are passed through verbatim. I should like to see subprocess
maintain (or keep maintaining, rather) a uniform interface, so that
arguments are not interpreted so long as the 'shell' argument equals False.

I understand completely that list2cmdline should implement C library
behaviour, and not protect arguments from the shell, but that doesn't mean
it should automatically define the behaviour of other functions such as
Popen and call.

Arve

On Mon, Feb 7, 2011 at 4:38 PM, R. David Murray <report@bugs.python.org>wrote:

>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> Well, it's not really a duplicate, but rather invalid, since the 2.7
> behavior is more correct than the 2.6 behavior.  Not that it matters all
> that much what particular resolution is attached to a bug.
>
> Arve: It is unfortunate that the bug fix in issue 8972 caused a regression
> for you, but at this point what is done is done.
>
> Also, for future reference, when attaching files to an issue please attach
> text files only.  They are much easier to review.
>
> ----------
> nosy: +r.david.murray
> resolution: duplicate -> invalid
> stage:  -> committed/rejected
> superseder:  -> subprocess.list2cmdline doesn't quote the & character
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11139>
> _______________________________________
>
msg128139 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-02-07 17:06
The point is that subprocess (now!) is *not* interpreting the arguments when shell is false.  It is passing them through to Windows.  What windows does with them after that is out of the control of subprocess (and always has been).
msg128144 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2011-02-07 18:12
I wasn't saying that subprocess is interpreting the arguments, but that the
shell is. Which was highly unexpected to me when outside of shell mode
(i.e., shell == False). From some further testing, though, I see that the
transformation from sequence to command line on Windows plays a bigger role
than I was aware of even though I've used subprocess since it was introduced
in the standard library. That is, I see that also in Python 2.6, arguments
to .bat scripts get interpreted if command arguments are passed to
subprocess as a string rather in a list.

My current conclusion is that there has been a hole in the subprocess design
the whole time, so that my case (start a Windows shell script like an
executable) has just worked by chance, due to that I have passed arguments
as a list which then got implicitly protected (even outside of shell mode).
Maybe subprocess.Popen should take another option to implement the old
behaviour of protecting arguments supplied as a list, given that this should
be a common enough case?

Arve

On Mon, Feb 7, 2011 at 6:06 PM, R. David Murray <report@bugs.python.org>wrote:

>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> The point is that subprocess (now!) is *not* interpreting the arguments
> when shell is false.  It is passing them through to Windows.  What windows
> does with them after that is out of the control of subprocess (and always
> has been).
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11139>
> _______________________________________
>
msg128145 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-02-07 18:19
As noted in issue 8972, it's not clear what such a mode should actually do.  If you have a concrete proposal you could make it, probably on the python-ideas mailing list.  But I'm personally not in favor of it; I prefer maintaining as much API consistency between windows and linux as we reasonably can.
msg128147 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2011-02-07 18:30
The non-obvious thing with 8972 is what to do with an argument list when the
'shell' option is True, isn't it? I can't see how it overlaps with my case.
My suggestion is a flag to enable argument protection in Popen, either just
for arguments passed as a list (i.e., like how it's done in 2.6) or maybe
also for the case where you pass the command as a string (I haven't
considered that case yet, since it's not directly interesting to me).

I can actually see that exarkun suggests something along the same lines in
8972: 'implement the cmd.exe quoting rules in a different function from the
MS C runtime rules'.

Arve

On Mon, Feb 7, 2011 at 7:19 PM, R. David Murray <report@bugs.python.org>wrote:

>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> As noted in issue 8972, it's not clear what such a mode should actually do.
>  If you have a concrete proposal you could make it, probably on the
> python-ideas mailing list.  But I'm personally not in favor of it; I prefer
> maintaining as much API consistency between windows and linux as we
> reasonably can.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11139>
> _______________________________________
>
msg128150 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-02-07 19:00
Yes, having a cmd.exe 'quote' function, parallel to the shutil 'quote' function, would be an excellent addition to the stdlib.  You could open a feature request for that.
msg128151 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2011-02-07 19:02
I'll keep it in mind.

On Mon, Feb 7, 2011 at 8:00 PM, R. David Murray <report@bugs.python.org>wrote:

>
> R. David Murray <rdmurray@bitdance.com> added the comment:
>
> Yes, having a cmd.exe 'quote' function, parallel to the shutil 'quote'
> function, would be an excellent addition to the stdlib.  You could open a
> feature request for that.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11139>
> _______________________________________
>
History
Date User Action Args
2011-02-07 19:02:11Arve.Knudsensetfiles: + unnamed

messages: + msg128151
nosy: r.david.murray, SilentGhost, Arve.Knudsen
2011-02-07 19:00:07r.david.murraysetnosy: r.david.murray, SilentGhost, Arve.Knudsen
messages: + msg128150
2011-02-07 18:30:57Arve.Knudsensetfiles: + unnamed

messages: + msg128147
nosy: r.david.murray, SilentGhost, Arve.Knudsen
2011-02-07 18:19:56r.david.murraysetnosy: r.david.murray, SilentGhost, Arve.Knudsen
messages: + msg128145
2011-02-07 18:14:50SilentGhostsetfiles: - unnamed
nosy: r.david.murray, SilentGhost, Arve.Knudsen
2011-02-07 18:14:47SilentGhostsetfiles: - unnamed
nosy: r.david.murray, SilentGhost, Arve.Knudsen
2011-02-07 18:12:17Arve.Knudsensetfiles: + unnamed

messages: + msg128144
nosy: r.david.murray, SilentGhost, Arve.Knudsen
2011-02-07 17:06:07r.david.murraysetnosy: r.david.murray, SilentGhost, Arve.Knudsen
messages: + msg128139
2011-02-07 16:39:24Arve.Knudsensetfiles: + unnamed

messages: + msg128135
nosy: r.david.murray, SilentGhost, Arve.Knudsen
2011-02-07 15:38:30r.david.murraysetnosy: + r.david.murray
messages: + msg128129
resolution: duplicate -> not a bug

superseder: subprocess.list2cmdline doesn't quote the & character
stage: committed/rejected
2011-02-07 15:33:36SilentGhostsetstatus: open -> closed

nosy: + SilentGhost
messages: + msg128125

resolution: duplicate
2011-02-07 15:07:33SilentGhostsetversions: + Python 3.1, Python 3.2, Python 3.3
2011-02-07 14:49:36Arve.Knudsencreate