classification
Title: test_popen fails when the directory contains a space
Type: behavior Stage: resolved
Components: Versions: Python 3.0
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: JosephArmbruster, astrand, christian.heimes, ggenellina, joearmbruster, squeegee, tim.golden
Priority: normal Keywords:

Created on 2007-11-20 07:36 by christian.heimes, last changed 2010-08-12 15:00 by tim.golden. This issue is now closed.

Files
File name Uploaded Description Edit
subprocesspatch.patch JosephArmbruster, 2007-11-20 23:20
Messages (10)
msg57696 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-20 07:36
Joseph found the bug during his testing of the py3k branch. The problem
is in subprocess.py line 716.

http://pastebin.com/fa947767
msg57701 - (view) Author: Joseph Armbruster (JosephArmbruster) Date: 2007-11-20 12:11
I believe the issue lies with the cmd command line parameters and
insufficient quoting:

Currently, if this string is passed into CreateProcess as args, the call
will fail:

C:\WINDOWS\System32\cmd.exe /c "C:\Documents and
Settings\joe\Desktop\Development\Python3k\dev\pcbuild9\python.exe" -c
"import sys
; print(sys.argv)" foo bar

For sanity, when I try to execute this from a command prompt manually, I
noticed this behavior:

C:\Documents and Settings\joe>C:\WINDOWS\System32\cmd.exe /c
"C:\Documents and Settings\joe\Desktop\Development\Python3k\dev\pcbui
ld9\python.exe" -c "import sys; print(sys.argv)" foo bar
'C:\Documents' is not recognized as an internal or external command,
operable program or batch file.

I read through cmd.exe and it is has this note:

"""
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:

1.  If all of the following conditions are met, then quote characters
    on the command line are preserved:

    - no /S switch
    - exactly two quote characters
    - no special characters between the two quote characters,
      where special is one of: &<>()@^|
    - there are one or more whitespace characters between the
      the two quote characters
    - the string between the two quote characters is the name
      of an executable file.

2.  Otherwise, old behavior is to see if the first character is
    a quote character and if so, strip the leading character and
    remove the last quote character on the command line, preserving
    any text after the last quote character.
"""

I believe args falls under section 2.
msg57717 - (view) Author: Joseph Armbruster (JosephArmbruster) Date: 2007-11-20 23:20
Applying quotations around args within the shell case appears to take
care of it, any thoughts?
msg57726 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-21 04:11
I like to have Peter Astrand look over the patch first. He has written
most of the subprocess module.
msg57731 - (view) Author: Peter ├ůstrand (astrand) * (Python committer) Date: 2007-11-21 07:48
I think there's some confusion in this bug. The report on
http://pastebin.com/fa947767 indicates a problem in test_popen. This is
a test for os.popen() and it does not have anything to do with the
subprocess module. I believe it is test_popen.py that should be fixed. 

In general, quoting in Windows is very difficult. The documentation
quoted in msg57701 might be correct for some modern version of Windows,
but not for, say, command.com on an older Windows version. I believe the
subprocess is fairly complete when it comes to quoting etc, so changes
should be avoided if possible. 

Since this bug is not about the subprocess module, I've re-assigned to
nobody, hope this is OK.
msg57732 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-21 08:14
In Python 3.x os.popen is implemented based on subprocess. I believe
it's still a problem with subprocess. Python 3.x also drops support for
Windows 95 to ME. Would the additional quoting be ok when the code
checks for COMPSPEC == "cmd.exe" first?

# Supply os.popen()
def popen(cmd, mode="r", buffering=None):
    if not isinstance(cmd, str):
        raise TypeError("invalid cmd type (%s, expected string)" %
type(cmd))
    if mode not in ("r", "w"):
        raise ValueError("invalid mode %r" % mode)
    import subprocess, io
    if mode == "r":
        proc = subprocess.Popen(cmd,
                                shell=True,
                                stdout=subprocess.PIPE,
                                bufsize=buffering)
        return _wrap_close(io.TextIOWrapper(proc.stdout), proc)
    else:
        proc = subprocess.Popen(cmd,
                                shell=True,
                                stdin=subprocess.PIPE,
                                bufsize=buffering)
        return _wrap_close(io.TextIOWrapper(proc.stdin), proc)
msg57765 - (view) Author: Peter ├ůstrand (astrand) * (Python committer) Date: 2007-11-22 18:54
>In Python 3.x os.popen is implemented based on subprocess. 

Oh, I see. 

>I believe it's still a problem with subprocess. 

I'm still not convinced of this. Isn't it better to do the quoting
outside subprocess; to let the caller do it? You say that "section 2" is 
our case, but how can we be sure of this: If subprocess is called with
args='"c:\program files\internet explorer\iexplore.exe"', then we have
case 1, right, and surrounding args with another pair of quotes would
mean failure, right?
msg63440 - (view) Author: Joseph Armbruster (JosephArmbruster) Date: 2008-03-10 12:55
Should this issue be linked to 1559298?  It appears to be the same issue
but was opened up earlier and is referencing a different version.
msg90105 - (view) Author: Russ Gibson (squeegee) Date: 2009-07-04 04:19
(Same comment I added to 1559298:)
What is needed is separate quoting for the command and the argument. 
Right now, test_popen only surrounds the entire command line with quotes:

"c:\Program Files\Python2.6\Python.exe -u c:\Documents and Settings\Russ
Gibson\cgi-bin\cgi.py"

It needs to test the above as thus:

"c:\Program Files\Python2.6\Python.exe" -u "c:\Documents and
Settings\Russ Gibson\cgi-bin\cgi.py"

From the command line, the second case works, but the first doesn't.
Neither work from os.popen in 2.6.2.  I have been unable to get popen*
to work with a command and an argument that both contain spaces.

To make sure that popen works in all cases under windows, test_popen*
needs to test with separate quotes around the command and argument. 
That will show the real problem with the current os.popen*
implementation(s).

(yeah, I know, shut up and provide a patch...)
msg113683 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2010-08-12 15:00
Fixed by issue2304
History
Date User Action Args
2010-08-12 15:00:08tim.goldensetstatus: open -> closed

type: behavior
components: - Library (Lib), Tests, Windows

nosy: + tim.golden
messages: + msg113683
resolution: duplicate
stage: resolved
2009-07-04 04:19:05squeegeesetnosy: + squeegee
messages: + msg90105
2008-03-10 12:55:23JosephArmbrustersetmessages: + msg63440
2008-01-06 22:29:44adminsetkeywords: - py3k
versions: Python 3.0
2007-11-22 18:54:26astrandsetmessages: + msg57765
2007-11-22 06:45:39ggenellinasetnosy: + ggenellina
2007-11-21 08:14:30christian.heimessetmessages: + msg57732
2007-11-21 07:48:02astrandsetassignee: astrand ->
messages: + msg57731
2007-11-21 04:11:11christian.heimessetmessages: + msg57726
2007-11-20 23:20:58JosephArmbrustersetfiles: + subprocesspatch.patch
messages: + msg57717
2007-11-20 12:11:15JosephArmbrustersetnosy: + JosephArmbruster
messages: + msg57701
2007-11-20 07:37:13christian.heimessetpriority: normal
2007-11-20 07:36:12christian.heimescreate