classification
Title: os.exec* mangles argv on windows (splits on spaces, etc)
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, Saimadhav.Heblikar, The Compiler, loewis, r.david.murray, rhettinger, steve.dower, tim.golden, tim.peters, zach.ware
Priority: normal Keywords: patch

Created on 2014-01-31 07:20 by The Compiler, last changed 2014-07-15 06:14 by rhettinger. This issue is now closed.

Files
File name Uploaded Description Edit
issue20451.patch Saimadhav.Heblikar, 2014-02-02 06:10 Escapes all args if the platform is NT.Does not modify behavior on non-NT review
Messages (9)
msg209748 - (view) Author: Florian Bruhin (The Compiler) * Date: 2014-01-31 07:20
The os.exec* functions seem to mangle arguments on Windows. So far I noticed the supplied argv gets split on spaces, and double-quotes get stripped when not escaped.

Example, on Windows 7:

>>> platform.platform()
'Windows-7-6.1.7601-SP1'
>>> os.execlp('python', 'python', '-c', "sys=__import__('sys');print(sys.argv)", 'Hello World')
['-c', 'Hello', 'World']

Same on Archlinux: ['-c', 'Hello World'] as expected.

Both running Python 3.3.3.
msg209952 - (view) Author: Saimadhav Heblikar (Saimadhav.Heblikar) * Date: 2014-02-02 06:10
i have tried a workaround for this issue by explicitly escaping args so that same result is produced on all platforms.this patch does NOT change the behavior on non-NT platforms.
If this patch is accepted,i also recommend to specify on the help pages,that there is no need to escape the args as it will done automatically.
msg210180 - (view) Author: Florian Bruhin (The Compiler) * Date: 2014-02-04 07:43
I can't test this right now, but I'd guess you also have to escape a " inside an argument (with \" presumably), and a \ with \\. Maybe more, no idea.

Actually the documentation doesn't say anything about me _having_ to escape anything, so I'd assume I don't have to. I'm calling a python function after all, and not using a shell.
msg210230 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-02-04 14:01
I believe the problem lies with the way that Windows implements the 'exec' functions.  Windows isn't posix, and sometimes its attempts to fake it go rather badly.  So, I'm not sure what the actual rules are, but whatever they are there should at least be a mention/pointer in the documentation about it.

Really, if you want to be cross platform you should use subprocess.  exec doesn't really even exec (replace the current process) on windows, if I understand correctly.

By the way, -c accepts strings with embedded newlines, something I didn't know for a long time :)
msg222969 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-13 19:41
I believe the patch on #1576120 is related to this but it was never reviewed.
msg222990 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-07-14 01:53
> I believe the problem lies with the way that Windows implements 
> the 'exec' functions.

Yes, that is that cause of the observed behavior.  See the first note in the remarks section on the relevant MSDN entry at http://msdn.microsoft.com/en-us/library/431x4c1w.aspx :

"""
Spaces embedded in strings may cause unexpected behavior; for example, passing _exec the string "hi there" will result in the new process getting two arguments, "hi" and "there". If the intent was to have the new process open a file named "hi there", the process would fail. You can avoid this by quoting the string: "\"hi there\"".
"""

The 'exec' functions are more low level and more arcane than most people would like.  Python exposes that functionality with all its beauty and all its warts.  David Murray is right in saying that you likely need one of the higher level modules such as subprocess with shell=True.

I looked through the code in Modules/posixmodule.c and didn't any code on our part that exacerbates the behavior.

Knowing that the cause is the underlying function still leaves the question of whether to apply a patch the mitigates the problem (possibly breaking some code that relies on the existing behavior) or whether to live with the facts-of-life that accompany low level O/S functions.
msg222993 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-07-14 03:30
I no longer have a Windows box to test this but any proposed patch needs to make sure it doesn't break code that is already manually escaping the inputs.
msg223034 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-07-14 15:30
Automatically quoting arguments is more complicated than simply concatenating quotes, unfortunately, and people are guaranteed to have come up with ways to make it work already.

My vote is for leaving this alone and letting the higher level functions be more clever. Changing this function is certain to break existing code in unpredictable ways.
msg223089 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-07-15 06:14
> My vote is for leaving this alone and letting the higher level
> functions be more clever. Changing this function is certain to 
> break existing code in unpredictable ways.

That is sensible.  Marking this as closed.
History
Date User Action Args
2014-07-15 06:14:44rhettingersetstatus: open -> closed
resolution: wont fix
messages: + msg223089
2014-07-14 15:30:09steve.dowersetmessages: + msg223034
2014-07-14 03:30:56rhettingersetnosy: + tim.peters, loewis, tim.golden, steve.dower
messages: + msg222993
2014-07-14 01:54:06rhettingersetmessages: - msg222989
2014-07-14 01:53:55rhettingersetmessages: + msg222990
2014-07-14 01:50:16rhettingersetnosy: + rhettinger
messages: + msg222989
2014-07-13 19:41:08BreamoreBoysetnosy: + BreamoreBoy

messages: + msg222969
versions: + Python 3.4, Python 3.5, - Python 3.3
2014-02-06 22:02:20zach.waresetnosy: + zach.ware
2014-02-04 14:01:49r.david.murraysetmessages: + msg210230
2014-02-04 07:43:09The Compilersetmessages: + msg210180
2014-02-02 06:10:49Saimadhav.Heblikarsetfiles: + issue20451.patch

nosy: + Saimadhav.Heblikar
messages: + msg209952

keywords: + patch
2014-01-31 08:32:19r.david.murraysetnosy: + r.david.murray
2014-01-31 07:20:39The Compilercreate