classification
Title: os.execv fails with spaced names on Windows
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: loewis, techtonik, terry.reedy
Priority: normal Keywords:

Created on 2013-09-22 02:21 by techtonik, last changed 2014-07-02 22:26 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
testexecv.py techtonik, 2013-09-22 02:21
Messages (14)
msg198242 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-22 02:21
If file to be executed with os.execv on Windows is located in directory with spaces, Python fails. This doesn't fail on Linux. To test, run:

  testexecv.py spaced

testexecv.py is attached.
msg198546 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-28 19:39
Please stop marking all versions. Use cntl-left click to select the appropriate ones (currently at most 2.7, 3.3, 3.4 for users). Which version did you run your file with?
msg198568 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 07:13
I tested with 2.7 and 3.3, but this is true for any version.

If the bug is actual for Python 2.6, 3.1 and 3.2 why should I uncheck them? Versions field description doesn't say that I should mark only latest change. In addition, people (unlikely, but still) may search for specific versions to see which bugs were reported against them and fixed in later releases.
msg198569 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-29 08:28
The versions are the versions that will be patched for the issue.
msg198574 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 09:34
It should be documented somehow then. At least in the field tooltip.
msg198580 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 11:05
See

    http://bugs.python.org/issue436259

This is a problem with Window's implementation of spawn*() and exec*().  Just use subprocess instead which gets this stuff right.

Note that on Windows exec*() is useless: it just starts a subprocess and exits the current process.  You can use subprocess to get the same effect.
msg198581 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 11:18
On Sun, Sep 29, 2013 at 2:05 PM, Richard Oudkerk <report@bugs.python.org> wrote:
> See  http://bugs.python.org/issue436259

I am not sure that I should see there. There is discussion of DOS,
which is not supported, also some complain about Windows execv
function, which deprecated since VC++ 2005 (which I hope also not
supported). Can you be more specific?

> This is a problem with Window's implementation of spawn*() and exec*().

> Note that on Windows exec*() is useless: it just starts a subprocess and exits the current process.  You can use subprocess to get the same effect.

Are you describing Windows implementation of _exec()
http://msdn.microsoft.com/en-us/library/431x4c1w.aspx or current
Python implementation?

> Just use subprocess instead which gets this stuff right.

subprocess doesn't replace os.exec*, see issue19060
msg198587 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 12:03
> I am not sure that I should see there. There is discussion of DOS,
> which is not supported, also some complain about Windows execv
> function, which deprecated since VC++ 2005 (which I hope also not
> supported). Can you be more specific?

_spawn*() and _exec*() are implemented by the C runtime library.  spawn*() and execv() are (deprecated) aliases.

The the first message is about someone's attempt to work around the problems with embedded spaces and double quotes by writing a function to escape each argument.  He says he had a partial success.

Surely this is basic reading comprehension?

> > Note that on Windows exec*() is useless: it just starts a subprocess and 
> > exits the current process.  You can use subprocess to get the same effect.
>
> Are you describing Windows implementation of _exec()
> http://msdn.microsoft.com/en-us/library/431x4c1w.aspx or current
> Python implementation?

The Windows implementaion of _exec().

> > Just use subprocess instead which gets this stuff right.
>
> subprocess doesn't replace os.exec*, see issue19060

On Unix subprocess does not replace os.exec*().  That is because on Unix exec*() replaces the current process with a new process with the *same pid*.  subprocess cannot do this.

But on Windows os.exec*() just starts an independent process with a *different pid* and exits the current process.  The line

    os.execv(path, args)

is equivalent to

    os.spawnv(os.P_NOWAIT, path, args)
    os._exit(0)
msg198606 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 17:09
On Sun, Sep 29, 2013 at 3:03 PM, Richard Oudkerk <report@bugs.python.org> wrote:
>
> _spawn*() and _exec*() are implemented by the C runtime library.  spawn*() and execv() are (deprecated) aliases.

It is said that execv() is deprecated, but it is not said that it is
alias of _execv(). It is only said that _execv() is C++ compliant.
http://msdn.microsoft.com/en-us/library/ms235416(v=vs.90).aspx

> The the first message is about someone's attempt to work around the problems with embedded spaces and double quotes by writing a function to escape each argument.  He says he had a partial success.

Don't we have such function already? I don't see the problem in
quoting the string.

> Surely this is basic reading comprehension?

I am mentally crippled. Sorry about that.

>> > Note that on Windows exec*() is useless: it just starts a subprocess and
>> > exits the current process.  You can use subprocess to get the same effect.
>>
>> Are you describing Windows implementation of _exec()
>> http://msdn.microsoft.com/en-us/library/431x4c1w.aspx or current
>> Python implementation?
>
> The Windows implementaion of _exec().

Does it start child process in foreground or in background? Did you
compile examples on
http://msdn.microsoft.com/en-us/library/431x4c1w.aspx page with new
VC++ to check? I don't possess the VC++ 10, so I can't do this myself.
And I believe that compiling with GCC may lead to different results.

>> > Just use subprocess instead which gets this stuff right.
>>
>> subprocess doesn't replace os.exec*, see issue19060
>
> On Unix subprocess does not replace os.exec*().  That is because on Unix exec*() replaces the current process with a new process with the *same pid*.  subprocess cannot do this.
>
> But on Windows os.exec*() just starts an independent process with a *different pid* and exits the current process.  The line
>
>     os.execv(path, args)
>
> is equivalent to
>
>     os.spawnv(os.P_NOWAIT, path, args)
>     os._exit(0)

I don't mind if it runs child process with different pid, but why it
runs new process in background. Unix version doesn't do this.
msg198614 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 17:48
> It is said that execv() is deprecated, but it is not said that it is
> alias of _execv(). It is only said that _execv() is C++ compliant.
> http://msdn.microsoft.com/en-us/library/ms235416(v=vs.90).aspx

Microsoft seems to have decided that all functions in the C runtime which don't begin with an underscore, and are not included in the ANSI C standard should be deprecated.  This includes all the fd functions like read(), write(), open(), close(), ...  There is no difference in behaviour between these and the underscore versions.

...

> Don't we have such function already? I don't see the problem in
> quoting the string.

No one seems to know how to write such a quoting function.

...

> Does it start child process in foreground or in background? Did you
> compile examples on
> http://msdn.microsoft.com/en-us/library/431x4c1w.aspx page with new
> VC++ to check? I don't possess the VC++ 10, so I can't do this myself.
> And I believe that compiling with GCC may lead to different results.

There is no such thing as a background task in Windows.  A process is either attached to a console, or it isn't.  When you use execv() to start a process, it inherits the parent's console.

On Unix try replacing os.execv(...) by

    os.spawnv(os.P_NOWAIT, ...)
    os._exit(0)

and you will probably get the same behaviour where the shell and the child process both behave as conflicting foreground tasks.

..

> I don't mind if it runs child process with different pid, but why it
> runs new process in background. Unix version doesn't do this.

The point is that the shell waits for its child process to finish by using waitpid() (or something similar) on the child's pid.  If the child uses execv() then the child is replaced by a grandchild process with the same pid.  From the point of view of the shell, the child and the grandchild are the same process, and waitpid() will not stop until the grandchild terminates.

This issue should be closed: just use subprocess instead.
msg198625 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 18:49
Hey. This ticket is about os.execv failing on spaced paths on Windows. It is not a duplicate of issue19124.
msg198626 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 19:03
On Sun, Sep 29, 2013 at 8:48 PM, Richard Oudkerk <report@bugs.python.org> wrote:
>
>> Don't we have such function already? I don't see the problem in
>> quoting the string.
>
> No one seems to know how to write such a quoting function.

Why escape quotes with slash and surrounding in quotes doesn't help?
http://msdn.microsoft.com/en-us/library/ms235416(v=vs.90).aspx
How does Linux and subprocess on Windows survive then?
If I am not mistaken both subprocess and execv on Windows use
CreateProcess. Does subprocess fail as well?

>> Does it start child process in foreground or in background? Did you
>> compile examples on
>> http://msdn.microsoft.com/en-us/library/431x4c1w.aspx page with new
>> VC++ to check? I don't possess the VC++ 10, so I can't do this myself.
>> And I believe that compiling with GCC may lead to different results.
>
> There is no such thing as a background task in Windows.  A process is either attached to a console, or it isn't.  When you use execv() to start a process, it inherits the parent's console.

All right. Then why does it start to interfere with running cmd.exe
(in issue19124)? If it inherits console, it should continue to "own"
it exclusively, and not return it back to parent cmd.exe

> On Unix try replacing os.execv(...) by
>
>     os.spawnv(os.P_NOWAIT, ...)
>     os._exit(0)
>
> and you will probably get the same behaviour where the shell and the child process both behave as conflicting foreground tasks.

Maybe Python code doesn't use _execv() at all on Windows and uses
these spawnv's?

> ..
>
>> I don't mind if it runs child process with different pid, but why it
>> runs new process in background. Unix version doesn't do this.
>
> The point is that the shell waits for its child process to finish by using waitpid() (or something similar) on the child's pid.  If the child uses execv() then the child is replaced by a grandchild process with the same pid.  From the point of view of the shell, the child and the grandchild are the same process, and waitpid() will not stop until the grandchild terminates.

I can not accept your point when you don't know for sure how cmd.exe
waits for child process to exit. Are you sure that it doesn't use some
blocking CreateProcess call? Are you sure that Python on Windows calls
exactly _execv and not some spawn surrogate?

> This issue should be closed: just use subprocess instead.

We need some quorum on this. I'd like to hear two more people that can
confirm and agree with your position. I don't want to think that
usability of execv() on Windows can not be improved, because people
who love Linux doesn't think that this OS deserves some care. I'd like
to run Python scripts with the same base behaviour regardless of
platform. If that's impossible, that should be documented.
msg198637 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 19:53
> Hey. This ticket is about os.execv failing on spaced paths on Windows. It 
> is not a duplicate of issue19124.

It is a duplicate of #436259 "[Windows] exec*/spawn* problem with spaces in args".
msg198644 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-29 20:17
Reminder: you are not supposed to re-open issues.

I agree with Richard that this is a duplicate as submitted. The difference is that we now have subprocess; that is our fix for several problems. I will re-close unless you make a *specific* suggestion for a doc change.
History
Date User Action Args
2014-07-02 22:26:30terry.reedysetstatus: open -> closed
resolution: duplicate
2013-09-29 20:17:09terry.reedysetmessages: + msg198644
2013-09-29 19:54:21sbtsetnosy: - sbt
2013-09-29 19:53:00sbtsetnosy: loewis, terry.reedy, techtonik, sbt
messages: + msg198637
2013-09-29 19:03:26techtoniksetmessages: + msg198626
2013-09-29 18:49:54techtoniksetstatus: closed -> open
resolution: duplicate -> (no value)
messages: + msg198625
2013-09-29 17:48:49sbtsetstatus: open -> closed
resolution: duplicate
messages: + msg198614

stage: test needed -> resolved
2013-09-29 17:09:00techtoniksetmessages: + msg198606
2013-09-29 12:03:26sbtsetmessages: + msg198587
2013-09-29 11:18:26techtoniksetmessages: + msg198581
title: os.execv fails with spaced names on Windows -> os.execv fails with spaced names on Windows
2013-09-29 11:05:17sbtsetnosy: + sbt
messages: + msg198580
2013-09-29 09:34:59techtoniksetmessages: + msg198574
2013-09-29 08:28:15terry.reedysetmessages: + msg198569
2013-09-29 07:13:55techtoniksetmessages: + msg198568
2013-09-28 19:39:20terry.reedysetversions: - Python 2.6, Python 3.1, Python 3.2, Python 3.5
nosy: + loewis, terry.reedy

messages: + msg198546

type: behavior
stage: test needed
2013-09-22 02:21:28techtonikcreate