classification
Title: os.execv executes in background on Windows
Type: behavior Stage: needs patch
Components: Documentation, Windows Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eryksun, jwilk, paul.moore, steve.dower, techtonik, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2013-09-29 10:46 by techtonik, last changed 2018-05-16 01:55 by eryksun.

Files
File name Uploaded Description Edit
testexecvchild.py techtonik, 2013-09-29 10:48
Messages (10)
msg198578 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 10:46
os.execv() starts process in background on Windows. Because it inherits stdin/stdout handlers from the process that launched Python interpreter, this becomes a source of numerous weird bugs, from polluting the stdout stream of parent to completely blocking its input.

Example session on Windows. Open cmd.exe and run attached testexecvchild.py. It starts child process with execv(). Child pauses for 2 seconds during which I type 'echo "Hello"' and hit Enter.

With Python 3 is pollutes parent output after 3 seconds:
    >python testexecvchild.py
    
    >echo "Hello"
    "Hello"
    
    >Traceback (most recent call last):
    File "testexecvchild.py", line 7, in <module>
    raw_input('xxx')
    NameError: name 'raw_input' is not defined

With Python 2 the stdin of cmd.exe is blocked:
    >py testexecvchild.py
    
    >echo "Hello"
    "Hello"
    
    >xxxecho "Hello"
    "Hello"
    
    >echo "Hello"
      testexecvchild.py
      passed
    echo "Hello"
    "Hello"

The same behavior on Linux:

    $ python testexecvchild.py
    echo "Hello"
    xxx  testexecvchild.py
      passed
msg198579 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 10:47
s/same behavior/same command/
msg198599 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 15:39
As I wrote in http://bugs.python.org/issue19066, on Windows execv() is equivalent to

    os.spawnv(os.P_NOWAIT, ...)
    os._exit(0)

This means that control is returned to cmd when the child process *starts* (and afterwards you have cmd and the child connected to the same console).

On Unix control is returned to the shell only once the child process *ends*.

Although it might be less memory efficient, you would actually get something closer to Unix behaviour by replacing os.execv(...) with

    sts = os.spawnv(os.P_WAIT, ...)
    _exit(sts)

or

    sts = subprocess.call(...)
    _exit(sts)

This is why I said that execv() is useless on Windows and that you should just use subprocess instead.
msg198613 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 17:31
On Sun, Sep 29, 2013 at 6:39 PM, Richard Oudkerk <report@bugs.python.org> wrote:
>
> Richard Oudkerk added the comment:
>
> As I wrote in http://bugs.python.org/issue19066, on Windows execv() is equivalent to
>
>     os.spawnv(os.P_NOWAIT, ...)
>     os._exit(0)

Where did you get that info? MSDN is silent about that.
http://msdn.microsoft.com/en-us/library/886kc0as(v=vs.90).aspx

> This means that control is returned to cmd when the child process *starts* (and afterwards you have cmd and the child connected to the same console).
>
> On Unix control is returned to the shell only once the child process *ends*.

That was my conclusion also.

> Although it might be less memory efficient, you would actually get something closer to Unix behaviour by replacing os.execv(...) with
>
>     sts = os.spawnv(os.P_WAIT, ...)
>     _exit(sts)
>
> or
>
>     sts = subprocess.call(...)
>     _exit(sts)
>
> This is why I said that execv() is useless on Windows and that you should just use subprocess instead.

The problem is not in what I should or should not use. The problem
that existing scripts that work on Unix and use os.execv() to launch
interactive scripts, on Windows behave absolutely weird and unusable
behavior. I previously experienced this with SCons, but couldn't get
the reason. Now I experience this with basic Android development tools
and dug down to this. It is clearly a big mess from this side of
Windows.
msg198616 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 17:53
> Where did you get that info? MSDN is silent about that.
> http://msdn.microsoft.com/en-us/library/886kc0as(v=vs.90).aspx

Reading the source code for the C runtime included with Visual Studio.

> The problem is not in what I should or should not use. The problem
> that existing scripts that work on Unix and use os.execv() to launch
> interactive scripts, on Windows behave absolutely weird and unusable
> behavior. I previously experienced this with SCons, but couldn't get
> the reason. Now I experience this with basic Android development tools
> and dug down to this. It is clearly a big mess from this side of
> Windows.

As said before (more than once), os.exec*() is useless on Windows: just use subprocess.
msg198627 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 19:08
On Sun, Sep 29, 2013 at 8:53 PM, Richard Oudkerk <report@bugs.python.org> wrote:
>
> Richard Oudkerk added the comment:
>
>> Where did you get that info? MSDN is silent about that.
>> http://msdn.microsoft.com/en-us/library/886kc0as(v=vs.90).aspx
>
> Reading the source code for the C runtime included with Visual Studio.

Visual Studio 10+ ? Is it available somewhere for a reference?

>> The problem is not in what I should or should not use. The problem
>> that existing scripts that work on Unix and use os.execv() to launch
>> interactive scripts, on Windows behave absolutely weird and unusable
>> behavior. I previously experienced this with SCons, but couldn't get
>> the reason. Now I experience this with basic Android development tools
>> and dug down to this. It is clearly a big mess from this side of
>> Windows.
>
> As said before (more than once), os.exec*() is useless on Windows: just use subprocess.

I value your expert opinion, but to increase the bus factor, I can not
leave it without asking for reasons.

Have you tried to run examples provided by MSDN - do they exhibit the
same behavior as Python script I attached earlier and described in the
first message?
msg198628 - (view) Author: anatoly techtonik (techtonik) Date: 2013-09-29 19:10
I can't use subprocess. These are official "business suite" scripts for Android development from Google.
msg198638 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-09-29 19:53
> Visual Studio 10+ ? Is it available somewhere for a reference?

Old versions of the relevant files are here:

http://www.controllogics.com/software/VB6/VC98/CRT/SRC/EXECVE.C
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/SPAWNVE.C
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/DOSPAWN.C
msg198974 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-10-05 01:09
In general, os module functions lightly wrap the corresponding operating system calls. It does not mask differences between OSes, or between versions of an OS. So the unix-windows difference is not a bug and the behavior will not change. 

The difference could, however, be described succinctly in the doc. In the first paragraph, after "On Unix, the new executable is loaded into the current process, and will have the same process id as the caller.", we could add something like

On Windows, the new process is executed in the background but can send output to a console if the original process was started in a console.

This has to be worded carefully because the process could instead have been started from an icon, including in Start Menu or Windows Explorer.
msg316754 - (view) Author: Eryk Sun (eryksun) * Date: 2018-05-16 01:54
The exec functions provided by the Windows C runtime really are practically useless, due to creating an orphaned process, disrupting synchronous operation, and returning the wrong status code. It might be more useful for Python 3.7.x to implement an internal win32_execv[e] function that calls _wspawnv[e] with _P_NOWAIT mode. Assign the child process to a silent-breakaway, kill-on-close Job. Wait for it to end, and exit with the child's status code.
History
Date User Action Args
2018-05-16 01:55:52eryksunsetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2018-05-16 01:54:36eryksunsetnosy: + eryksun

messages: + msg316754
versions: + Python 3.7, Python 3.8, - Python 2.7, Python 3.3, Python 3.4
2018-05-15 16:58:35jwilksetnosy: + jwilk
2013-10-05 01:10:00terry.reedysetassignee: docs@python
components: + Documentation, - Library (Lib)
versions: + Python 3.4
nosy: + docs@python, terry.reedy

messages: + msg198974
stage: resolved -> needs patch
2013-09-29 19:54:28sbtsetnosy: - sbt
2013-09-29 19:53:59sbtsetnosy: techtonik, sbt
messages: + msg198638
2013-09-29 19:11:10techtoniksetstatus: closed -> open
resolution: rejected ->
2013-09-29 19:10:05techtoniksetmessages: + msg198628
2013-09-29 19:08:26techtoniksetmessages: + msg198627
2013-09-29 17:53:56sbtsetstatus: open -> closed
resolution: rejected
messages: + msg198616

stage: resolved
2013-09-29 17:31:36techtoniksetmessages: + msg198613
2013-09-29 15:39:51sbtsetnosy: + sbt
messages: + msg198599
2013-09-29 10:48:25techtoniksetfiles: + testexecvchild.py
2013-09-29 10:47:58techtoniksetmessages: + msg198579
2013-09-29 10:46:42techtonikcreate