This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author eryksun
Recipients Leonardo Francalanci, eryksun, martin.panter, r.david.murray
Date 2017-09-18.22:35:51
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1505774152.05.0.180373831426.issue31447@psf.upfronthosting.co.za>
In-reply-to
Content
> when I said "also with close_fds=True", I meant that I tried 
> WITHOUT overriding stdin, stdout, and stderr AND setting 
> close_fds=True, but it didn't work. 

Console applications (e.g. python.exe) duplicate their standard handles into a child console process when the bInheritHandles argument of CreateProcess is false, i.e. when subprocess.Popen is called with the default close_fds=True. We know it's duplication instead of inheritance because the actual handle values change and also because it doesn't matter whether or not the standard handles are flagged as inheritable in the creating process. In the typical case this ensures that console processes have valid standard handles. 

However, if you make the standard handles non-inheritable in the parent and also inherit handles when creating the child, then the parent's standard handles will be neither inherited nor duplicated. The child will still inherit the standard handle *values* from the parent (i.e. copied from the parent's PEB ProcessParameters). These handle values will be invalid. Either the handle won't be defined, or it will reference an arbitrary, unrelated kernel object.

> What worked was not overriding stdin/out/err and adding
>
> os.set_inheritable(0, False)
> os.set_inheritable(1, False)
> os.set_inheritable(2, False)
>
> before the call (no need to set close_fds)

In this case cmd.exe gets duplicated standard handles, but you've flagged these handles as non-inheritable. When cmd.exe calls CreateProcess to execute waitfor.exe, it uses bInheritHandles=TRUE. Consequently, its non-inheritable standard handles are neither inherited by nor duplicated to waitfor.exe. Let's attach a debugger to the waitfor.exe process and examine the standard handle values in its PEB ProcessParameters:

    0:003> ?? @$peb->ProcessParameters->StandardInput
    void * 0x00000000`00000008
    0:003> !handle 8
    Handle 8
      Type          Event

    0:003> ?? @$peb->ProcessParameters->StandardOutput
    void * 0x00000000`0000000c
    0:003> !handle c
    Handle c
      Type          WaitCompletionPacket

    0:003> ?? @$peb->ProcessParameters->StandardError
    void * 0x00000000`00000010
    0:003> !handle 10
    Handle 10
      Type          IoCompletion

waitfor.exe doesn't care that its standard handle values reference non-File objects, i.e. an Event, a WaitCompletionPacket, and an IoCompletion port. However, what if by coincidence one of its standard handle values is actually a valid File handle that's inherited for some other reason? That would be a right mess. That's why I suggested overriding the standard handles to the NUL device. For example:

    import subprocess

    script = r'''
    import os
    import subprocess

    os.set_inheritable(0, False)
    os.set_inheritable(1, False)
    os.set_inheritable(2, False)

    cmdline = 'cmd.exe /s /c waitfor.exe signal /t 200'
    subprocess.run(cmdline, timeout=4, stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    '''

    args = ['python.exe', '-c', script]
    proc = subprocess.Popen(args, encoding='ansi', stdin=subprocess.PIPE,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    proc_out, proc_err = proc.communicate(timeout=20)
    print('proc_out:', proc_out)
    print('proc_err:', proc_err)
History
Date User Action Args
2017-09-18 22:35:52eryksunsetrecipients: + eryksun, r.david.murray, martin.panter, Leonardo Francalanci
2017-09-18 22:35:52eryksunsetmessageid: <1505774152.05.0.180373831426.issue31447@psf.upfronthosting.co.za>
2017-09-18 22:35:52eryksunlinkissue31447 messages
2017-09-18 22:35:51eryksuncreate