Author eryksun
Recipients eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Date 2018-02-17.15:51:53
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1518882714.97.0.467229070634.issue32865@psf.upfronthosting.co.za>
In-reply-to
Content
File descriptors in Windows are implemented by the C runtime library's low I/O layer. The CRT maps native File handles to Unix-style file descriptors. Additionally, in order to support inheritance for spawn/exec, the CRT passes inheritable FDs in a reserved field of the CreateProcess STARTUPINFO record. A spawned child process uses this information to initialize its FD table at startup.

Python's implementation of os.pipe creates File handles directly via CreatePipe. These handles are non-inheritable, which we want. However, it wraps them with inheritable file descriptors via _open_osfhandle, which we don't want. The solution is to include the flag _O_NOINHERIT in the _open_osfhandle call, which works even though this flag isn't explicitly documented as supported for this function.

Here's an example of the issue.

    >>> fdr, fdw = os.pipe()
    >>> fdr, fdw
    (3, 4)
    >>> msvcrt.get_osfhandle(3), msvcrt.get_osfhandle(4)
    (440, 444)
    >>> os.get_handle_inheritable(440)
    False
    >>> os.get_handle_inheritable(444)
    False

Note that os.get_inheritable assumes that FD and handle heritability are in sync, so it only queries handle information. The following FDs are actually inheritable, while the underlying File handles are not:

    >>> os.get_inheritable(3)
    False
    >>> os.get_inheritable(4)
    False

This is a flawed assumption baked into _Py_get_inheritable and _Py_set_inheritable, especially since the latter has no effect on FD heritability. The CRT has no public functions to query and modify the heritability of existing FDs. Until it does (and maybe Steve Dower can request this from the MSVC devs), I see no point in pretending something works when it doesn't. This only creates problems.

Let's spawn a child to show that these FDs are inherited in a bad state, which is a potential source of bugs and data corruption:

    >>> os.spawnl(os.P_WAIT, sys.executable, 'python')

    Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40)
    [MSC v.1900 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os, msvcrt
    >>> msvcrt.get_osfhandle(3), msvcrt.get_osfhandle(4)
    (440, 444)

As you can see, file descriptors 3 and 4 were inherited with the handle values from the parent process, but the handles themselves were not inherited. We'll be lucky if the handle values happen to be unassigned and remain so. However, they may be assigned or subsequently assigned to random kernel objects (e.g. a File, Job, Process, Thread, etc).

On a related note, Python always opens files as non-inheritable, even for os.open (which IMO makes no sense; we have O_NOINHERIT or O_CLOEXEC for that). It's assumed that the FD can be made inheritable via os.set_inheritable, but this does not work on Windows, and as things stand with the public CRT API, it cannot work. For example:

    >>> os.open('test.txt', os.O_WRONLY)
    3
    >>> msvcrt.get_osfhandle(3)
    460
    >>> os.set_inheritable(3, True)
    >>> os.get_handle_inheritable(460)
    True

    >>> os.spawnl(os.P_WAIT, sys.executable, 'python')

    Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40)
    [MSC v.1900 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import msvcrt
    >>> msvcrt.get_osfhandle(3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [Errno 9] Bad file descriptor

The Windows handle was of course inherited, but that's not useful in this scenario, since the CRT didn't know to pass the FD in the process STARTUPINFO. It's essentially a leaked handle in the child.
History
Date User Action Args
2018-02-17 15:51:55eryksunsetrecipients: + eryksun, paul.moore, tim.golden, zach.ware, steve.dower
2018-02-17 15:51:54eryksunsetmessageid: <1518882714.97.0.467229070634.issue32865@psf.upfronthosting.co.za>
2018-02-17 15:51:54eryksunlinkissue32865 messages
2018-02-17 15:51:53eryksuncreate