classification
Title: os.pipe should return inheritable descriptors (Windows)
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Daniel.Goertzen, castironpi, christian.heimes, ggenellina, haypo, jnoller, sbt, tim.golden
Priority: normal Keywords: patch

Created on 2008-12-21 02:15 by castironpi, last changed 2014-02-13 13:52 by haypo. This issue is now closed.

Files
File name Uploaded Description Edit
os_pipe_test.py castironpi, 2008-12-21 02:15 test showing subprocess inheriting handle
inheritable_pipes.diff ggenellina, 2008-12-26 23:57 patch to posixmodule.c including test case and documentation updates. review
Messages (10)
msg78136 - (view) Author: Aaron Brady (castironpi) Date: 2008-12-21 02:15
os.pipe should return inheritable descriptors on Windows.

Patch below, test attached.  New pipe() returns descriptors, which
cannot be inherited.  However, their permissions are set correctly, so
msvcrt.get_osfhandle and msvcrt.open_osfhandle can be used to obtain an
inheritable handle.

Docs should contain a note to the effect.  'On Windows, use
msvcrt.get_osfhandle to obtain a handle to the descriptor which can be
inherited.  In a subprocess, use msvcrt.open_osfhandle to obtain a new
corresponding descriptor.'

--- posixmodule_orig.c  2008-12-20 20:01:38.296875000 -0600
+++ posixmodule_new.c   2008-12-20 20:01:54.359375000 -0600
@@ -6481,8 +6481,12 @@
        HANDLE read, write;
        int read_fd, write_fd;
        BOOL ok;
+       SECURITY_ATTRIBUTES sAttribs;
        Py_BEGIN_ALLOW_THREADS
-       ok = CreatePipe(&read, &write, NULL, 0);
+       sAttribs.nLength = sizeof( sAttribs );
+       sAttribs.lpSecurityDescriptor = NULL;
+       sAttribs.bInheritHandle = TRUE;
+       ok = CreatePipe(&read, &write, &sAttribs, 0);
        Py_END_ALLOW_THREADS
        if (!ok)
                return win32_error("CreatePipe", NULL);
msg78166 - (view) Author: Gabriel Genellina (ggenellina) Date: 2008-12-22 03:03
From the thread in c.l.p: 

Pros (of changing os.pipe() to return inheritable pipes):
 
- as it isn't explicitely documented whether os.pipe() returns 
inheritable pipes or not, both versions are "right" according to the 
documentation.

- if someone relies on pipes being non-inheritable on Windows (why?), 
this is undocumented behaviour, and Python has the right to change it.

- would improve POSIX compatibility, it mimics what os.pipe()
does on those OS

- inheritable pipes are less surprising for guys coming from other OS

- inheritable pipes are a lot more useful than non-inheritable ones 
when doing IPC (probably its main usage).
 
Cons:
 
- os.pipe has behaved that way since long time ago.

- some programs *might* break, if they relied on pipes being 
non-inheritable on Windows, even if that was undocumented behaviour.
msg78321 - (view) Author: Gabriel Genellina (ggenellina) Date: 2008-12-26 23:57
Patch to posixmodule.c including test case and documentation updates.
Note: I've only run the tests on Windows.
msg79844 - (view) Author: Aaron Brady (castironpi) Date: 2009-01-14 08:41
This is currently accomplished in 'multiprocessing.forking' with a
'duplicate' function.

Use (line #213):
    rfd, wfd = os.pipe()
    
    # get handle for read end of the pipe and make it inheritable
    rhandle = duplicate(msvcrt.get_osfhandle(rfd), inheritable=True)

Definition (line #192).

Should it be included in the public interface and documented, or perhaps
a public entry point to it made?
msg192660 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-08 15:23
Victor, this fits nicely with your recent PEP.
msg192733 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-07-09 09:40
> - would improve POSIX compatibility, it mimics what os.pipe()
> does on those OS

I disagree.

On Windows fds can only be inherited if you start processes using the spanwn*() family of functions.  If you start them using CreateProcess() then the underlying *handles* are inherited, but the *fds* are not.

In Python 2, os.spawn*() used spawn*(), so making os.pipe() return inheritable fds would have made some sense.  But in Python 3 os.spawn*() is implemented using subprocess/CreateProcess so fds will NOT be inherited (even if the wrapped handles are).

Note that subprocess *does* know how to redirect the standard streams to fds returned by os.pipe().

So for Python 3 I don't think there is any point in changing things.
msg192736 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-07-09 10:04
Oops.  I confused os.popen() with os.spawn*().  os.spawnv() IS still implemented using spawnv() in Python 3.
msg193637 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-24 12:00
Changing the default inheritance value of os.pipe() is not acceptable because it would break backward compatibility.

Giving access to Windows extra parameter is nice, but we have to find a way to propose a portable API. That's what I'm trying to do with the PEP 446 (and it's previous version, the PEP 433).
msg194775 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-08-09 23:15
os.pipe() creates non-inheritable pipes on Windows, whereas it creates inheritable pipes on UNIX. IMO the reason is an implementation artifact: os.pipe() calls CreatePipe() on Windows (native API), whereas it calls pipe() on UNIX (POSIX API). The call to CreatePipe() was added in Python in 1994, before the introduction of pipe() in the POSIX API in Windows 98.
msg211156 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-02-13 13:52
The PEP 446 has been implemented in Python 3.4 and all file descriptors and sockets are now created non-inheritable by default. Use os.set_inheritable() to make the pipe fds inheritable.
History
Date User Action Args
2014-02-13 13:52:42hayposetstatus: open -> closed
resolution: fixed
messages: + msg211156
2013-08-09 23:15:38hayposetmessages: + msg194775
2013-07-31 14:40:47tim.goldensetassignee: tim.golden ->
2013-07-24 12:00:55hayposetmessages: + msg193637
2013-07-09 10:04:52sbtsetmessages: + msg192736
2013-07-09 09:40:09sbtsetmessages: + msg192733
2013-07-09 01:16:05pitrousetnosy: + sbt
2013-07-08 15:23:45christian.heimessetnosy: + christian.heimes, haypo

messages: + msg192660
versions: + Python 3.4, - Python 2.6, Python 3.0, Python 3.1, Python 2.7
2010-08-11 12:37:31Daniel.Goertzensetnosy: + Daniel.Goertzen
2010-08-06 15:07:27tim.goldensetassignee: tim.golden

nosy: + tim.golden
2009-01-19 16:38:54jnollersetnosy: + jnoller
2009-01-14 08:41:50castironpisetmessages: + msg79844
2008-12-26 23:57:33ggenellinasetfiles: + inheritable_pipes.diff
keywords: + patch
messages: + msg78321
2008-12-22 03:03:24ggenellinasetnosy: + ggenellina
messages: + msg78166
2008-12-21 02:15:24castironpicreate