This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing: garbage collector fails to GC Pipe() end when spawning child process
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: madmaze, sbt, spresse1
Priority: normal Keywords:

Created on 2013-06-02 17:11 by spresse1, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
bugon.tar.gz spresse1, 2013-06-02 17:11 Tarball holding example of functional and nonfunctional code.
bugon.tar.gz spresse1, 2013-06-02 20:03 Corrected indentation in nonfunctional.py
Messages (15)
msg190492 - (view) Author: (spresse1) Date: 2013-06-02 17:11
[Code demonstrating issue attached]

When overloading multiprocessing.Process and using pipes, a reference to a pipe spawned in the parent is not properly garbage collected in the child.  This causes the write end of the pipe to be held open with no reference to it in the child process, and therefore no way to close it.  Therefore, it can never throw EOFError.

Expected behavior:
1. Create a pipe with multiprocessing.Pipe(False)
2. Pass read end to a class which subclasses multiprocessing.Process
3. Close write end in parent process
4. Receive EOFError from read end

Actual behavior:
1. Create a pipe with multiprocessing.Pipe(False)
2. Pass read end to a class which subclasses multiprocessing.Process
3. Close write end in parent process
4. Never receive EOFError from read end

Examining the processes in /proc/[pid]/fds/ indicates that a write pipe is still open in the child process, though none should be.  Additionally, no write pipe is open in the parent process.  It is my belief that this is the write pipe spawned in the parent, and is remaining around incorrectly in the child, though there are no references to it.

Tested on 2.7.3 and 3.2.3
msg190493 - (view) Author: (spresse1) Date: 2013-06-02 17:41
Now also tested with source-built python 3.3.2.  Issue still exists, same example files.
msg190496 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-02 19:49
The way to deal with this is to pass the write end of the pipe to the child process so that the child process can explicitly close it -- there is no reason to expect garbage collection to make this happen automatically.

You don't explain the difference between functional.py and nonfunctional.py.  The most obvious thing is the fact that nonfunctional.py seems to have messed up indentation: you have a while loop in the class declaration instead of in the run() method.
msg190497 - (view) Author: (spresse1) Date: 2013-06-02 20:03
The difference is that nonfunctional.py does not pass the write end of the parent's pipe to the child.  functional.py does, and closes it immediately after breaking into a new process.  This is what you mentioned to me as a workaround.  Corrected code (for indentation) attached.

Why SHOULDN'T I expect this pipe to be closed automatically in the child?  Per the documentation for multiprocessing.Connection.close():
"This is called automatically when the connection is garbage collected."

The write end of that pipe goes out of scope and has no references in the child thread.  Therefore, per my understanding, it should be garbage collected (in the child thread).  Where am I wrong about this?
msg190502 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-02 20:38
> The write end of that pipe goes out of scope and has no references in the 
> child thread.  Therefore, per my understanding, it should be garbage 
> collected (in the child thread).  Where am I wrong about this?

The function which starts the child process by (indirectly) invoking os.fork() never gets a chance to finish in the child process, so nothing "goes out of scope".

Anyway, relying on garbage collection to close resources for you is always a bit dodgy.
msg190504 - (view) Author: (spresse1) Date: 2013-06-02 20:47
So you're telling me that when I spawn a new child process, I have to deal with the entirety of my parent process's memory staying around forever?  I would have expected this to call to fork(), which gives the child plenty of chance to clean up, then call exec() which loads the new executable.  Either that or the same instance of the python interpreter is used, just with the knowledge that it should execute the child function and then exit.  Keeping all the state that will never be used in the second case seems sloppy on the part of python.

The semantics in this case are much better if the pipe gets GC'd.  I see no reason my child process should have to know about pipe ends it never uses in order to close them.
msg190506 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-02 21:34
> So you're telling me that when I spawn a new child process, I have to 
> deal with the entirety of my parent process's memory staying around 
> forever?

With a copy-on-write implementation of fork() this quite likely to use less memory than starting a fresh process for the child process.  And it is certainly much faster.

> I would have expected this to call to fork(), which gives the child 
> plenty of chance to clean up, then call exec() which loads the new 
> executable.

There is an experimental branch (http://hg.python.org/sandbox/sbt) which optionally behaves like that.  Note that "clean up" means close all fds not explcitly passed, and has nothing to do with garbage collection.
msg190509 - (view) Author: (spresse1) Date: 2013-06-02 22:59
>> So you're telling me that when I spawn a new child process, I have to 
>> deal with the entirety of my parent process's memory staying around 
>> forever?
>
> With a copy-on-write implementation of fork() this quite likely to use 
> less memory than starting a fresh process for the child process.  And 
> it is certainly much faster.

Fair enough.

>> I would have expected this to call to fork(), which gives the child 
>> plenty of chance to clean up, then call exec() which loads the new 
>> executable.
>
> There is an experimental branch (http://hg.python.org/sandbox/sbt) 
> which optionally behaves like that.  Note that "clean up" means close 
> all fds not explcitly passed, and has nothing to do with garbage 
> collection.

I appreciate the pointer, but I am writing code intended for distribution - using an experimental branch isn't useful.

What I'm still trying to grasp is why Python explicitly leaves the parent processes info around in the child.  It seems like there is no benefit (besides, perhaps, speed) and that this choice leads to non-intuitive behavior - like this.
msg190511 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-02 23:45
> What I'm still trying to grasp is why Python explicitly leaves the
> parent processes info around in the child.  It seems like there is
> no benefit (besides, perhaps, speed) and that this choice leads to
> non-intuitive behavior - like this.

The Windows implementation does not use fork() but still exhibits the 
same behaviour in this respect (except in the experimental branch 
mentioned before).  The real issue is that fds/handles will get 
inherited by the child process unless you explicitly close them. 
(Actually on Windows you need to find a way to inject specific handles 
from the parent to child process).

The behaviour you call non-intuitive is natural to someone used to using 
fork() and pipes on Unix.  multiprocessing really started as a 
cross-platform work-around for the lack of fork() on Windows.

Using fork() is also a lot more flexible: many things that work fine on 
Unix will not work correctly on Windows because of pickle-issues.

The main problem with fork() is that forking a process with multiple 
threads can be problematic.
msg190512 - (view) Author: (spresse1) Date: 2013-06-03 00:02
I'm actually a nix programmer by trade, so I'm pretty familiar with that behavior =p  However, I'm also used to inheriting some way to refer to these fds, so that I can close them.  Perhaps I've just missed somewhere a call to ask the process for a list of open fds?  This would, to me, be an acceptable workaround - I could close all the fds I didn't wish to inherit.

Whats really bugging me is that it remains open and I can't fetch a reference.  If I could do either of these, I'd be happy.

Maybe this is more an issue with the semantics of multiprocessing?  In that this behavior is perfectly reasonable with os.fork() but makes some difficulty here.

Perhaps I really want to be implementing with os.fork().  Sigh, I was trying to save myself some effort...
msg190522 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-03 08:04
On 03/06/2013 1:02am, spresse1 wrote:
> Whats really bugging me is that it remains open and I can't fetch a reference.
> If I could do either of these, I'd be happy.
> ...
> Perhaps I really want to be implementing with os.fork().  Sigh, I was trying to
> save myself some effort...

I don't see how using os.fork() would make things any easier.  In either 
case you need to prepare a list of fds which the child process should 
close before it starts, or alternatively a list of fds *not* to close.

The real issue is that there is no way for multiprocessing (or 
os.fork()) to automatically infer which fds the child process is going 
to use: if don't explicitly close unneeded ones then the child process 
will inherit all of them.

It might be helpful if multiprocessing exposed a function to close all 
fds except those specified -- see close_all_fds_except() at

http://hg.python.org/sandbox/sbt/file/5d4397a38445/Lib/multiprocessing/popen_spawn_posix.py#l81

Remembering not to close stdout (fd=1) and stderr (fd=2), you could use 
it like

     def foo(reader):
         close_all_fds_except([1, 2, reader.fileno()])
         ...

     r, w = Pipe(False)
     p = Process(target=foo, args=(r,))
msg190544 - (view) Author: (spresse1) Date: 2013-06-03 14:07
> I don't see how using os.fork() would make things any easier.  In either 
> case you need to prepare a list of fds which the child process should 
> close before it starts, or alternatively a list of fds *not* to close.

With fork() I control where the processes diverge much more readily.  I could create the pipe in the main process, fork, close unnecessary fds, then call into the class that represents the operation of the subprocess.  (ie: do it the c way).  This way the class never needs to know about pipes it doesnt care about and I can ensure that unnecessary pipes get closed.  So I get the clean, understandable semantics I was after and my pipes get closed.  The only thing I lose is windows interoperability.

I could reimplement the close_all_fds_except() call (in straight python, using os.closerange()).  That seems like a reasonable solution, if a bit of a hack.  However, given that pipes are exposed by multiprocessing, it might make sense to try to get this function incorperated into the main version of it?

I also think that with introspection it would be possible for the subprocessing module to be aware of which file descriptors are still actively referenced.  (ie: 0,1,2 always referenced, introspect through objects in the child to see if they have the file.fileno() method) However, I can't state this as a certainty without going off and actually implementing such a version.  Additionally, I can make absolutely no promises as to the speed of this.  Perhaps, if it functioned, it would be an option one could turn on for cases like mine.
msg190546 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-03 14:33
On 03/06/2013 3:07pm, spresse1 wrote:
> I could reimplement the close_all_fds_except() call (in straight python, using
> os.closerange()).  That seems like a reasonable solution, if a bit of a hack.
> However, given that pipes are exposed by multiprocessing, it might make sense
> to try to get this function incorperated into the main version of it?

close_all_fds_except() is already pure python:

    try:
        MAXFD = os.sysconf("SC_OPEN_MAX")
    except:
        MAXFD = 256

    def close_all_fds_except(fds):
        fds = list(fds) + [-1, MAXFD]
        fds.sort()
        for i in range(len(fds) - 1):
            os.closerange(fds[i]+1, fds[i+1])

> I also think that with introspection it would be possible for the subprocessing
> module to be aware of which file descriptors are still actively referenced.
> (ie: 0,1,2 always referenced, introspect through objects in the child to see if
> they have the file.fileno() method) However, I can't state this as a certainty
> without going off and actually implementing such a version.  Additionally, I can
> make absolutely no promises as to the speed of this.  Perhaps, if it functioned,
> it would be an option one could turn on for cases like mine.

So you want a way to visit all objects directly or indirectly referenced 
by the process object, so you can check whether they have a fileno() 
method?  At the C level all object types which support GC define a 
tp_traverse function, so maybe that could be made available from pure 
Python.

But really, this sounds rather fragile.
msg190547 - (view) Author: (spresse1) Date: 2013-06-03 14:57
Oooh, thanks.  I'll use that.

> But really, this sounds rather fragile.

Absolutely.  I concur there is no good way to do this.
msg190550 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-03 17:03
Actually, you can use gc.get_referents(obj) which returns the direct children of obj (and is presumably implemented using tp_traverse).

I will close.
History
Date User Action Args
2022-04-11 14:57:46adminsetgithub: 62320
2013-06-03 17:03:50sbtsetstatus: open -> closed
resolution: rejected
messages: + msg190550

stage: resolved
2013-06-03 14:57:59spresse1setmessages: + msg190547
2013-06-03 14:33:24sbtsetmessages: + msg190546
2013-06-03 14:07:01spresse1setmessages: + msg190544
2013-06-03 08:05:00sbtsetmessages: + msg190522
2013-06-03 00:02:00spresse1setmessages: + msg190512
2013-06-02 23:45:16sbtsetmessages: + msg190511
2013-06-02 22:59:43spresse1setmessages: + msg190509
2013-06-02 21:34:54sbtsetmessages: + msg190506
2013-06-02 20:47:53spresse1setmessages: + msg190504
2013-06-02 20:38:09sbtsetmessages: + msg190502
2013-06-02 20:03:44spresse1setfiles: + bugon.tar.gz

messages: + msg190497
2013-06-02 19:49:10sbtsetnosy: + sbt
messages: + msg190496
2013-06-02 17:41:58spresse1setmessages: + msg190493
2013-06-02 17:20:28madmazesetnosy: + madmaze
2013-06-02 17:11:07spresse1create