Author sbt
Recipients asksol, brandon-rhodes, catalin.iacob, christian.heimes, cool-RR, dholth, gregory.p.smith, jnoller, mrmekon, ned.deily, neologix, numbernine, pitrou, rcoyner, sbt, vsekhar
Date 2012-12-27.15:59:39
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1356623980.06.0.702267444211.issue8713@psf.upfronthosting.co.za>
In-reply-to
Content
> Richard, apart from performance, what's the advantage of this approach over the 
> fork+exec version?

It is really just performance.  For context running the unittests in a 1 cpu linux VM gives me

fork:
real    0m53.868s
user    0m1.496s
sys     0m9.757s

fork+exec:
real    1m30.951s
user    0m24.598s
sys     0m25.614s

forkserver:
real    0m54.087s
user    0m1.572s             # excludes descendant processes
sys     0m2.336s             # excludes descendant processes

So running the unit tests using fork+exec takes about 4 times as much cpu time.

Starting then immediately joining a trivial process in a loop gives

fork:        0.025 seconds/process
fork+exec:   0.245 seconds/process
forkserver:  0.016 seconds/process

So latency is about 10 times higher with fork+exec.

> Because it seems more complicated, and although I didn't have a look a this last 
> patch, I guess that most of the fork+exec version could be factorized with the
> Windows version, no?

The different fork methods are now implemented in separate files.  The line counts are

  117 popen_spawn_win32.py
   80 popen_fork.py
  184 popen_spawn_posix.py
  191 popen_forkserver.py

I don't think any more sharing between the win32 and posix cases is possible.  (Note that popen_spawn_posix.py implements a cleanup helper process which is also used by the "forkserver" method.)

> Since it's only intented to be used as a "debugging"/special-purpose replacement - it 
> would probably be better if it could be made as simple as possible.

Actually, avoiding the whole fork+threads mess is a big motivation.  multiprocessing uses threads in a few places (like implementing Queue), and tries to do so as safely as possible.  But unless you turn off garbage collection you cannot really control what code might be running in a background thread when the main thread forks.

> Also, as you've noted, FD passing isn't supported by all Unices out there
> (and we've had some reliability issues on OS-X, too).

OSX does not seem to allow passing multiple ancilliary messages at once -- but you can send multiple fds in a single ancilliary message.  Also, when you send fds on OSX you have to wait for a response from the other end before doing anything else.  Not doing that was the cause of the previous fd passing failures in test_multiprocessing.
History
Date User Action Args
2012-12-27 15:59:40sbtsetrecipients: + sbt, gregory.p.smith, pitrou, christian.heimes, ned.deily, jnoller, rcoyner, asksol, cool-RR, dholth, brandon-rhodes, neologix, catalin.iacob, numbernine, vsekhar, mrmekon
2012-12-27 15:59:40sbtsetmessageid: <1356623980.06.0.702267444211.issue8713@psf.upfronthosting.co.za>
2012-12-27 15:59:40sbtlinkissue8713 messages
2012-12-27 15:59:39sbtcreate