Message178301
> Richard, apart from performance, what's the advantage of this approach over the
> fork+exec version?
It is really just performance. For context running the unittests in a 1 cpu linux VM gives me
fork:
real 0m53.868s
user 0m1.496s
sys 0m9.757s
fork+exec:
real 1m30.951s
user 0m24.598s
sys 0m25.614s
forkserver:
real 0m54.087s
user 0m1.572s # excludes descendant processes
sys 0m2.336s # excludes descendant processes
So running the unit tests using fork+exec takes about 4 times as much cpu time.
Starting then immediately joining a trivial process in a loop gives
fork: 0.025 seconds/process
fork+exec: 0.245 seconds/process
forkserver: 0.016 seconds/process
So latency is about 10 times higher with fork+exec.
> Because it seems more complicated, and although I didn't have a look a this last
> patch, I guess that most of the fork+exec version could be factorized with the
> Windows version, no?
The different fork methods are now implemented in separate files. The line counts are
117 popen_spawn_win32.py
80 popen_fork.py
184 popen_spawn_posix.py
191 popen_forkserver.py
I don't think any more sharing between the win32 and posix cases is possible. (Note that popen_spawn_posix.py implements a cleanup helper process which is also used by the "forkserver" method.)
> Since it's only intented to be used as a "debugging"/special-purpose replacement - it
> would probably be better if it could be made as simple as possible.
Actually, avoiding the whole fork+threads mess is a big motivation. multiprocessing uses threads in a few places (like implementing Queue), and tries to do so as safely as possible. But unless you turn off garbage collection you cannot really control what code might be running in a background thread when the main thread forks.
> Also, as you've noted, FD passing isn't supported by all Unices out there
> (and we've had some reliability issues on OS-X, too).
OSX does not seem to allow passing multiple ancilliary messages at once -- but you can send multiple fds in a single ancilliary message. Also, when you send fds on OSX you have to wait for a response from the other end before doing anything else. Not doing that was the cause of the previous fd passing failures in test_multiprocessing. |
|
Date |
User |
Action |
Args |
2012-12-27 15:59:40 | sbt | set | recipients:
+ sbt, gregory.p.smith, pitrou, christian.heimes, ned.deily, jnoller, rcoyner, asksol, cool-RR, dholth, brandon-rhodes, neologix, catalin.iacob, numbernine, vsekhar, mrmekon |
2012-12-27 15:59:40 | sbt | set | messageid: <1356623980.06.0.702267444211.issue8713@psf.upfronthosting.co.za> |
2012-12-27 15:59:40 | sbt | link | issue8713 messages |
2012-12-27 15:59:39 | sbt | create | |
|