Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(75873)

#8713: multiprocessing needs option to eschew fork() under Linux

Can't Edit
Can't Publish+Mail
Start Review
Created:
7 years, 7 months ago by brandon
Modified:
5 years, 8 months ago
Reviewers:
greg, shibturn, pitrou
CC:
gregory.p.smith, AntoinePitrou, christian.heimes, ned.deily, jnoller_gmail.com, ezio.melotti, rcoyner_gmail.com, asksol, dholth, brandon-rhodes, Charles-Fran├žois Natali, catalin.iacob, devnull_psf.upfronthosting.co.za, sbt, numbernine.python_complexnumber.net, vivek_viveksekhar.ca, piotr.dobrogost, mrmekon_gmail.com, stan_mtrr.org
Visibility:
Public.

Patch Set 1 #

Total comments: 2

Patch Set 2 #

Total comments: 10

Patch Set 3 #

Patch Set 4 #

Total comments: 28

Patch Set 5 #

Total comments: 5

Patch Set 6 #

Unified diffs Side-by-side diffs Delta from patch set Stats Patch
Doc/includes/mp_benchmarks.py View 1 2 3 4 5 1 chunk +0 lines, -239 lines 0 comments Download
Doc/includes/mp_newtype.py View 1 2 3 4 5 3 chunks +1 line, -13 lines 0 comments Download
Doc/includes/mp_pool.py View 1 2 3 4 5 2 chunks +88 lines, -249 lines 0 comments Download
Doc/includes/mp_synchronize.py View 1 2 3 4 5 1 chunk +0 lines, -278 lines 0 comments Download
Doc/includes/mp_webserver.py View 1 2 3 4 5 1 chunk +0 lines, -70 lines 0 comments Download
Doc/includes/mp_workers.py View 1 2 3 4 5 1 chunk +0 lines, -13 lines 0 comments Download
Doc/library/multiprocessing.rst View 1 2 3 4 5 12 chunks +159 lines, -87 lines 0 comments Download
Doc/whatsnew/3.4.rst View 1 2 3 4 5 3 chunks +13 lines, -6 lines 0 comments Download
Lib/multiprocessing/connection.py View 1 2 3 4 5 7 chunks +41 lines, -20 lines 0 comments Download
Lib/multiprocessing/dummy/__init__.py View 1 2 3 4 5 2 chunks +2 lines, -2 lines 0 comments Download
Lib/multiprocessing/forking.py View 1 2 3 4 5 1 chunk +0 lines, -477 lines 0 comments Download
Lib/multiprocessing/forkserver.py View 1 2 3 4 5 1 chunk +238 lines, -0 lines 0 comments Download
Lib/multiprocessing/heap.py View 1 2 3 4 5 5 chunks +42 lines, -14 lines 0 comments Download
Lib/multiprocessing/__init__.py View 1 2 3 4 5 8 chunks +72 lines, -35 lines 0 comments Download
Lib/multiprocessing/managers.py View 1 2 3 4 5 12 chunks +23 lines, -21 lines 0 comments Download
Lib/multiprocessing/pool.py View 1 2 3 4 5 19 chunks +44 lines, -40 lines 0 comments Download
Lib/multiprocessing/popen_fork.py View 1 2 3 4 5 1 chunk +87 lines, -0 lines 0 comments Download
Lib/multiprocessing/popen_forkserver.py View 1 2 3 4 5 1 chunk +75 lines, -0 lines 0 comments Download
Lib/multiprocessing/popen.py View 1 2 3 4 5 1 chunk +78 lines, -0 lines 0 comments Download
Lib/multiprocessing/popen_spawn_posix.py View 1 2 3 4 5 1 chunk +75 lines, -0 lines 0 comments Download
Lib/multiprocessing/popen_spawn_win32.py View 1 2 3 4 5 1 chunk +102 lines, -0 lines 0 comments Download
Lib/multiprocessing/process.py View 1 2 3 4 5 11 chunks +30 lines, -30 lines 0 comments Download
Lib/multiprocessing/queues.py View 1 2 3 4 5 6 chunks +19 lines, -17 lines 0 comments Download
Lib/multiprocessing/reduction.py View 1 2 3 4 5 5 chunks +161 lines, -200 lines 0 comments Download
Lib/multiprocessing/resource_sharer.py View 1 2 3 4 5 1 chunk +158 lines, -0 lines 0 comments Download
Lib/multiprocessing/semaphore_tracker.py View 1 2 3 4 5 1 chunk +135 lines, -0 lines 0 comments Download
Lib/multiprocessing/sharedctypes.py View 1 2 3 4 5 1 chunk +5 lines, -2 lines 0 comments Download
Lib/multiprocessing/spawn.py View 1 2 3 4 5 1 chunk +258 lines, -0 lines 0 comments Download
Lib/multiprocessing/synchronize.py View 1 2 3 4 5 7 chunks +58 lines, -15 lines 0 comments Download
Lib/multiprocessing/util.py View 1 2 3 4 5 6 chunks +60 lines, -10 lines 0 comments Download
Lib/test/mp_fork_bomb.py View 1 2 3 4 5 1 chunk +5 lines, -0 lines 0 comments Download
Lib/test/regrtest.py View 1 2 3 4 5 1 chunk +1 line, -1 line 0 comments Download
Lib/test/test_multiprocessing_fork.py View 1 2 3 4 5 1 chunk +7 lines, -0 lines 0 comments Download
Lib/test/test_multiprocessing_forkserver.py View 1 2 3 4 5 1 chunk +7 lines, -0 lines 0 comments Download
Lib/test/_test_multiprocessing.py View 1 2 3 4 5 1 chunk +3755 lines, -0 lines 0 comments Download
Lib/test/test_multiprocessing.py View 1 2 3 4 5 1 chunk +0 lines, -3606 lines 0 comments Download
Lib/test/test_multiprocessing_spawn.py View 1 2 3 4 5 1 chunk +7 lines, -0 lines 0 comments Download
Makefile.pre.in View 1 2 3 4 5 1 chunk +7 lines, -2 lines 0 comments Download
Modules/_multiprocessing/multiprocessing.c View 1 2 3 4 5 1 chunk +1 line, -0 lines 0 comments Download
Modules/_multiprocessing/multiprocessing.h View 1 2 3 4 5 1 chunk +1 line, -0 lines 0 comments Download
Modules/_multiprocessing/semaphore.c View 1 2 3 4 5 9 chunks +64 lines, -14 lines 0 comments Download

Messages

Total messages: 8
gregory.p.smith
http://bugs.python.org/review/8713/diff/4040/Lib/multiprocessing/forking.py File Lib/multiprocessing/forking.py (right): http://bugs.python.org/review/8713/diff/4040/Lib/multiprocessing/forking.py#newcode160 Lib/multiprocessing/forking.py:160: if self.pid == 0: this code between fork() and ...
6 years, 3 months ago #1
gregory.p.smith
http://bugs.python.org/review/8713/diff/4040/Lib/multiprocessing/forking.py File Lib/multiprocessing/forking.py (right): http://bugs.python.org/review/8713/diff/4040/Lib/multiprocessing/forking.py#newcode160 Lib/multiprocessing/forking.py:160: if self.pid == 0: On 2012/12/27 21:07:14, gregory.p.smith wrote: ...
6 years, 3 months ago #2
sbt
http://bugs.python.org/review/8713/diff/6917/Lib/multiprocessing/__init__.py File Lib/multiprocessing/__init__.py (right): http://bugs.python.org/review/8713/diff/6917/Lib/multiprocessing/__init__.py#newcode268 Lib/multiprocessing/__init__.py:268: from multiprocessing.forking import set_executable On 2012/12/27 23:38:12, gregory.p.smith wrote: ...
6 years, 3 months ago #3
AntoinePitrou
Great work! Here are some comments. http://bugs.python.org/review/8713/diff/8847/Lib/multiprocessing/__init__.py File Lib/multiprocessing/__init__.py (right): http://bugs.python.org/review/8713/diff/8847/Lib/multiprocessing/__init__.py#newcode249 Lib/multiprocessing/__init__.py:249: child processes on ...
5 years, 8 months ago #4
sbt
Thanks for the review! http://bugs.python.org/review/8713/diff/8847/Lib/multiprocessing/heap.py File Lib/multiprocessing/heap.py (right): http://bugs.python.org/review/8713/diff/8847/Lib/multiprocessing/heap.py#newcode65 Lib/multiprocessing/heap.py:65: 'pym-%d-%d' % (os.getpid(), next(self._counter))) On ...
5 years, 8 months ago #5
AntoinePitrou
http://bugs.python.org/review/8713/diff/8847/Lib/multiprocessing/semaphore_tracker.py File Lib/multiprocessing/semaphore_tracker.py (right): http://bugs.python.org/review/8713/diff/8847/Lib/multiprocessing/semaphore_tracker.py#newcode115 Lib/multiprocessing/semaphore_tracker.py:115: print('[semaphore_tracker] %r: successfully unlinked' % On 2013/08/09 16:16:13, sbt ...
5 years, 8 months ago #6
AntoinePitrou
Three nits. http://bugs.python.org/review/8713/diff/8869/Lib/multiprocessing/heap.py File Lib/multiprocessing/heap.py (right): http://bugs.python.org/review/8713/diff/8869/Lib/multiprocessing/heap.py#newcode41 Lib/multiprocessing/heap.py:41: print((-1, size, name)) I suppose this one ...
5 years, 8 months ago #7
sbt
5 years, 8 months ago #8
I have pushed the changes.

http://bugs.python.org/review/8713/diff/8869/Lib/multiprocessing/popen.py
File Lib/multiprocessing/popen.py (right):

http://bugs.python.org/review/8713/diff/8869/Lib/multiprocessing/popen.py#new...
Lib/multiprocessing/popen.py:43: start_helpers=True,
forkserver_preload=['__main__']):
On 2013/08/12 10:55:39, AntoinePitrou wrote:
> Sounds a bit weird to hardcode a forkserver-specific parameter. How about a
> generic **method_args? Or perhaps a generic preload_modules?
> 

I think I will use a set_forkserver_preload() function instead.

http://bugs.python.org/review/8713/diff/8869/Lib/multiprocessing/spawn.py
File Lib/multiprocessing/spawn.py (right):

http://bugs.python.org/review/8713/diff/8869/Lib/multiprocessing/spawn.py#new...
Lib/multiprocessing/spawn.py:143: sys_path[i] = process.ORIGINAL_DIR
On 2013/08/12 10:55:39, AntoinePitrou wrote:
> How can ``i`` be defined here? Do you want .append() instead?

That should have gone in the else-branch.  The idea is to replace '' by
ORIGINAL_DIR.
Sign in to reply to this message.

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+