classification
Title: multiprocessing.pool: Proposal: "waitforslot"
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, asksol, jnoller, r.david.murray, sbt
Priority: normal Keywords: patch

Created on 2010-07-13 13:47 by asksol, last changed 2015-04-09 19:09 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing-trunk@82502-apply-semaphore.patch asksol, 2010-07-13 13:47 patch against trunk@82502
Messages (5)
msg110193 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-07-13 13:47
This patch adds the `waitforslot` argument to apply_async. If set to `True`, apply_async will not return until there is a worker available to process the job.

This is implemented by a semaphore that is released by the result handler whenever a new result is ready. The semaphore is also released
when the supervisor (worker_handler) finds a worker process that has been
unexpectedly terminated.

This is already in use by Celery 2.0, which ships with its own modified version of multiprocessing.pool.

I'm not sure about the name ``waitforslot``, I think I may even hate it, but haven't been able to come up with a better name for it yet.
msg220825 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-17 13:09
Could somebody please review the attached patch.
msg221804 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2014-06-28 17:52
Since there are no new features added to Python 2, this would be a Python 3 only feature.

I think for Python 3 it is better to concentrate on developing concurrent.futures rather than multiprocessing.Pool.
msg221963 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2014-06-30 14:45
This patch is quite dated now and I have fixed many bugs since.  The feature is available in billiard and is working well but The code has diverged quite a lot from python trunk.  I will be updating billiard to reflect the changes for Python 3.4 soon (billiard is currently 3.3).

I think we can forget about taking individual patches from billiard for now,
and instead maybe merge the codebases at some point if there's interest.
we have a version of multiprocessing.Pool using async IO and one pipe per process that drastically improves performance
and also avoids the threads+forking issues (well, not the initial fork), but I have not yet adapted it to use the new asyncio module in 3.4

So suggestion is to close this and rather get a discussion going for combining our efforts.
msg240364 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-09 19:09
Closed per OP's request.
History
Date User Action Args
2015-04-09 19:09:36r.david.murraysetstatus: open -> closed

nosy: + r.david.murray
messages: + msg240364

resolution: out of date
stage: patch review -> resolved
2014-06-30 14:45:53asksolsetmessages: + msg221963
2014-06-28 17:52:38sbtsetmessages: + msg221804
2014-06-17 19:28:26ned.deilysetnosy: + sbt
2014-06-17 13:09:58BreamoreBoysetnosy: + BreamoreBoy

messages: + msg220825
versions: + Python 3.5, - Python 3.3
2010-12-14 19:07:02r.david.murraysetstage: patch review
type: enhancement
versions: + Python 3.3, - Python 2.6, Python 2.7
2010-07-13 13:47:29asksolsetfiles: + multiprocessing-trunk@82502-apply-semaphore.patch
keywords: + patch
2010-07-13 13:47:02asksolcreate