classification
Title: multiprocessing Pool should allow custom task queue
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: jnoller Nosy List: asksol, iritkatriel, jnoller, masher, petri.lehtinen, pitrou, swindmill
Priority: normal Keywords: patch

Created on 2009-07-03 13:23 by masher, last changed 2021-06-29 19:09 by iritkatriel.

Files
File name Uploaded Description Edit
multiproc.patch masher, 2009-07-03 16:35 patch: set up _taskqueue in _setup_queues
Messages (12)
msg90052 - (view) Author: Matthew Leon Grinshpun (masher) Date: 2009-07-03 13:23
Multiprocessing's Pool class __init__ method is written in a way that
makes it very difficult for a subclass to modify self._taskqueue. There
are very good reasons for wanting to do this - ie, making the taskqueue
block when it reaches a certain size. It seems to me that simply moving
the taskqueue assignment into _setup_queues() would do the trick.

The following usenet group discussion contains further details:

http://groups.google.com/group/comp.lang.python/browse_thread/thread/e6c67a09fb20cdec#
msg90056 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-03 13:50
I'm not against this; new features will only be committed to the 2.7 and 
3.1.x branches however. 

Something to help speed this up would be an actual patch with docs/tests 
for the module made against python trunk. That way I could review and 
simply submit.
msg90067 - (view) Author: Matthew Leon Grinshpun (masher) Date: 2009-07-03 16:35
I have attached a patch. All I did was shift the one line from __init__
to _setup_queues.

That's it.

You could take advantage of the change in the following way:

class BlockingPool(pool.Pool):
    def _setup_queues(self):
        pool.Pool._setup_queues(self)
        self._taskqueue = Queue(3)
        self._inqueue = Queue(3)
        self._quick_put = self._inqueue.put

Because of the design of _handle_tasks thread, you need to cap both
_taskqueue and _inqueue for blocking to work out.

I can't see any potential problems caused by this change, but I have
little experience working with this module so far.
msg90070 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-03 18:11
Thanks Matthew - I agree, it's simple. But fixing the code is 1/3 of the 
patch. I'll need to add tests/update the ones there as well as modify the 
.rst documentation.
msg115259 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-08-31 11:03
are there really any test/doc changes needed for this?
msg115260 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-08-31 11:12
> are there really any test/doc changes needed for this?

Yes. At bare minimum we need unit tests for the new possible usage of
the API; I'd like docs and an example which show the new usage as
well.
msg119499 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-10-24 08:44
Matthew, would you be willing to write tests + documentation for this?
msg119511 - (view) Author: Matthew Leon Grinshpun (masher) Date: 2010-10-24 13:09
I should be able to do this in November. For the moment I'm a bit busy.
msg148170 - (view) Author: Sterling Windmill (swindmill) Date: 2011-11-23 06:38
I would also like to see this functionality as I'm currently using more memory than I'd like when using multiprocessing.Pool
msg148244 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-24 12:31
As the name implies, _setup_queues is a private method. It feels a bit weird to recommend overriding it in a subclass.
msg148246 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2011-11-24 12:58
@swindmill, if you provide a doc/test patch then this can probably be merged.

@pitrou, We could change it to `setup_queues`, though I don't think
even changing the name of "private" methods is a good idea.  It could simply be an alias to `_setup_queues` or vice versa.
msg396753 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-06-29 19:09
It should be possible to do this now through the context, but it's not documented:

>>> import multiprocessing as mp
>>> ctx = mp.get_context()
>>> ctx.SimpleQueue
<bound method BaseContext.SimpleQueue of <multiprocessing.context.SpawnContext object at 0x000001202D30E980>>
>>> ctx.SimpleQueue = list
>>> ctx.SimpleQueue
<class 'list'>
History
Date User Action Args
2021-06-29 19:09:17iritkatrielsetnosy: + iritkatriel
messages: + msg396753
2011-11-24 12:58:12asksolsetmessages: + msg148246
2011-11-24 12:31:34pitrousetversions: + Python 3.3, - Python 3.2
nosy: + pitrou

messages: + msg148244

stage: test needed -> needs patch
2011-11-23 07:35:23petri.lehtinensetnosy: + petri.lehtinen
2011-11-23 06:38:03swindmillsetnosy: + swindmill
messages: + msg148170
2010-10-24 13:09:55mashersetmessages: + msg119511
2010-10-24 08:44:39asksolsetmessages: + msg119499
2010-08-31 11:12:41jnollersetmessages: + msg115260
2010-08-31 11:11:55asksolsetstage: needs patch -> test needed
2010-08-31 11:03:41asksolsetmessages: + msg115259
2010-08-27 15:30:27BreamoreBoysetstage: needs patch
versions: + Python 3.2, - Python 3.1
2010-08-27 13:53:20asksolsetnosy: + asksol
2009-07-03 18:11:56jnollersetmessages: + msg90070
2009-07-03 16:35:40mashersetfiles: + multiproc.patch
keywords: + patch
messages: + msg90067
2009-07-03 13:51:00jnollersetpriority: normal

messages: + msg90056
2009-07-03 13:35:58benjamin.petersonsetassignee: jnoller

nosy: + jnoller
2009-07-03 13:23:17mashercreate