classification
Title: Add worker process lifetime to multiprocessing.Pool - patch included
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jnoller Nosy List: charlesc, jnoller
Priority: normal Keywords: needs review, patch

Created on 2009-09-22 02:28 by charlesc, last changed 2010-01-27 03:36 by jnoller. This issue is now closed.

Files
File name Uploaded Description Edit
worker-lifetime-python2.6.2.patch charlesc, 2010-01-18 20:48 4th version of patch adding worker lifetime to multiprocessing.Pool
mp_patch_6963_trunk.patch jnoller, 2010-01-26 03:45 trunk based patch
Messages (18)
msg92971 - (view) Author: Charles Cazabon (charlesc) Date: 2009-09-22 02:28
Worker processes with multiprocessing.Pool live for the duration of the
Pool.  If the tasks they run happen to leak memory (from a C extension
module, or from creating cycles of unreachable objects, etc) or open
files or other resources, there's no easy way to clean them up.

Similarly, if one task passed to the pool allocates a large amount of
memory, but further tasks are small, that additional memory isn't
returned to the system because the process involved hasn't exited.

A common approach to this problem (as used by Apache, mod_wsgi, and
various other software) is to allow worker processes to exit (and be
replaced with fresh processes) after completing a specified amount of
work.  The attached patch (against Python 2.6.2, but applies to various
other versions with some fuzz) implements this as optional new behaviour
in multiprocessing.Pool().  An additional optional argument is specified
for the maximum number of tasks a worker process performs before it
exits and is replaced with a fresh worker process.
msg92987 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-09-22 12:35
Hi Charles; I don't see a doc update for this (see multiprocessing.rst) or 
unit tests. I'm not against it, but before I can commit this, I'll need 
those things
msg92999 - (view) Author: Charles Cazabon (charlesc) Date: 2009-09-22 15:36
Alright, I'll add those.  Thanks for looking at it; first pass was
mostly to ensure it wouldn't be wasted work.
msg93191 - (view) Author: Charles Cazabon (charlesc) Date: 2009-09-28 04:03
Updated patch attached; handles some of the Pool methods that weren't
handled before.  Now includes documentation and unit test additions as well.
msg93517 - (view) Author: Charles Cazabon (charlesc) Date: 2009-10-04 00:42
Jesse: this is ready for your review now.  Thanks,

Charles
msg93916 - (view) Author: Charles Cazabon (charlesc) Date: 2009-10-13 03:22
Can someone review this patch?  I believe it's sufficient for inclusion
now, as it includes docs and unit tests, but if anything about it
requires further attention I'd be happy to listen to change requests. 
We'd like get this into mainline where everyone can benefit rather than
continuing to maintain it out-of-tree.
msg93920 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-10-13 13:43
I plan on reviewing the patch once my work with PyCon is completed, which 
should be within the next few weeks. The next release this would show up 
in (2.7/3.2) is not for some time.
msg93921 - (view) Author: Charles Cazabon (charlesc) Date: 2009-10-13 14:17
Okay, thanks, Jesse.  Didn't realize the Con was on.
msg93923 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-10-13 14:24
Well, it's in feb, but I'm involved in some intense planning right now.
msg95243 - (view) Author: Charles Cazabon (charlesc) Date: 2009-11-14 16:43
Hi Jesse -- Any chance you'll be able to review this in time for it to
make it into trunk for the 2.7 alpha release?

Charles
msg95250 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-11-14 18:43
On Sat, Nov 14, 2009 at 11:43 AM, Charles Cazabon
<report@bugs.python.org> wrote:
>
> Charles Cazabon <charlesc-python@pyropus.ca> added the comment:
>
> Hi Jesse -- Any chance you'll be able to review this in time for it to
> make it into trunk for the 2.7 alpha release?

2.7 isn't slated until next year some time, so yes.
msg97832 - (view) Author: Charles Cazabon (charlesc) Date: 2010-01-15 18:46
Ping... two alphas into 2.7.  Have you had a chance to review this functionality?

Thanks.
msg97835 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-01-15 19:06
I'm fine with the functionality; I'm going to test it out and look at committing it by mid-week next week. I apologize, I've been pretty maxed out.
msg98032 - (view) Author: Charles Cazabon (charlesc) Date: 2010-01-18 20:48
No problem, Jesse, I realize you're busy.  

I've updated the patch very slightly to handle an issue I had where the worker maintainer hung once.
msg98129 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-01-22 02:49
I'm working on this now; I'm going to need to port the patch to trunk before moving forward with it. Shouldn't take me long.
msg98313 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-01-26 03:45
Attached is the ported patch for Python 2.7/trunk. Please review it to make sure I didn't completely flub anything. I noticed you had forgotten the maxtasksperchild argument in the unit test, so I added that. I also expanded the docs a little but, paraphrasing your original post w.r.t to the justification.

Let me know what you think, or if you have issues. Once you give the go ahead, I will commit to trunk and then merge to Py3k. 

Once again, thanks for the patch
msg98382 - (view) Author: Charles Cazabon (charlesc) Date: 2010-01-26 21:27
Thanks, Jesse -- it looks good.  If there are bugs remaining in the patch, they're mine and not yours.
msg98402 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-01-27 03:36
Committed to trunk in r77794
Merged to Py3k in r77795
History
Date User Action Args
2010-01-27 03:36:56jnollersetstatus: open -> closed
resolution: fixed
messages: + msg98402
2010-01-26 21:27:20charlescsetmessages: + msg98382
2010-01-26 03:45:48jnollersetfiles: + mp_patch_6963_trunk.patch

messages: + msg98313
2010-01-22 02:49:40jnollersetmessages: + msg98129
2010-01-18 20:48:52charlescsetfiles: + worker-lifetime-python2.6.2.patch

messages: + msg98032
2010-01-18 20:47:12charlescsetfiles: - worker-lifetime-python2.6.2.patch
2010-01-15 19:06:29jnollersetmessages: + msg97835
2010-01-15 18:55:45brian.curtinsetpriority: normal
keywords: + needs review
stage: patch review
versions: + Python 3.2, - Python 3.1
2010-01-15 18:46:00charlescsetmessages: + msg97832
2009-11-14 18:43:19jnollersetmessages: + msg95250
2009-11-14 16:43:00charlescsetmessages: + msg95243
2009-10-13 14:24:16jnollersetmessages: + msg93923
2009-10-13 14:17:33charlescsetmessages: + msg93921
2009-10-13 13:43:47jnollersetmessages: + msg93920
2009-10-13 03:22:42charlescsetmessages: + msg93916
2009-10-04 00:42:52charlescsetmessages: + msg93517
2009-09-28 04:03:25charlescsetfiles: + worker-lifetime-python2.6.2.patch

messages: + msg93191
2009-09-28 04:02:07charlescsetfiles: - worker-lifetime-python2.6.2.patch
2009-09-22 15:36:17charlescsetmessages: + msg92999
2009-09-22 12:35:02jnollersetmessages: + msg92987
2009-09-22 03:41:59benjamin.petersonsetassignee: jnoller

nosy: + jnoller
2009-09-22 02:33:03charlescsetfiles: + worker-lifetime-python2.6.2.patch
2009-09-22 02:32:35charlescsetfiles: - worker-lifetime-python2.6.2.patch
2009-09-22 02:28:13charlesccreate