This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Feature request: maxtasksperchild for ProcessPoolExecutor
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: cool-RR, gregory.p.smith, loganasherjones, pitrou
Priority: normal Keywords: patch

Created on 2021-07-24 12:14 by cool-RR, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 27373 merged loganasherjones, 2021-07-26 17:39
PR 32187 open loganasherjones, 2022-03-30 01:55
Messages (13)
msg398143 - (view) Author: Ram Rachum (cool-RR) * Date: 2021-07-24 12:14
I love `concurrent.futures`, and I'd like to use it wherever I can. There's a feature in `multiprocessing.Pool` that I wish would also be available in `ProcessPoolExecutor`: The `maxtasksperchild` argument.

Documentation: "maxtasksperchild is the number of tasks a worker process can complete before it will exit and be replaced with a fresh worker process, to enable unused resources to be freed. The default maxtasksperchild is None, which means worker processes will live as long as the pool."

I want to be able to set it to 1, so each process will only execute one task and then be replaced with a fresh process.
msg398240 - (view) Author: Logan Jones (loganasherjones) * Date: 2021-07-26 16:05
I think I have a solution for this, but I'm pretty new to contributing. Still writing up some tests.
msg398243 - (view) Author: Ram Rachum (cool-RR) * Date: 2021-07-26 16:12
Awesome. You can link to experimental code here. Even if it were not accepted to Python, it could be useful for me and for other people who might see this issue.
msg406689 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-11-20 20:19
New changeset fdc0e09c3316098b038996c428e88931f0a4fcdb by Logan Jones in branch 'main':
bpo-44733: Add max_tasks_per_child to ProcessPoolExecutor (GH-27373)
https://github.com/python/cpython/commit/fdc0e09c3316098b038996c428e88931f0a4fcdb
msg411449 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-01-24 07:27
[bpo-46464](https://bugs.python.org/issue46464) describes a deadlock that the pre-requisite for this feature causes.

Spawning new children directly via fork() is impossible once a thread in the parent process has been started and concurrent.futures.process starts a thread.
msg411450 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-01-24 07:29
If we want this feature to stay in, it needs an implementation that does not rely on a multithreaded processing calling fork(). I'm going to have to revert the existing implementation for bpo-46464.
msg411452 - (view) Author: Ram Rachum (cool-RR) * Date: 2022-01-24 07:54
Oh that sucks. Logan and Antoine worked on this feature for so long. Thanks for reporting Greg.
msg411471 - (view) Author: Logan Jones (loganasherjones) * Date: 2022-01-24 12:44
Based on my reading I’m hopeful that this change can make it in quickly once I find the time. The previous implementation didn’t care how the processes were created. 

I will look as soon as I can
msg411546 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-01-25 02:36
from discussion on the other bug it looks like we should have a way to keep this; we just need to not allow it when the mp_context to be used is the 'fork' one.
msg411547 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-01-25 02:42
As this is a new feature, it would also be reasonable to have specifying max_tasks_per_child without explicitly specifying a mp_context default to a safe mp_context.
msg416310 - (view) Author: Logan Jones (loganasherjones) * Date: 2022-03-30 00:35
Okay, I'm actually able to work on this again. What is the best way to make this change real. Should I be working off of main?
msg416314 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-03-30 00:59
yep, branch off of a recent main.
msg416406 - (view) Author: Logan Jones (loganasherjones) * Date: 2022-03-31 00:07
Ok I now have a PR up with the features requested. Let me know if you need anything else!
History
Date User Action Args
2022-04-11 14:59:47adminsetgithub: 88896
2022-03-31 00:07:58loganasherjonessetmessages: + msg416406
2022-03-30 01:55:28loganasherjonessetstage: needs patch -> patch review
pull_requests: + pull_request30264
2022-03-30 00:59:01gregory.p.smithsetmessages: + msg416314
2022-03-30 00:35:53loganasherjonessetmessages: + msg416310
2022-01-25 02:42:06gregory.p.smithsetmessages: + msg411547
2022-01-25 02:36:18gregory.p.smithsetmessages: + msg411546
2022-01-24 12:44:27loganasherjonessetmessages: + msg411471
2022-01-24 07:54:10cool-RRsetmessages: + msg411452
2022-01-24 07:29:17gregory.p.smithsetassignee: gregory.p.smith
resolution: fixed ->
2022-01-24 07:29:00gregory.p.smithsetstatus: closed -> open

messages: + msg411450
stage: resolved -> needs patch
2022-01-24 07:27:43gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg411449
2021-11-20 20:20:03pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-11-20 20:19:50pitrousetmessages: + msg406689
2021-07-26 17:39:39loganasherjonessetkeywords: + patch
stage: patch review
pull_requests: + pull_request25911
2021-07-26 16:12:19cool-RRsetmessages: + msg398243
2021-07-26 16:05:54loganasherjonessetmessages: + msg398240
2021-07-25 12:57:33loganasherjonessetnosy: + loganasherjones
2021-07-24 12:14:41cool-RRcreate