This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing.pool.Pool.apply block infinitely when stressed while using maxtasksperchild
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: davin, harsh8398, pitrou
Priority: normal Keywords:

Created on 2022-03-30 09:53 by harsh8398, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
example.py harsh8398, 2022-03-30 09:53 Example python script which produces the issue
Messages (1)
msg416331 - (view) Author: Harsh Patel (harsh8398) Date: 2022-03-30 09:53
We have a long-running process, which uses thread pool to get tasks and these threads then schedule the tasks and send them to multiprocessing.Pool (with maxtasksperchild limit set) using apply function. The original codebase is large but I have extracted this logic into a simple script for convenience so that you can see the issue. The issue is that after sometime the apply function call gets blocked infinitely. This only happens when we use maxtasksperchild and cause the library for multiple process recreation.

How script works:
Script creates an object of the Manager class which is responsible for creating a pool of threads and creating multiprocessing pool. Each thread is provided jobs to execute which is done by Scheduler.get_ready_jobs function call. I have designed this function in a way that 100 jobs are returned after every 5 calls to this function else it returns no jobs. Each thread while performing job sends processing_func to multiprocessing pool for execution which just prints "processing something" and exits.

Environment:
Python 3.7.10
Clang 12.0.5
MacOS v11.6 intel

Steps to reproduce:
1. Download the attached Python script and execute it
2. Let it run for 10-15 seconds and interrupt the execution

You will notice that some threads will exit successfully but for some, you will see the last message printed is "executing cpu_pool apply <thread_name>" which indicates these threads are blocked even though the actual processing_func is just calling a print function.

Warning: You will also not be able to exit the process without killing it.

As a note, this is my first Python issue report so please let me know if you need more information.
History
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91315
2022-03-30 09:53:14harsh8398create