This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author athompson6735
Recipients athompson6735, davin, docs@python, nemeskeyd, pitrou, terry.reedy
Date 2021-08-23.17:12:14
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1629738734.84.0.302466810346.issue43306@roundup.psfhosted.org>
In-reply-to
Content
I ran into this bug answering this question on Stack Overflow: https://stackoverflow.com/questions/68890437/cannot-use-result-from-multiprocess-pool-directly

I have minimized the code required to replicate the behavior, but it boils down to: when using "spawn" to create a multiprocessing pool, if an exception occurs during the bootstrapping phase of the new child or during the initialization function with any start method, it is just cleaned up, and another takes its place (which will also fail). This creates an infinite loop of creating child workers, workers exiting due to an exception, and re-populating the pool with new workers.


```
import multiprocessing
multiprocessing.set_start_method("spawn")
# bootstraping only problem with spawn

def task():
    print("task")

if __name__ == "__main__":
    with multiprocessing.Pool() as p:
        p.apply(task)
else:
    raise Exception("raise in child during bootstraping phase")
    
#################################################
# or
#################################################
    
import multiprocessing
# multiprocessing.set_start_method("fork")
# fork or spawn doesn't matter

def task():
    print("task")

def init():
    raise Exception("raise in child during initialization function")

if __name__ == "__main__":
    with multiprocessing.Pool(initializer=init) as p:
        p.apply(task)
```

If Pool._join_exited_workers could determine if a worker exited before bootstrapping, or the initialization function completed, It would indicate a likely significant problem. I'm fine with exceptions in the worker target function not being re-raised in the parent, however it seems the Pool should stop trying if it's failing to create new workers.
History
Date User Action Args
2021-08-23 17:12:14athompson6735setrecipients: + athompson6735, terry.reedy, pitrou, docs@python, nemeskeyd, davin
2021-08-23 17:12:14athompson6735setmessageid: <1629738734.84.0.302466810346.issue43306@roundup.psfhosted.org>
2021-08-23 17:12:14athompson6735linkissue43306 messages
2021-08-23 17:12:14athompson6735create