classification
Title: multiprocessing halts when child process crashes/quits
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: João Eiras
Priority: normal Keywords:

Created on 2020-02-25 17:24 by João Eiras, last changed 2020-02-25 17:24 by João Eiras.

Messages (1)
msg362651 - (view) Author: João Eiras (João Eiras) Date: 2020-02-25 17:24
Hi.

When one of the processes in a multiprocessing.pool picks up a task then then somehow crashes (and by crash I mean crashing the python process with something like a SEGV) or is killed, the pool in the main process will notice one of the workers died and will repopulate the pool, but it does not keep track which task was being handled by the process that died. As consequence, a caller waiting for a result will get stuck forever.

Example:
    with multiprocessing.Pool(1) as pool:
        result = pool.map_async(os._exit, [1]).get(timeout=2)

I found this because I was trying to use a lock with a spawned process on linux and that caused a crash and my program froze, but that is another issue.
History
Date User Action Args
2020-02-25 17:24:16João Eirascreate