classification
Title: multiprocessing.get hangs if the pool is closed in the signal handler
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: tkc17
Priority: normal Keywords:

Created on 2021-06-19 19:49 by tkc17, last changed 2021-06-19 19:49 by tkc17.

Files
File name Uploaded Description Edit
mp_test.py tkc17, 2021-06-19 19:49 Test script to repro the issue
Messages (1)
msg396151 - (view) Author: Chaitanya (tkc17) * Date: 2021-06-19 19:49
This is similar to https://bugs.python.org/issue22393 but the child is killed in the Pool.terminate() using SIGTERM i.e., proper way.

Run the attached script that runs an indefinite method using multiprocessing.apply_async, and in the signal handler for SIGTERM does the pool clean up (close, terminate and join), on issuing a SIGTERM (`kill <parent PID>`), the cleanup works fine, but the parent process is stuck in get() as during termination the event is not set.

Able to repro with all 3.6/3.9/3.10.


See logs below
```
[DEBUG/MainProcess] created semlock with handle 140035884298240
[DEBUG/MainProcess] created semlock with handle 140035884294144
[DEBUG/MainProcess] created semlock with handle 140035884290048
[DEBUG/MainProcess] created semlock with handle 140035884285952
[DEBUG/MainProcess] created semlock with handle 140035884281856
[DEBUG/MainProcess] created semlock with handle 140035884277760
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-1] child process calling self.run()
[INFO/MainProcess] <function close_thread_pools at 0x7f5ca3a20b80>
[INFO/MainProcess] <function close_thread_pools at 0x7f5ca3a20b80>
[INFO/MainProcess] <function close_thread_pools at 0x7f5ca3a20b80>
[INFO/ForkPoolWorker-1] Signals for 28462
[INFO/ForkPoolWorker-1] Handlers.SIG_DFL
[INFO/ForkPoolWorker-1] <built-in function default_int_handler>
[INFO/ForkPoolWorker-1] Handlers.SIG_DFL
[INFO/MainProcess] 28461: Caught signal 15
[INFO/MainProcess] 28461: Closing pools
[INFO/MainProcess] 28461: Terminating pools
[DEBUG/MainProcess] terminating pool
[DEBUG/MainProcess] finalizing pool
[DEBUG/MainProcess] helping task handler/workers to finish
[DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[DEBUG/MainProcess] joining worker handler
[DEBUG/MainProcess] worker handler exiting
[DEBUG/MainProcess] result handler found thread._state=TERMINATE
[DEBUG/MainProcess] ensuring that outqueue is not full
[DEBUG/MainProcess] result handler exiting: len(cache)=1, thread._state=TERMINATE
[DEBUG/MainProcess] task handler got sentinel
[DEBUG/MainProcess] task handler sending sentinel to result handler
[DEBUG/MainProcess] task handler sending sentinel to workers
[DEBUG/MainProcess] task handler exiting
[DEBUG/MainProcess] terminating workers
[DEBUG/MainProcess] joining task handler
[DEBUG/MainProcess] joining result handler
[DEBUG/MainProcess] joining pool workers
[DEBUG/MainProcess] cleaning up worker 28462
[INFO/MainProcess] 28461: Joining pools
[DEBUG/MainProcess] joining pool
[INFO/MainProcess] 28461: Closed pools
```

```
(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.9/threading.py", line 312, in wait
    waiter.acquire()
  File "/usr/lib/python3.9/threading.py", line 574, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 762, in wait
    self._event.wait(timeout)
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 765, in get
    self.wait(timeout)
  File "/home/tatac/workspaces/gerrit/bwt-device-health-manager/mp_test.py", line 66, in <module>
    a.get()
(gdb) 
```
History
Date User Action Args
2021-06-19 19:49:48tkc17create