classification
Title: multiprocessing pool finalizer can fail if triggered in background pool thread
Type: behavior Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: python-dev, sbt
Priority: normal Keywords:

Created on 2012-06-18 14:33 by sbt, last changed 2012-06-18 15:08 by python-dev.

Messages (2)
msg163101 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-18 14:33
Multiprocessing's process pool originally used a finalizer to shutdown the pool when the pool object is garbage collected.

Since the maxtasksperchild feature was added, the worker_handler thread holds a reference to the pool, preventing garbage collection at least until that thread finished.

In some cases the last reference to the pool is owned by the worker_handler thread, and as it exits, the finalizer is triggered.  Since the finalizer tries to join the worker_handler thread, an error will be raised since a thread cannot join itself.

It would have been better if pools had used the context manager protocol rather than using RAII -- see Issue #15064.

Unless/until RAII usage is fixed, a quick fix would be to just make the finalizer skip trying to join the current thread.
msg163106 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-18 15:08
New changeset 4c07b9c49b75 by Richard Oudkerk in branch '2.7':
Issue #15101: Make pool finalizer avoid joining current thread
http://hg.python.org/cpython/rev/4c07b9c49b75

New changeset e1cd1f430ff1 by Richard Oudkerk in branch '3.2':
Issue #15101: Make pool finalizer avoid joining current thread.
http://hg.python.org/cpython/rev/e1cd1f430ff1

New changeset 59e0a51c5fc3 by Richard Oudkerk in branch 'default':
Issue #15101: Make pool finalizer avoid joining current thread.
http://hg.python.org/cpython/rev/59e0a51c5fc3
History
Date User Action Args
2012-06-18 15:08:39python-devsetnosy: + python-dev
messages: + msg163106
2012-06-18 14:33:40sbtcreate