Title: multiprocessing pool finalizer can fail if triggered in background pool thread
Type: behavior Stage: resolved
Components: Versions:
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, python-dev, sbt
Priority: normal Keywords:

Created on 2012-06-18 14:33 by sbt, last changed 2015-01-29 22:40 by davin. This issue is now closed.

Messages (3)
msg163101 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-18 14:33
Multiprocessing's process pool originally used a finalizer to shutdown the pool when the pool object is garbage collected.

Since the maxtasksperchild feature was added, the worker_handler thread holds a reference to the pool, preventing garbage collection at least until that thread finished.

In some cases the last reference to the pool is owned by the worker_handler thread, and as it exits, the finalizer is triggered.  Since the finalizer tries to join the worker_handler thread, an error will be raised since a thread cannot join itself.

It would have been better if pools had used the context manager protocol rather than using RAII -- see Issue #15064.

Unless/until RAII usage is fixed, a quick fix would be to just make the finalizer skip trying to join the current thread.
msg163106 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-18 15:08
New changeset 4c07b9c49b75 by Richard Oudkerk in branch '2.7':
Issue #15101: Make pool finalizer avoid joining current thread

New changeset e1cd1f430ff1 by Richard Oudkerk in branch '3.2':
Issue #15101: Make pool finalizer avoid joining current thread.

New changeset 59e0a51c5fc3 by Richard Oudkerk in branch 'default':
Issue #15101: Make pool finalizer avoid joining current thread.
msg234996 - (view) Author: Davin Potts (davin) * Date: 2015-01-29 22:40
Having reviewed the code commits made by Richard more than 2.5 years ago in conjunction with his related code commits from already-closed Issue #15064, I am led to conclude that all the items described in this issue have been addressed and implemented.

Specifically, the "quick fix" has been implemented in Richard's commits from 2.5+ years ago, use of context managers was introduced in certain places as part of Issue #15064, and any further opportunities for leveraging context managers should reasonably be described in a new, separate issue.

The only outstanding item might be the desire for a test case to ensure Richard's improvements are not inadvertently undone in the future but given the nature of the changes and the age of this issue, I'm voting to go ahead and close this issue as resolved years ago.
Date User Action Args
2015-01-29 22:40:54davinsetstatus: open -> closed

nosy: + davin
messages: + msg234996

resolution: fixed
stage: resolved
2012-06-18 15:08:39python-devsetnosy: + python-dev
messages: + msg163106
2012-06-18 14:33:40sbtcreate