This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: A manager's server never joins its threads
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: davin, gregory.p.smith, pitrou, sbt, tsutsumi
Priority: normal Keywords:

Created on 2013-08-15 18:13 by pitrou, last changed 2022-04-11 14:57 by admin.

Messages (4)
msg195267 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-15 18:13
The Server class in multiprocessing.managers creates daemon threads and never joins them. This is in contrast with e.g. the Pool class, which creates daemon threads but uses util.Finalize to join them.

(not joining daemon threads may have adverse effects such as resource leaks or otherwise unexpected glitches)

Issue spawned from http://bugs.python.org/issue8713#msg195178.
msg242805 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-05-09 10:55
I've checked the code and this doesn't appear to have been implemented.  I looked into providing a patch myself, but all the returns from calls to util.Finalize that I could find were assigned to different attribute names, so I'm not confident enough to proceed, sorry :(
msg266959 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-06-02 21:36
Capturing some notes from looking at this after Dan Sully asked me questions about the issue during today's pycon sprints:

within multiprocessing/managers.py serve_forever():

the accepter thread is an infinite loop. i think this issue also wants to join() on the threads that the accepter spawns. which also implies they need to be kept track of and that we should change the accepter's while True: into a "while some event that indicates it should exit is not set"...

but we'd also not want a pool keeping track of accepter's infinitely spawned threads to grow forever, so something needs to effectively garbage collect those periodically.
msg267323 - (view) Author: Yusuke Tsutsumi (tsutsumi) * Date: 2016-06-04 21:41
I'm interested in taking this on. I can do the following:

* keep track of the threads in a field attached to the class
* handle thread synchronization and cleanup in the finally block
* set a flag on the exception, which ensures the loop completes, vs immediately passing.

The part I'm unclear on is a good garbage collection strategy. I'm thinking:

* set some configurable period to clean up the threads (every 10s by default. Rapid enough to ensure cleanup, but not fast enough to run every time and significantly slow down responses)
* at the specific threshold, run the cleanup script.
* on cleanup, check every thread if it's alive. explicit del the ones that are not. 

If it sounds good, I can start a patch.
History
Date User Action Args
2022-04-11 14:57:49adminsetgithub: 62951
2016-06-17 08:55:16ned.deilysetnosy: + davin
2016-06-04 21:41:37tsutsumisetnosy: + tsutsumi
messages: + msg267323
2016-06-03 01:03:38BreamoreBoysetnosy: - BreamoreBoy
2016-06-02 21:36:42gregory.p.smithsetnosy: + gregory.p.smith

messages: + msg266959
versions: + Python 3.5, Python 3.6, - Python 3.4
2015-05-09 10:55:04BreamoreBoysetnosy: + BreamoreBoy
messages: + msg242805
2013-08-15 18:13:16pitroucreate