classification
Title: multiprocessing Pool keeps objects (tasks, args, results) alive too long
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, pitrou, sbt
Priority: normal Keywords:

Created on 2017-03-20 18:14 by pitrou, last changed 2017-03-24 15:04 by pitrou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 743 merged pitrou, 2017-03-20 18:43
PR 800 merged pitrou, 2017-03-24 13:15
PR 801 merged pitrou, 2017-03-24 13:30
PR 803 merged pitrou, 2017-03-24 14:53
Messages (6)
msg289894 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-20 18:14
The various workers in multiprocessing.Pool keep a reference to the last encountered task or task result.  This means some data may be kept alive even after the caller is done with them, as long as some other task doesn't clobber the relevant variables.

Specifically, Pool._handle_tasks(), Pool._handle_results() and the toplevel worker() function fail to clear references at the end of each loop.

Originally reported at https://github.com/dask/distributed/issues/956
msg289895 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-20 18:17
Quick patch below.  I'll make a PR once I have time to :-)

diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py
index ffdf426..945afa2 100644
--- a/Lib/multiprocessing/pool.py
+++ b/Lib/multiprocessing/pool.py
@@ -128,6 +128,8 @@ def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None,
             util.debug("Possible encoding error while sending result: %s" % (
                 wrapped))
             put((job, i, (False, wrapped)))
+
+        task = job = result = func = args = kwds = None
         completed += 1
     util.debug('worker exiting after %d tasks' % completed)
 
@@ -402,6 +404,8 @@ class Pool(object):
                 if set_length:
                     util.debug('doing set_length()')
                     set_length(i+1)
+            finally:
+                task = taskseq = job = None
         else:
             util.debug('task handler got sentinel')
 
@@ -445,6 +449,7 @@ class Pool(object):
                 cache[job]._set(i, obj)
             except KeyError:
                 pass
+            task = job = obj = None
 
         while cache and thread._state != TERMINATE:
             try:
@@ -461,6 +466,7 @@ class Pool(object):
                 cache[job]._set(i, obj)
             except KeyError:
                 pass
+            task = job = obj = None
 
         if hasattr(outqueue, '_reader'):
             util.debug('ensuring that outqueue is not full')
msg290086 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-24 12:52
New changeset 8988945cdc27ffa86ba8c624e095b51c459f5154 by Antoine Pitrou in branch 'master':
bpo-29861: release references to multiprocessing Pool tasks (#743)
https://github.com/python/cpython/commit/8988945cdc27ffa86ba8c624e095b51c459f5154
msg290088 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-24 13:45
New changeset cc3331fec8b7a61c3f06c097eac85bfa38490758 by Antoine Pitrou in branch '3.6':
bpo-29861: release references to multiprocessing Pool tasks (#743) (#800)
https://github.com/python/cpython/commit/cc3331fec8b7a61c3f06c097eac85bfa38490758
msg290089 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-24 14:19
New changeset 80cb6ed4db9bae09de1e9ad6001d11cb45a4c1cc by Antoine Pitrou in branch '3.5':
bpo-29861: release references to multiprocessing Pool tasks (#743) (#801)
https://github.com/python/cpython/commit/80cb6ed4db9bae09de1e9ad6001d11cb45a4c1cc
msg290092 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-24 15:03
New changeset 5084ff7ddfe68969d95af12075016f309718aea8 by Antoine Pitrou in branch '2.7':
bpo-29861: release references to multiprocessing Pool tasks (#743) (#803)
https://github.com/python/cpython/commit/5084ff7ddfe68969d95af12075016f309718aea8
History
Date User Action Args
2017-03-24 15:04:38pitrousetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2017-03-24 15:03:48pitrousetmessages: + msg290092
2017-03-24 14:53:25pitrousetpull_requests: + pull_request708
2017-03-24 14:19:20pitrousetmessages: + msg290089
2017-03-24 13:45:36pitrousetmessages: + msg290088
2017-03-24 13:30:30pitrousetpull_requests: + pull_request706
2017-03-24 13:15:28pitrousetpull_requests: + pull_request705
2017-03-24 12:52:13pitrousetmessages: + msg290086
2017-03-20 18:43:52pitrousetpull_requests: + pull_request656
2017-03-20 18:17:28pitrousetmessages: + msg289895
2017-03-20 18:14:14pitroucreate