classification
Title: ProcessPool workers hold onto return value of last task in memory
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: bquinlan, dchevell, pablogsal, pitrou
Priority: normal Keywords: patch, patch, patch, patch

Created on 2019-01-11 08:36 by dchevell, last changed 2019-03-16 22:29 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11514 merged dchevell, 2019-01-11 08:41
PR 11514 merged dchevell, 2019-01-11 08:41
PR 11514 merged dchevell, 2019-01-11 08:41
PR 11514 merged dchevell, 2019-01-11 08:41
Messages (2)
msg333444 - (view) Author: David Chevell (dchevell) * Date: 2019-01-11 08:36
ProcessPoolExecutor workers will hold onto the return value of their last task in memory until the next task is received. Since the return value has already been propagated to the parent process's `Future` or else effectively discarded, this is holding onto objects unnecessarily.

Simple case to reproduce:

    import concurrent.futures
    import time

    executor = concurrent.futures.ProcessPoolExecutor(max_workers=1)

    def big_val():
        return [{1:1} for i in range(1, 1000000)]

    executor.submit(big_val)

    # Observe the memory usage of the process worker during the sleep interval
    time.sleep(10)


This should be easily fixed by having the worker explicitly `del r` after calling `_sendback_result` as it already does this for `call_item`
msg338105 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-03-16 22:28
New changeset 962bdeab191ee64459caa199209331005797ea7a by Pablo Galindo (Dave Chevell) in branch 'master':
bpo-35715: Liberate return value of _process_worker (GH-11514)
https://github.com/python/cpython/commit/962bdeab191ee64459caa199209331005797ea7a
History
Date User Action Args
2019-03-16 22:29:06pablogsalsetkeywords: patch, patch, patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-03-16 22:28:53pablogsalsetnosy: + pablogsal
messages: + msg338105
2019-01-13 18:25:56xtreaksetkeywords: patch, patch, patch, patch
nosy: + bquinlan, pitrou
2019-01-11 08:42:00dchevellsetkeywords: + patch
stage: patch review
pull_requests: + pull_request11078
2019-01-11 08:41:56dchevellsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11077
2019-01-11 08:41:52dchevellsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11076
2019-01-11 08:41:48dchevellsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11075
2019-01-11 08:36:21dchevellcreate