Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ProcessPoolExecutor exception memory leak #90943

Closed
themylogin mannequin opened this issue Feb 18, 2022 · 2 comments
Closed

ProcessPoolExecutor exception memory leak #90943

themylogin mannequin opened this issue Feb 18, 2022 · 2 comments
Labels
3.9 only security fixes performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@themylogin
Copy link
Mannequin

themylogin mannequin commented Feb 18, 2022

BPO 46787
Nosy @themylogin
PRs
  • bpo-46787: Fix ProcessPoolExecutor exception memory leak #31408
    		•
    Files
  • 1.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2022-02-18.15:16:10.733>
labels = ['library', '3.9', 'performance']
title = 'ProcessPoolExecutor exception memory leak'
updated_at = <Date 2022-02-18.15:23:21.328>
user = 'https://github.com/themylogin'

    bugs.python.org fields:

    activity = <Date 2022-02-18.15:23:21.328>
actor = 'python-dev'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-18.15:16:10.733>
creator = 'themylogin'
dependencies = []
files = ['50628']
hgrepos = []
issue_num = 46787
keywords = ['patch']
message_count = 1.0
messages = ['413485']
nosy_count = 2.0
nosy_names = ['python-dev', 'themylogin']
pr_nums = ['31408']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46787'
versions = ['Python 3.9']
    @themylogin
    Copy link
    Mannequin Author

    themylogin mannequin commented Feb 18, 2022

    If an exception occurs in ProcessPoolExecutor work item, all the exception frame local variables are not garbage collected (or are garbage collected too lately) because they are referenced by the exception's traceback.

    Attached file is a test case. With unpatched python 3.9 (debian bullseye) it prints:

    root@truenas[~/freenas/freenas]# python test.py
    At iteration 0 memory usage is 226070528
    At iteration 1 memory usage is 318763008
    At iteration 2 memory usage is 318509056
    At iteration 3 memory usage is 321662976
    At iteration 4 memory usage is 321404928
    At iteration 5 memory usage is 324292608
    At iteration 6 memory usage is 324296704
    At iteration 7 memory usage is 326922240
    At iteration 8 memory usage is 326922240
    At iteration 9 memory usage is 329543680

    With the proposed patch there is no memory usage growth:

    At iteration 0 memory usage is 226410496
    At iteration 1 memory usage is 226451456
    At iteration 2 memory usage is 226451456
    At iteration 3 memory usage is 226443264
    At iteration 4 memory usage is 226443264
    At iteration 5 memory usage is 226435072
    At iteration 6 memory usage is 226426880
    At iteration 7 memory usage is 226426880
    At iteration 8 memory usage is 226435072
    At iteration 9 memory usage is 226426880

    @themylogin themylogin mannequin added 3.9 only security fixes stdlib Python modules in the Lib dir performance Performance or resource usage labels Feb 18, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @clchiou
    Copy link

    clchiou commented Sep 25, 2022 •
    edited

    I wonder if this patch also needs to be applied to Lib/multiprocessing/pool.py as it appears to me that Lib/concurrent/futures/process.py and Lib/multiprocessing/pool.py are quite similar in this aspect: sending exceptions with (stringified) traceback across process boundary. What do you think? @themylogin

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes performance Performance or resource usage stdlib Python modules in the Lib dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @clchiou @kumaraditya303