Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Free lists are still used after being finalized (cleared) #85064

Closed
skrah mannequin opened this issue Jun 6, 2020 · 8 comments
Closed

Free lists are still used after being finalized (cleared) #85064

skrah mannequin opened this issue Jun 6, 2020 · 8 comments
Labels
3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage

Comments

@skrah
Copy link
Mannequin

skrah mannequin commented Jun 6, 2020

BPO 40887
Nosy @vstinner, @skrah
PRs
  • bpo-40887: Fix finalize_interp_clear() for free lists #20698
  • bpo-40887: Don't use finalized free lists #20700
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-06-08.00:15:40.459>
created_at = <Date 2020-06-06.08:23:56.632>
labels = ['interpreter-core', '3.10', 'performance']
title = 'Free lists are still used after being finalized (cleared)'
updated_at = <Date 2020-06-23.09:37:03.379>
user = 'https://github.com/skrah'

    bugs.python.org fields:

    activity = <Date 2020-06-23.09:37:03.379>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-08.00:15:40.459>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-06-06.08:23:56.632>
creator = 'skrah'
dependencies = []
files = []
hgrepos = []
issue_num = 40887
keywords = ['patch']
message_count = 8.0
messages = ['370813', '370922', '370924', '370925', '370936', '370937', '370972', '372147']
nosy_count = 2.0
nosy_names = ['vstinner', 'skrah']
pr_nums = ['20698', '20700']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue40887'
versions = ['Python 3.10']
    @skrah
    Copy link
    Mannequin Author

    skrah mannequin commented Jun 6, 2020

    I'm opening a separate issue to prevent bpo-40521 from getting too big.

    Lists and tuples sometimes leak (starting 69ac6e5 and later):

    ==1445== 56 bytes in 1 blocks are definitely lost in loss record 1,542 of 4,898
    ==1445== at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
    ==1445== by 0x550487: _PyObject_GC_Alloc (gcmodule.c:2233)
    ==1445== by 0x550487: _PyObject_GC_Malloc (gcmodule.c:2260)
    ==1445== by 0x550487: _PyObject_GC_New (gcmodule.c:2272)
    ==1445== by 0x44CB04: PyList_New (listobject.c:144)
    ==1445== by 0x4E3DE1: init_filters (_warnings.c:88)
    ==1445== by 0x4E3DE1: warnings_init_state (_warnings.c:120)
    ==1445== by 0x4E3DE1: _PyWarnings_InitState (_warnings.c:1372)
    ==1445== by 0x521720: pycore_init_import_warnings (pylifecycle.c:687)
    ==1445== by 0x521720: pycore_interp_init (pylifecycle.c:735)
    ==1445== by 0x5246A0: pyinit_config (pylifecycle.c:763)
    ==1445== by 0x5246A0: pyinit_core (pylifecycle.c:924)
    ==1445== by 0x5246A0: Py_InitializeFromConfig (pylifecycle.c:1134)
    ==1445== by 0x4285DC: pymain_init (main.c:66)
    ==1445== by 0x4296A1: pymain_main (main.c:653)
    ==1445== by 0x4296A1: Py_BytesMain (main.c:686)
    ==1445== by 0x578882F: (below main) (libc-start.c:291)

    ==1445== 64 bytes in 1 blocks are definitely lost in loss record 2,259 of 4,898
    ==1445== at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
    ==1445== by 0x550611: _PyObject_GC_Alloc (gcmodule.c:2233)
    ==1445== by 0x550611: _PyObject_GC_Malloc (gcmodule.c:2260)
    ==1445== by 0x550611: _PyObject_GC_NewVar (gcmodule.c:2289)
    ==1445== by 0x48452C: tuple_alloc (tupleobject.c:76)
    ==1445== by 0x48452C: _PyTuple_FromArray (tupleobject.c:413)
    ==1445== by 0x435EE0: _PyObject_MakeTpCall (call.c:165)
    ==1445== by 0x436947: _PyObject_FastCallDictTstate (call.c:113)
    ==1445== by 0x436947: PyObject_VectorcallDict (call.c:142)
    ==1445== by 0x61DFC5: builtin___build_class__ (bltinmodule.c:232)
    ==1445== by 0x5E8A39: cfunction_vectorcall_FASTCALL_KEYWORDS (methodobject.c:440)
    ==1445== by 0x41F4D5: _PyObject_VectorcallTstate (abstract.h:114)
    ==1445== by 0x41F4D5: PyObject_Vectorcall (abstract.h:123)
    ==1445== by 0x41F4D5: call_function (ceval.c:5111)
    ==1445== by 0x42220E: _PyEval_EvalFrameDefault (ceval.c:3542)
    ==1445== by 0x4E6882: _PyEval_EvalFrame (pycore_ceval.h:40)
    ==1445== by 0x4E6882: _PyEval_EvalCode (ceval.c:4366)
    ==1445== by 0x4E6A65: _PyEval_EvalCodeWithName (ceval.c:4398)

    @vstinner
    Copy link
    Member

    vstinner commented Jun 7, 2020

    Lists and tuples sometimes leak (starting 69ac6e5 and later):

    It's not a regression. It's just that bpo-40521 helped Valgrind to detect such bug :-)

    I wrote bpo-20698 to fix this issue.

    I'm also working on a more generic approach to ensure that we don't add new objects to free lists after free lists are cleared.

    @vstinner
    Copy link
    Member

    vstinner commented Jun 7, 2020

    I'm curious about this bug, so I looked at code changes.

    Before Python 3.6, warnings.c "garbage" is declared as a static variable. It is initialized once but never cleared.

    In Python 3.7, the variable moved into _PyRuntimeState.gc.

    In Python 3.8, _PyGC_Fini() started to clear the "garbage" list. This function is called by Py_FinalizeEx() *after* PyList_Fini().

    @vstinner
    Copy link
    Member

    vstinner commented Jun 7, 2020

    New changeset 7907f8c by Victor Stinner in branch 'master':
    bpo-40887: Fix finalize_interp_clear() for free lists (GH-20698)
    7907f8c

    @vstinner vstinner changed the title Leaks in new free lists Free lists are still used after being finalized (cleared) Jun 7, 2020
    @vstinner vstinner changed the title Leaks in new free lists Free lists are still used after being finalized (cleared) Jun 7, 2020
    @vstinner
    Copy link
    Member

    vstinner commented Jun 8, 2020

    New changeset bcb1983 by Victor Stinner in branch 'master':
    bpo-40887: Don't use finalized free lists (GH-20700)
    bcb1983

    @vstinner
    Copy link
    Member

    vstinner commented Jun 8, 2020

    Thanks Stefan for the bug report. It's now fixed and I made sure that such bug cannot strike back.

    @vstinner vstinner added 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jun 8, 2020
    @vstinner vstinner closed this as completed Jun 8, 2020
    @vstinner vstinner added performance Performance or resource usage 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jun 8, 2020
    @vstinner vstinner closed this as completed Jun 8, 2020
    @vstinner vstinner added the performance Performance or resource usage label Jun 8, 2020
    @skrah
    Copy link
    Mannequin Author

    skrah mannequin commented Jun 8, 2020

    It's not a regression. It's just that bpo-40521 helped Valgrind to detect such bug :-)

    Yes, I suspected that previously reachable global objects were just unreachable after the new free lists. Thanks for the fix!

    @vstinner
    Copy link
    Member

    All free lists now have assertions to ensure that they are no longer used after their finalization.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner