This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Free lists are still used after being finalized (cleared)
Type: resource usage Stage: resolved
Components: Interpreter Core Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: skrah, vstinner
Priority: normal Keywords: patch

Created on 2020-06-06 08:23 by skrah, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20698 merged vstinner, 2020-06-07 23:02
PR 20700 merged vstinner, 2020-06-07 23:37
Messages (8)
msg370813 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-06-06 08:23
I'm opening a separate issue to prevent #40521 from getting too big.


Lists and tuples sometimes leak (starting 69ac6e58f and later):

==1445== 56 bytes in 1 blocks are definitely lost in loss record 1,542 of 4,898
==1445==    at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
==1445==    by 0x550487: _PyObject_GC_Alloc (gcmodule.c:2233)
==1445==    by 0x550487: _PyObject_GC_Malloc (gcmodule.c:2260)
==1445==    by 0x550487: _PyObject_GC_New (gcmodule.c:2272)
==1445==    by 0x44CB04: PyList_New (listobject.c:144)
==1445==    by 0x4E3DE1: init_filters (_warnings.c:88)
==1445==    by 0x4E3DE1: warnings_init_state (_warnings.c:120)
==1445==    by 0x4E3DE1: _PyWarnings_InitState (_warnings.c:1372)
==1445==    by 0x521720: pycore_init_import_warnings (pylifecycle.c:687)
==1445==    by 0x521720: pycore_interp_init (pylifecycle.c:735)
==1445==    by 0x5246A0: pyinit_config (pylifecycle.c:763)
==1445==    by 0x5246A0: pyinit_core (pylifecycle.c:924)
==1445==    by 0x5246A0: Py_InitializeFromConfig (pylifecycle.c:1134)
==1445==    by 0x4285DC: pymain_init (main.c:66)
==1445==    by 0x4296A1: pymain_main (main.c:653)
==1445==    by 0x4296A1: Py_BytesMain (main.c:686)
==1445==    by 0x578882F: (below main) (libc-start.c:291)

==1445== 64 bytes in 1 blocks are definitely lost in loss record 2,259 of 4,898
==1445==    at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
==1445==    by 0x550611: _PyObject_GC_Alloc (gcmodule.c:2233)
==1445==    by 0x550611: _PyObject_GC_Malloc (gcmodule.c:2260)
==1445==    by 0x550611: _PyObject_GC_NewVar (gcmodule.c:2289)
==1445==    by 0x48452C: tuple_alloc (tupleobject.c:76)
==1445==    by 0x48452C: _PyTuple_FromArray (tupleobject.c:413)
==1445==    by 0x435EE0: _PyObject_MakeTpCall (call.c:165)
==1445==    by 0x436947: _PyObject_FastCallDictTstate (call.c:113)
==1445==    by 0x436947: PyObject_VectorcallDict (call.c:142)
==1445==    by 0x61DFC5: builtin___build_class__ (bltinmodule.c:232)
==1445==    by 0x5E8A39: cfunction_vectorcall_FASTCALL_KEYWORDS (methodobject.c:440)
==1445==    by 0x41F4D5: _PyObject_VectorcallTstate (abstract.h:114)
==1445==    by 0x41F4D5: PyObject_Vectorcall (abstract.h:123)
==1445==    by 0x41F4D5: call_function (ceval.c:5111)
==1445==    by 0x42220E: _PyEval_EvalFrameDefault (ceval.c:3542)
==1445==    by 0x4E6882: _PyEval_EvalFrame (pycore_ceval.h:40)
==1445==    by 0x4E6882: _PyEval_EvalCode (ceval.c:4366)
==1445==    by 0x4E6A65: _PyEval_EvalCodeWithName (ceval.c:4398)
msg370922 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-07 23:04
> Lists and tuples sometimes leak (starting 69ac6e58f and later):

It's not a regression. It's just that bpo-40521 helped Valgrind to detect such bug :-)

I wrote bpo-20698 to fix this issue.

I'm also working on a more generic approach to ensure that we don't add new objects to free lists after free lists are cleared.
msg370924 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-07 23:13
I'm curious about this bug, so I looked at code changes.

Before Python 3.6, warnings.c "garbage" is declared as a static variable. It is initialized once but never cleared.

In Python 3.7, the variable moved into _PyRuntimeState.gc.

In Python 3.8, _PyGC_Fini() started to clear the "garbage" list. This function is called by Py_FinalizeEx() *after* PyList_Fini().
msg370925 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-07 23:22
New changeset 7907f8cbc6923240edb0b5b63adafb871c4c8875 by Victor Stinner in branch 'master':
bpo-40887: Fix finalize_interp_clear() for free lists (GH-20698)
https://github.com/python/cpython/commit/7907f8cbc6923240edb0b5b63adafb871c4c8875
msg370936 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-08 00:14
New changeset bcb198385dee469d630a184182df9dc1463e2c47 by Victor Stinner in branch 'master':
bpo-40887: Don't use finalized free lists (GH-20700)
https://github.com/python/cpython/commit/bcb198385dee469d630a184182df9dc1463e2c47
msg370937 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-08 00:15
Thanks Stefan for the bug report. It's now fixed and I made sure that such bug cannot strike back.
msg370972 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-06-08 11:19
> It's not a regression. It's just that bpo-40521 helped Valgrind to detect such bug :-)

Yes, I suspected that previously reachable global objects were just unreachable after the new free lists.  Thanks for the fix!
msg372147 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-23 09:37
All free lists now have assertions to ensure that they are no longer used after their finalization.
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85064
2020-06-23 09:37:03vstinnersetmessages: + msg372147
2020-06-08 11:19:09skrahsetmessages: + msg370972
2020-06-08 00:15:40vstinnersetstatus: open -> closed

components: + Interpreter Core
versions: + Python 3.10
messages: + msg370937
type: resource usage
resolution: fixed
stage: patch review -> resolved
2020-06-08 00:14:54vstinnersetmessages: + msg370936
2020-06-07 23:37:41vstinnersettitle: Leaks in new free lists -> Free lists are still used after being finalized (cleared)
2020-06-07 23:37:16vstinnersetpull_requests: + pull_request19915
2020-06-07 23:22:39vstinnersetmessages: + msg370925
2020-06-07 23:13:57vstinnersetmessages: + msg370924
2020-06-07 23:04:21vstinnersetmessages: + msg370922
2020-06-07 23:02:48vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19913
2020-06-06 08:23:56skrahcreate