classification
Title: Py_Finalize() doesn't clear all Python objects at exit
Type: resource usage Stage: patch review
Components: Interpreter Core Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex Budovski, amaury.forgeotdarc, isoschiz, kylotan, pconnell, santoso.wijaya, shihai1991, tlesher, vstinner, ysj.ray
Priority: normal Keywords: patch

Created on 2007-01-15 10:26 by kylotan, last changed 2020-01-16 16:27 by shihai1991.

Pull Requests
URL Status Linked Edit
PR 17835 merged shihai1991, 2020-01-05 13:28
PR 17883 closed shihai1991, 2020-01-07 00:36
PR 18030 open shihai1991, 2020-01-16 11:57
PR 18032 open shihai1991, 2020-01-16 16:27
Messages (15)
msg61054 - (view) Author: B Sizer (kylotan) Date: 2007-01-15 10:26
This C code:

#include <Python.h>
int main(int argc, char *argv[])
{
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
    Py_Initialize(); Py_Finalize();
}

Produces this output:
[7438 refs]
[7499 refs]
[7550 refs]
[7601 refs]
[7652 refs]
[7703 refs]
[7754 refs]

A similar program configured to call the Py_Initialize()/Py_Finalize() 1000 times ends up with:
...
[58295 refs]
[58346 refs]
[58397 refs]

This is with a fresh debug build of Python 2.5.0 on Windows XP, using Visual C++ 2003.
msg110895 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-20 13:41
Does the title of this issue accurately reflect the current status of the Python interpreter?
msg111024 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-07-21 09:20
Yes, some objects are not cleaned in finalization.
This is not a problem in usual cases though, when the interpreter is
started only once.
msg130729 - (view) Author: ysj.ray (ysj.ray) Date: 2011-03-13 07:12
> Does the title of this issue accurately reflect the current status of the Python interpreter?

Yes, here is the running result on current 3.3 latest code:
[37182 refs]
[39415 refs]
[41607 refs]
[43799 refs]
[45991 refs]
[48183 refs]
[50375 refs]


This seems to be a known bug that Py_Finalize() doesn't free all objects according doc http://docs.python.org/dev/c-api/init.html?highlight=py_finalize#Py_Finalize
msg248761 - (view) Author: Alex Budovski (Alex Budovski) Date: 2015-08-18 06:20
Interestingly enough, some of the leaked memory came from the finalize routine itself! Here's one example:

0:004> !heap -p -a 0x000000DB144346F0
    address 000000db144346f0 found in
    _HEAP @ db0cae0000
              HEAP_ENTRY Size Prev Flags            UserPtr UserSize - state
        000000db14434690 030a 0000  [00]   000000db144346c0    03074 - (busy)
        7ffc55628b04 ntdll!RtlpCallInterceptRoutine+0x0000000000000040
        7ffc555f9f36 ntdll!RtlAllocateHeap+0x0000000000079836
        7ffc2a60c4da ucrtbased!calloc_base+0x000000000000123a
        7ffc2a60c27d ucrtbased!calloc_base+0x0000000000000fdd
        7ffc2a60f34f ucrtbased!malloc_dbg+0x000000000000002f
        7ffc2a60fdde ucrtbased!malloc+0x000000000000001e
        5a5e6ef9 python36_d!_PyMem_RawMalloc+0x0000000000000029
        5a5e78c7 python36_d!_PyMem_DebugAlloc+0x0000000000000087
        5a5e5e6f python36_d!_PyMem_DebugMalloc+0x000000000000001f
        5a5e7230 python36_d!PyMem_Malloc+0x0000000000000030
        5a582047 python36_d!new_keys_object+0x0000000000000077
        5a57f7c5 python36_d!dictresize+0x0000000000000085
        5a57a4b2 python36_d!PyDict_Merge+0x0000000000000112
        5a57bf33 python36_d!PyDict_Update+0x0000000000000023
        5a75fb1d python36_d!PyImport_Cleanup+0x000000000000045d
        5a778f9e python36_d!Py_Finalize+0x000000000000005e
msg355187 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:11
I tested on the master branch of Python:
---
#include <Python.h>

void func()
{
    Py_Initialize(); Py_Finalize();
    Py_ssize_t cnt = _Py_GetRefTotal();
    printf("sys.gettotalrefcount(): %zd\n", cnt);
}

int main(int argc, char *argv[])
{
    Py_SetProgramName(L"./_testembed");
    for (int i=0; i < 10; i++) {
        func();
    }
}
---

Each iteration leaks around 5,000 Python objects:
---
sys.gettotalrefcount(): 15113
sys.gettotalrefcount(): 19527
sys.gettotalrefcount(): 23941
sys.gettotalrefcount(): 28355
sys.gettotalrefcount(): 32769
sys.gettotalrefcount(): 37183
sys.gettotalrefcount(): 41597
sys.gettotalrefcount(): 46011
sys.gettotalrefcount(): 50425
sys.gettotalrefcount(): 54839
---
msg355189 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:12
I marked bpo-6741 as a duplicate of this issue.
msg355191 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:16
I marked bpo-26888 as a duplicate of this issue.
msg355193 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:17
I marked bpo-21387 as a duplicate of this issue.
msg355194 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:19
One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP 489 "Multi-phase extension module initialization".
msg355201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:32
I marked bpo-32026 as a duplicate of this issue.
msg359342 - (view) Author: hai shi (shihai1991) * Date: 2020-01-05 13:30
One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP 489 "Multi-phase extension module initialization".
> I try to port _json extension module to multiphase initialization module, but the baseline(using victor's code) in my vm not changed~
msg359482 - (view) Author: hai shi (shihai1991) * Date: 2020-01-07 00:41
Compare to _Py_ForgetReference(), _Py_INC_REFTOTAL in _Py_NewReference() looks redundant.

REF: https://github.com/python/cpython/blob/master/Include/object.h#L442

master brach baseline in my vm:
```
sys.gettotalrefcount(): 18049
sys.gettotalrefcount(): 22463
```

after PR17883
```
sys.gettotalrefcount(): 17589
sys.gettotalrefcount(): 22000
```
msg359830 - (view) Author: hai shi (shihai1991) * Date: 2020-01-12 03:11
FWIW, i counted the difference of each file's refs after `Py_Finalize()`.

[('Objects/dictobject.c', 21434), ('Python/marshal.c', 8135), ('Objects/codeobject.c', 6245), ('Objects/listobject.c', 6037), ('Objects/tupleobject.c', 4169), ('Objects/boolobject.c', 2433), ('Objects/object.c', 2364), ('Objects/unicodeobject.c', 1541), ('Objects/longobject.c', 1387), ('Objects/funcobject.c', 528), ('Objects/classobject.c', 528), ('Objects/abstract.c', 463), ('Python/structmember.c', 369), ('./Include/objimpl.h', 277), ('Objects/stringlib/partition.h', 273), ('Python/import.c', 259), ('Python/codecs.c', 197), ('./Modules/signalmodule.c', 61), ('./Modules/_threadmodule.c', 59), ('Objects/exceptions.c', 15), ('Objects/bytesobject.c', 5), ('./Modules/_weakref.c', 4), ('Python/_warnings.c', 3), ('./Modules/timemodule.c', 1), ('./Modules/_codecsmodule.c', 1), ('Objects/bytearrayobject.c', 1), ('Python/compile.c', 1), ('Objects/sliceobject.c', 0), ('Objects/memoryobject.c', 0), ('Python/context.c', -1), ('Objects/clinic/longobject.c.h', -1), ('Objects/enumobject.c', -1), ('Modules/gcmodule.c', -1), ('Objects/namespaceobject.c', -1), ('Objects/stringlib/unicode_format.h', -2), ('Objects/rangeobject.c', -3), ('Python/pystate.c', -4), ('Objects/fileobject.c', -14), ('./Modules/_io/clinic/bufferedio.c.h', -17), ('./Modules/_io/iobase.c', -21), ('Python/modsupport.c', -28), ('./Modules/_io/fileio.c', -28), ('Python/pylifecycle.c', -37), ('./Modules/_io/textio.c', -39), ('Objects/genobject.c', -53), ('Objects/weakrefobject.c', -54), ('./Modules/_io/bufferedio.c', -56), ('./Python/sysmodule.c', -68), ('./Modules/_io/_iomodule.c', -82), ('Python/errors.c', -90), ('Objects/descrobject.c', -110), ('Objects/structseq.c', -113), ('Python/bltinmodule.c', -118), ('Objects/setobject.c', -339), ('Objects/moduleobject.c', -454), ('./Modules/posixmodule.c', -614), ('./Modules/_abc.c', -664), ('Objects/call.c', -755), ('Objects/typeobject.c', -2035), ('Objects/frameobject.c', -6538), ('Python/ceval.c', -7857), ('./Include/object.h', -48292)]
msg360063 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-15 16:32
New changeset ed154c387efc5f978ec97900ec9e0ec6631d5498 by Victor Stinner (Hai Shi) in branch 'master':
bpo-1635741: Port _json extension module to multiphase initialization (PEP 489) (GH-17835)
https://github.com/python/cpython/commit/ed154c387efc5f978ec97900ec9e0ec6631d5498
History
Date User Action Args
2020-01-16 16:27:33shihai1991setpull_requests: + pull_request17428
2020-01-16 11:57:57shihai1991setpull_requests: + pull_request17425
2020-01-15 16:32:55vstinnersetmessages: + msg360063
2020-01-12 03:11:46shihai1991setmessages: + msg359830
2020-01-07 00:41:22shihai1991setmessages: + msg359482
2020-01-07 00:36:19shihai1991setpull_requests: + pull_request17299
2020-01-05 13:30:21shihai1991setnosy: + shihai1991
messages: + msg359342
2020-01-05 13:28:20shihai1991setkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request17262
2019-10-23 00:32:00vstinnersetmessages: + msg355201
2019-10-23 00:31:44vstinnerlinkissue32026 superseder
2019-10-23 00:19:27vstinnersetmessages: + msg355194
2019-10-23 00:17:06vstinnersetmessages: + msg355193
2019-10-23 00:16:52vstinnerlinkissue21387 superseder
2019-10-23 00:16:01vstinnersetmessages: + msg355191
2019-10-23 00:15:53vstinnerlinkissue26888 superseder
2019-10-23 00:12:59vstinnersetmessages: + msg355189
2019-10-23 00:12:45vstinnerlinkissue6741 superseder
2019-10-23 00:11:30vstinnersetnosy: + vstinner
title: Interpreter seems to leak references after finalization -> Py_Finalize() doesn't clear all Python objects at exit
messages: + msg355187

versions: + Python 3.9, - Python 3.1, Python 2.7, Python 3.2
2015-08-18 06:20:57Alex Budovskisetnosy: + Alex Budovski
messages: + msg248761
2014-02-03 18:32:01BreamoreBoysetnosy: - BreamoreBoy
2013-04-20 10:04:07isoschizsetnosy: + pconnell, isoschiz
2011-03-31 11:49:57tleshersetnosy: + tlesher
2011-03-13 09:18:08santoso.wijayasetnosy: + santoso.wijaya
2011-03-13 07:12:36ysj.raysetnosy: amaury.forgeotdarc, kylotan, ysj.ray, BreamoreBoy
messages: + msg130729
2011-02-14 03:07:28ysj.raysetnosy: + ysj.ray
2010-07-21 10:21:13amaury.forgeotdarclinkissue8258 superseder
2010-07-21 09:20:21amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg111024
2010-07-20 13:41:15BreamoreBoysetnosy: + BreamoreBoy

messages: + msg110895
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6, Python 3.0
2009-03-30 19:04:21ajaksu2setstage: test needed
type: resource usage
versions: + Python 2.6, Python 3.0, - Python 2.5
2007-01-15 10:26:05kylotancreate