Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Py_Finalize() doesn't clear all Python objects at exit #44470

Closed
kylotan mannequin opened this issue Jan 15, 2007 · 188 comments
Closed

Py_Finalize() doesn't clear all Python objects at exit #44470

kylotan mannequin opened this issue Jan 15, 2007 · 188 comments
Labels
3.11 bug and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage topic-subinterpreters

Comments

@kylotan
Copy link
Mannequin

kylotan mannequin commented Jan 15, 2007

BPO 1635741
Nosy @amauryfa, @orsenthil, @vstinner, @tiran, @phsilva, @encukou, @ambv, @ericsnowcurrently, @serhiy-storchaka, @phmc, @DimitrisJim, @corona10, @miss-islington, @shihai1991, @erlend-aasland, @koubaa, @h-vetinari, @csabesz88, @kumaraditya303, @CharlieZhao95
PRs
  • bpo-1635741: Port _json extension module to multiphase initialization(PEP 489) #17835
  • bpo-1635741: Remove redundant _Py_INC_TPALLOCS in _Py_NewReference() #17883
  • bpo-1635741: Port _abc extension module to multiphase initialization(PEP 489) #18030
  • bpo-1635741: Port _asyncio extension module to multiphase initialization(PEP 489) #18032
  • bpo-1635741: Port _bisect extension module to multiphase initialization(PEP 489) #18049
  • bpo-1635741: Port _bz2 extension module to multiphase initialization(PEP 489) #18050
  • bpo-1635741: Port _codecs extension module to multiphase initialization (PEP 489) #18065
  • bpo-1635741: Port _collections extension module to multiphase initialization(PEP 489) #18066
  • bpo-1635741: Port _locale extension module to multiphase initialization (PEP 489) #18358
  • bpo-1635741: Calling Py_INCREF() after PyModule_AddObject() success to run #18365
  • bpo-1635741: Port _contextvars extension module to multiphase initialization (PEP … #18374
  • bpo-1635741: Port _crypt extension module to multiphase initialization(PEP 489) #18404
  • bpo-1635741: Fix refleaks of timemoudle in PyInit_time() #18486
  • bpo-1635741: Port audioop extension module to multiphase initialization(PEP 489) #18608
  • bpo-1635741: Fix potential refleaks in binascii module #18613
  • bpo-1635741: Port _ctypes_test extension module to multiphase initialization(PEP … #19012
  • bpo-1635741: Port statistics module to multiphase initialization #19015
  • bpo-1635741: Fix possible refleaks in tupleobject.c #19018
  • [3.7] bpo-1635741: Fix a possible refleak in tupleobject.c (GH-19018) #19022
  • [3.8] bpo-1635741: Fix a possible refleak in tupleobject.c (GH-19018) #19021
  • bpo-1635741: Port itertools module to multiphase initialization. #19044
  • bpo-1635741: Port _heapq module to multiphase initialization. #19057
  • bpo-39337: Add a test case for normalizing of codec names #19069
  • [WIP] bpo-1635741: Port _collections module to multiphase initialization. #19071
  • bpo-1635741: Port _collections module to multiphase initialization.  #19074
  • bpo-1635741: Port _weakref extension module to multiphase initialization(PEP 489) #19084
  • bpo-1635741: Enhance get_binascii_state() #19100
  • bpo-1635741: Port time module to multiphase initialization (PEP 489) #19107
  • bpo-1635741: Port _datetime module to multiphase initialization (PEP 489) #19122
  • Revert "bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084)" #19128
  • bpo-40050: Fix importlib._bootstrap_external #19135
  • bpo-1635741: Port _weakref extension module to multiphase initialization (PEP 489) (GH-19084) #19140
  • bpo-1635741: Port operator module to multiphase initialization (PEP 489) #19150
  • bpo-1635741: Port _functools module to multiphase initialization (PEP-489) #19151
  • bpo-1635741: Port _uuid module to multiphase initialization. #19242
  • bpo-1635741: Port math module to multiphase initialization #19243
  • bpo-1635741: Port resource extension module to multiphase initialization(PEP 489) #19252
  • bpo-1635741: Enhance PyModule_AddIntMacro by ADD_INT macro #19307
  • bpo-1635741: Port _lzma module to multiphase initialization  #19382
  • bpo-1635741: Port mmap module to multiphase initialization #19459
  • bpo-1635741: Port _stat module to multiphase initialization #19798
  • bpo-1635741: Fix compiler warning in _stat.c #19822
  • bpo-1635741: Port syslog module to multiphase initialization #19907
  • bpo-1635741: Port errno module to multiphase initialization #19923
  • bpo-1635741: Port fcntl module to multiphase initialization #20540
  • [WIP] bpo-1635741: Py_Finalize() finalizes builtin static types #20763
  • bpo-1635741: Port nis module to multiphase initialization. #20808
  • bpo-1635741: Port _dbm module to multiphase initialization #20848
  • bpo-1635741: Port _gdbm module to multiphase initialization #20920
  • bpo-1635741: Enable unicode_release_interned() without insure or valgrind. #21087
  • bpo-1635741: Port termios module to multiphase initialization #21168
  • bpo-1635741: Port sha256 module to multiphase init (PEP 489) #21189
  • bpo-1635741: Release Unicode interned strings at exit #21269
  • bpo-1635741: Fix unicode_dealloc() for mortal interned string #21270
  • bpo-1635741: Port faulthandler module to multiphase initialization #21294
  • bpo-1635741 - try to port pickle to multi-stage init #21319
  • bpo-1635741: port winapi to multi-stage init #21371
  • bpo-1635741 unicodedata #21375
  • bpo-1635741: port multiprocessing to multiphase init #21378
  • bpo-1635741: prepare for multi-phase init for unicodedata #21418
  • bpo-1635741: Clean sysdict and builtins of interpreter #21605
  • bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818
  • bpo-1635741: port sha3 module to multi-phase init #21855
  • bpo-1635741: port blake2 module to multi-phase init #21856
  • bpo-1635741: Fix refleaks of encodings module by removing the encodings._aliases #21896
  • bpo-1635741: Fix reference cycle by calling explicit gc collection in main interpreter #21902
  • bpo-1635741 port some simple modules to multi-phase initialization #21985
  • bpo-1635741 port _curses_panel to multi-phase init (PEP 489) #21986
  • bpo-1635741 port zlib module to multi-phase init #21995
  • bpo-1635741 port _testbuffer to multi-phase init #22003
  • bpo-1635741 port signalmodule to multi-phase init (PEP 489) #22049
  • bpo-1635741: port opcode module to multi-phase init (PEP 489) #22050
  • bpo-1635741: - port overlapped to multi-phase #22051
  • bpo-1635741: testbuffer heap types #22131
  • bpo-1635741: sha256 heap types #22134
  • bpo-1635741: Port the termios to multi-phase init (PEP 489) #22139
  • bpo-1635741 - port unicodedata to multi-phase init #22145
  • bpo-1635741: Port _string module to multi-phase init #22148
  • bpo-1635741: Port mashal module to multi-phase init #22149
  • bpo-1635741: port scproxy to multi-phase init #22164
  • bpo-1635741 port cmath to multi-phase init (pep 489) #22165
  • bpo-1635741 port _lsprof to multi-phase init (PEP 489) #22220
  • bpo-1635741: port pyexpat to multi-phase init (PEP 489) #22222
  • bpo-1635741: convert an _lsprof method to argument clinic #22240
  • bpo-41790: update error in documentation #22242
  • bpo-1635741: replace UCD_Check with Py_IS_TYPE #22328
  • bpo-1635741: Port _bisect module to multi-phase init #22415
  • bpo-41861: Convert _sqlite3 CursorType and ConnectionType to heap types #22478
  • bpo-1635741: pyexpat error handling #22489
  • bpo-1635741: convert unicode ucd type to heap type #22490
  • bpo-1635741: Add a global module state to unicodedata #22712
  • bpo-1635741: Move _PyUnicode_Name_CAPI to the internal C API #22713
  • bpo-1635741: Port _csv to multi-phase init #22838
  • bpo-1635741: Fix NULL ptr deref in multiprocessing #22880
  • gh-103092: Prep curses module for multi-phase init #23091
  • bpo-1635741: Add PyModule_AddObjectRef() function #23122
  • bpo-1635741: _ast uses PyModule_AddObjectRef() #23146
  • bpo-1635741: _contextvars uses PyModule_AddType() #23147
  • bpo-1635741: _sqlite3 uses PyModule_AddObjectRef #23148
  • bpo-1635741: _warnings uses PyModule_AddObjectRef() #23151
  • bpo-1635741: sqlite3 now uses Py_NewRef and Py_XNewRef #23170
  • bpo-1635741: Enhance _datetime error handling #23139
  • bpo-1635741: port pickle to multi-phase init (PEP 489) #23188
  • bpo-1635741: Fix typo in PyModule_AddObjectRef() doc #23234
  • bpo-40170: Fix PyType_Ready() refleak on static type #23236
  • bpo-1635741: In pickle module, inject module state from class methods #23304
  • bpo-1635741: Port _hashlib to multiphase initialization (GH-23358) #23358
  • bpo-1635741: Port _random to multiphase initialization #23359
  • bpo-1635741: Port grp and pwd to multiphase initialization #23360
  • bpo-1635741: Port symtable module to multiphase initialization #23361
  • bpo-1635741: Port _queue to multiphase initialization #23376
  • bpo-1635741: Port gc module to multiphase initialization #23377
  • bpo-1635741: Convert _imp to multi-phase init #23378
  • bpo-1635741: Port _warnings to the multi-phase init #23379
  • bpo-1635741: Port _random to the multi-phase init #23381
  • bpo-1635741: Port spwd to multiphase initialization (GH-23390) #23390
  • bpo-1635741: Convert _sre types to heap types and establish module state (PEP 384) #23393
  • bpo-1635741: Port _struct to multiphase initialization #23398
  • Revert "bpo-1635741: Port _struct to multiphase initialization" #23401
  • bpo-1635741: Fix _struct for build bot error #23402
  • bpo-1635741: Port _posixshmem extension module to multiphase initialization #23404
  • bpo-1635741: Port _posixsubprocess module to multiphase init (GH-23406) #23406
  • bpo-1635741: Port select module to multiphase init (GH-23409) #23409
  • bpo-1635741: Port resource extension module to module state (GH-23462) #23462
  • bpo-42327: Add PyModule_Add() (smaller). #23443
  • bpo-42327: Add PyModule_Add() (smaller). #23443
  • bpo-1635741: port _elementtree to multi-phase init (PEP 489) #23535
  • Add symbols of the stable ABI to python3dll.c #23598
  • bpo-1635741: Refactor _threadmodule.c #23793
  • bpo-1635741: Port _thread to multiphase init #23811
  • bpo-1635741: Fix ref. leak introduced by bf64d90 #23972
  • bpo-1635741: Adapt _multibytecodec to multi-phase initialization #24095
  • bpo-1635741: Convert socket.SocketType to heap type, and establish global state #24175
  • bpo-1635741: Fix PyModule_AddObjectRef to use EXPORT_FUNC #24205
  • bpo-1635741: Port _datetime extension module to multiphase initialization (PEP 489) #30522
  • bpo-1635741: port _ctypes extension module to multiphase initialization  #30525
  • bpo-46417: Call _PyDebug_PrintTotalRefs() later #30744
  • bpo-1635741: test_embed cheks that Python does not leak #31555
  • bpo-1635741: Fix winreg reference leaks #31560
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2022-01-27.13:42:43.202>
    created_at = <Date 2007-01-15.10:26:05.000>
    labels = ['expert-subinterpreters', 'interpreter-core', '3.11', 'performance']
    title = "Py_Finalize() doesn't clear all Python objects at exit"
    updated_at = <Date 2022-02-25.11:34:18.230>
    user = 'https://bugs.python.org/kylotan'

    bugs.python.org fields:

    activity = <Date 2022-02-25.11:34:18.230>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2022-01-27.13:42:43.202>
    closer = 'vstinner'
    components = ['Interpreter Core', 'Subinterpreters']
    creation = <Date 2007-01-15.10:26:05.000>
    creator = 'kylotan'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 1635741
    keywords = ['patch']
    message_count = 185.0
    messages = ['61054', '110895', '111024', '130729', '248761', '355187', '355189', '355191', '355193', '355194', '355201', '359342', '359482', '359830', '360063', '361427', '361428', '361466', '361798', '362068', '362124', '362143', '362144', '362195', '363935', '363936', '363937', '363940', '363941', '364234', '364330', '364379', '364463', '364521', '364609', '364656', '364833', '364836', '364845', '364871', '364883', '364906', '364909', '364951', '364952', '364968', '364971', '364972', '364973', '364975', '364987', '365008', '365017', '365043', '365386', '365388', '365484', '365584', '365611', '367686', '367796', '368096', '368318', '370561', '370763', '371119', '371569', '371683', '372061', '372077', '372096', '372102', '372103', '372797', '372912', '372938', '372942', '375169', '375272', '375286', '375287', '375288', '375290', '375311', '375567', '376218', '376221', '376240', '376288', '376360', '376461', '376474', '376477', '376480', '376493', '376500', '376552', '376554', '376555', '376556', '376566', '376567', '376571', '376572', '376616', '376687', '376993', '377253', '377364', '377365', '377523', '378115', '378685', '379280', '379287', '379667', '380216', '380270', '380280', '380317', '380318', '380331', '380332', '380340', '380346', '380351', '380733', '380758', '380854', '380857', '381124', '381300', '381317', '381318', '381351', '381352', '381376', '381379', '381391', '381392', '381395', '381396', '381400', '381411', '381423', '381426', '381431', '381432', '381434', '381476', '381477', '381571', '381572', '383155', '383168', '383276', '383283', '383285', '383287', '383454', '383636', '383785', '383799', '383801', '383844', '383847', '383880', '384039', '384042', '384326', '384352', '384646', '384978', '384987', '385010', '388728', '410809', '411116', '411278', '411311', '411872', '411961', '411997', '413938', '413979']
    nosy_count = 26.0
    nosy_names = ['amaury.forgeotdarc', 'orsenthil', 'kylotan', 'vstinner', 'christian.heimes', 'tlesher', 'phsilva', 'petr.viktorin', 'ysj.ray', 'santoso.wijaya', 'lukasz.langa', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'pconnell', 'isoschiz', 'Jim Fasarakis-Hilliard', 'corona10', 'miss-islington', 'shihai1991', 'erlendaasland', 'koubaa', 'h-vetinari', 'TCsaba', 'kumaraditya', 'CharlieZhao']
    pr_nums = ['17835', '17883', '18030', '18032', '18049', '18050', '18065', '18066', '18358', '18365', '18374', '18404', '18486', '18608', '18613', '19012', '19015', '19018', '19022', '19021', '19044', '19057', '19069', '19071', '19074', '19084', '19100', '19107', '19122', '19128', '19135', '19140', '19150', '19151', '19242', '19243', '19252', '19307', '19382', '19459', '19798', '19822', '19907', '19923', '20540', '20763', '20808', '20848', '20920', '21087', '21168', '21189', '21269', '21270', '21294', '21319', '21371', '21375', '21378', '21418', '21605', '21818', '21855', '21856', '21896', '21902', '21985', '21986', '21995', '22003', '22049', '22050', '22051', '22131', '22134', '22139', '22145', '22148', '22149', '22164', '22165', '22220', '22222', '22240', '22242', '22328', '22415', '22478', '22489', '22490', '22712', '22713', '22838', '22880', '23091', '23122', '23146', '23147', '23148', '23151', '23170', '23139', '23188', '23234', '23236', '23304', '23358', '23359', '23360', '23361', '23376', '23377', '23378', '23379', '23381', '23390', '23393', '23398', '23401', '23402', '23404', '23406', '23409', '23462', '23443', '23443', '23535', '23598', '23793', '23811', '23972', '24095', '24175', '24205', '30522', '30525', '30744', '31555', '31560']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'resource usage'
    url = 'https://bugs.python.org/issue1635741'
    versions = ['Python 3.11']

    @kylotan
    Copy link
    Mannequin Author

    kylotan mannequin commented Jan 15, 2007

    This C code:

    #include <Python.h>
    int main(int argc, char *argv[])
    {
        Py_Initialize(); Py_Finalize();
        Py_Initialize(); Py_Finalize();
        Py_Initialize(); Py_Finalize();
        Py_Initialize(); Py_Finalize();
        Py_Initialize(); Py_Finalize();
        Py_Initialize(); Py_Finalize();
        Py_Initialize(); Py_Finalize();
    }

    Produces this output:
    [7438 refs]
    [7499 refs]
    [7550 refs]
    [7601 refs]
    [7652 refs]
    [7703 refs]
    [7754 refs]

    A similar program configured to call the Py_Initialize()/Py_Finalize() 1000 times ends up with:
    ...
    [58295 refs]
    [58346 refs]
    [58397 refs]

    This is with a fresh debug build of Python 2.5.0 on Windows XP, using Visual C++ 2003.

    @kylotan kylotan mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jan 15, 2007
    @devdanzin devdanzin mannequin added performance Performance or resource usage labels Mar 30, 2009
    @BreamoreBoy
    Copy link
    Mannequin

    BreamoreBoy mannequin commented Jul 20, 2010

    Does the title of this issue accurately reflect the current status of the Python interpreter?

    @amauryfa
    Copy link
    Member

    Yes, some objects are not cleaned in finalization.
    This is not a problem in usual cases though, when the interpreter is
    started only once.

    @ysjray
    Copy link
    Mannequin

    ysjray mannequin commented Mar 13, 2011

    Does the title of this issue accurately reflect the current status of the Python interpreter?

    Yes, here is the running result on current 3.3 latest code:
    [37182 refs]
    [39415 refs]
    [41607 refs]
    [43799 refs]
    [45991 refs]
    [48183 refs]
    [50375 refs]

    This seems to be a known bug that Py_Finalize() doesn't free all objects according doc http://docs.python.org/dev/c-api/init.html?highlight=py_finalize#Py_Finalize

    @AlexBudovski
    Copy link
    Mannequin

    AlexBudovski mannequin commented Aug 18, 2015

    Interestingly enough, some of the leaked memory came from the finalize routine itself! Here's one example:

    0:004> !heap -p -a 0x000000DB144346F0
    address 000000db144346f0 found in
    _HEAP @ db0cae0000
    HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
    000000db14434690 030a 0000 [00] 000000db144346c0 03074 - (busy)
    7ffc55628b04 ntdll!RtlpCallInterceptRoutine+0x0000000000000040
    7ffc555f9f36 ntdll!RtlAllocateHeap+0x0000000000079836
    7ffc2a60c4da ucrtbased!calloc_base+0x000000000000123a
    7ffc2a60c27d ucrtbased!calloc_base+0x0000000000000fdd
    7ffc2a60f34f ucrtbased!malloc_dbg+0x000000000000002f
    7ffc2a60fdde ucrtbased!malloc+0x000000000000001e
    5a5e6ef9 python36_d!_PyMem_RawMalloc+0x0000000000000029
    5a5e78c7 python36_d!_PyMem_DebugAlloc+0x0000000000000087
    5a5e5e6f python36_d!_PyMem_DebugMalloc+0x000000000000001f
    5a5e7230 python36_d!PyMem_Malloc+0x0000000000000030
    5a582047 python36_d!new_keys_object+0x0000000000000077
    5a57f7c5 python36_d!dictresize+0x0000000000000085
    5a57a4b2 python36_d!PyDict_Merge+0x0000000000000112
    5a57bf33 python36_d!PyDict_Update+0x0000000000000023
    5a75fb1d python36_d!PyImport_Cleanup+0x000000000000045d
    5a778f9e python36_d!Py_Finalize+0x000000000000005e

    @vstinner
    Copy link
    Member

    I tested on the master branch of Python:
    ---

    #include <Python.h>
    
    void func()
    {
        Py_Initialize(); Py_Finalize();
        Py_ssize_t cnt = _Py_GetRefTotal();
        printf("sys.gettotalrefcount(): %zd\n", cnt);
    }
    
    int main(int argc, char *argv[])
    {
        Py_SetProgramName(L"./_testembed");
        for (int i=0; i < 10; i++) {
            func();
        }
    }

    Each iteration leaks around 5,000 Python objects:
    ---
    sys.gettotalrefcount(): 15113
    sys.gettotalrefcount(): 19527
    sys.gettotalrefcount(): 23941
    sys.gettotalrefcount(): 28355
    sys.gettotalrefcount(): 32769
    sys.gettotalrefcount(): 37183
    sys.gettotalrefcount(): 41597
    sys.gettotalrefcount(): 46011
    sys.gettotalrefcount(): 50425
    sys.gettotalrefcount(): 54839
    ---

    @vstinner vstinner added the 3.9 only security fixes label Oct 23, 2019
    @vstinner vstinner changed the title Interpreter seems to leak references after finalization Py_Finalize() doesn't clear all Python objects at exit Oct 23, 2019
    @vstinner vstinner added the 3.9 only security fixes label Oct 23, 2019
    @vstinner vstinner changed the title Interpreter seems to leak references after finalization Py_Finalize() doesn't clear all Python objects at exit Oct 23, 2019
    @vstinner
    Copy link
    Member

    I marked bpo-6741 as a duplicate of this issue.

    @vstinner
    Copy link
    Member

    I marked bpo-26888 as a duplicate of this issue.

    @vstinner
    Copy link
    Member

    I marked bpo-21387 as a duplicate of this issue.

    @vstinner
    Copy link
    Member

    One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP-489 "Multi-phase extension module initialization".

    @vstinner
    Copy link
    Member

    I marked bpo-32026 as a duplicate of this issue.

    @shihai1991
    Copy link
    Member

    One part of this issue is that all C extensions of the stdlib should be updated to implement the PEP-489 "Multi-phase extension module initialization".

    I try to port _json extension module to multiphase initialization module, but the baseline(using victor's code) in my vm not changed~

    @shihai1991
    Copy link
    Member

    Compare to _Py_ForgetReference(), _Py_INC_REFTOTAL in _Py_NewReference() looks redundant.

    REF: https://github.com/python/cpython/blob/master/Include/object.h#L442

    master brach baseline in my vm:

    sys.gettotalrefcount(): 18049
    sys.gettotalrefcount(): 22463
    

    after PR17883

    sys.gettotalrefcount(): 17589
    sys.gettotalrefcount(): 22000
    

    @shihai1991
    Copy link
    Member

    FWIW, i counted the difference of each file's refs after Py_Finalize().

    [('Objects/dictobject.c', 21434), ('Python/marshal.c', 8135), ('Objects/codeobject.c', 6245), ('Objects/listobject.c', 6037), ('Objects/tupleobject.c', 4169), ('Objects/boolobject.c', 2433), ('Objects/object.c', 2364), ('Objects/unicodeobject.c', 1541), ('Objects/longobject.c', 1387), ('Objects/funcobject.c', 528), ('Objects/classobject.c', 528), ('Objects/abstract.c', 463), ('Python/structmember.c', 369), ('./Include/objimpl.h', 277), ('Objects/stringlib/partition.h', 273), ('Python/import.c', 259), ('Python/codecs.c', 197), ('./Modules/signalmodule.c', 61), ('./Modules/_threadmodule.c', 59), ('Objects/exceptions.c', 15), ('Objects/bytesobject.c', 5), ('./Modules/_weakref.c', 4), ('Python/_warnings.c', 3), ('./Modules/timemodule.c', 1), ('./Modules/_codecsmodule.c', 1), ('Objects/bytearrayobject.c', 1), ('Python/compile.c', 1), ('Objects/sliceobject.c', 0), ('Objects/memoryobject.c', 0), ('Python/context.c', -1), ('Objects/clinic/longobject.c.h', -1), ('Objects/enumobject.c', -1), ('Modules/gcmodule.c', -1), ('Objects/namespaceobject.c', -1), ('Objects/stringlib/unicode_format.h', -2), ('Objects/rangeobject.c', -3), ('Python/pystate.c', -4), ('Objects/fileobject.c', -14), ('./Modules/_io/clinic/bufferedio.c.h', -17), ('./Modules/_io/iobase.c', -21), ('Python/modsupport.c', -28), ('./Modules/_io/fileio.c', -28), ('Python/pylifecycle.c', -37), ('./Modules/_io/textio.c', -39), ('Objects/genobject.c', -53), ('Objects/weakrefobject.c', -54), ('./Modules/_io/bufferedio.c', -56), ('./Python/sysmodule.c', -68), ('./Modules/_io/_iomodule.c', -82), ('Python/errors.c', -90), ('Objects/descrobject.c', -110), ('Objects/structseq.c', -113), ('Python/bltinmodule.c', -118), ('Objects/setobject.c', -339), ('Objects/moduleobject.c', -454), ('./Modules/posixmodule.c', -614), ('./Modules/_abc.c', -664), ('Objects/call.c', -755), ('Objects/typeobject.c', -2035), ('Objects/frameobject.c', -6538), ('Python/ceval.c', -7857), ('./Include/object.h', -48292)]

    @vstinner
    Copy link
    Member

    New changeset ed154c3 by Victor Stinner (Hai Shi) in branch 'master':
    bpo-1635741: Port _json extension module to multiphase initialization (PEP-489) (GH-17835)
    ed154c3

    @shihai1991
    Copy link
    Member

    i thinkt that not checking PyModule_AddObject()'s result may cause this probleam too.

    1. python-ast.c have one question, i fix it in PR18358.
    2. most of the questions in extension module, for example: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L2019-L2022

    @shihai1991
    Copy link
    Member

    update the above info:

    1. python-ast.c have one question, i fix it in PR18365.

    @shihai1991
    Copy link
    Member

    1. python-ast.c have one question, i fix it in PR18365.
    2. most of the questions in extension module, for example: https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L2019-L2022

    brandt does relevant work already in PR17276PR38823.

    @miss-islington
    Copy link
    Contributor

    New changeset 1ea45ae by Hai Shi in branch 'master':
    bpo-1635741: Port _codecs extension module to multiphase initialization (PEP-489) (GH-18065)
    1ea45ae

    @shihai1991
    Copy link
    Member

    Leave a note for myself:
    I check the remaining object roughly(though dump_refs function), most of remaining object is 'str', such as:
    '0x7f779cf88880 [13] str'->'0x7f779cf88880 [26] str'

    So far, I don't know which file and fileno create those object. MAYBE I need find a hack way to sign this mallocing operation?(not sure)

    @vstinner
    Copy link
    Member

    New changeset b2b6e27 by Hai Shi in branch 'master':
    bpo-1635741: Port _crypt extension module to multiphase initialization (PEP-489) (GH-18404)
    b2b6e27

    @vstinner
    Copy link
    Member

    New changeset 7d79568 by Hai Shi in branch 'master':
    bpo-1635741: Port _contextvars module to multiphase initialization (PEP-489) (GH-18374)
    7d79568

    @erlend-aasland erlend-aasland added the 3.10 only security fixes label Nov 4, 2021
    @erlend-aasland erlend-aasland added interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-subinterpreters 3.10 only security fixes and removed docs Documentation in the Doc dir labels Nov 4, 2021
    @vstinner
    Copy link
    Member

    I wrote PR 20763 to "finalize" static types in Py_Finalize(). It mostly works, but "./Programs/_testembed test_forced_io_encoding" crash. (...)

    I created bpo-46417 follow-up issue: "[subinterpreters] Clear static types in Py_Finalize()".

    @vstinner
    Copy link
    Member

    Using "./python -IsS" command, sys.modules now only has 3 extensions which are not created by PyModuleDef_Init():

    • builtins
    • _io
    • sys

    The builtins and sys extensions use many static types. Converting these static types to heap types is blocked by bpo-40601. Soon, bpo-46417 will clear these types. Py_Finalize() does its best to clear explicitly builtins and sys namespaces.

    @vstinner
    Copy link
    Member

    In bpo-46417, I pushed a change to call _PyDebug_PrintTotalRefs() after destroying the interpreter. I backported locally this change to other Python versions to compare the progress on old Python versions:

    ./python -I -X showrefcount -c pass command:

    • Python 3.7: [22636 refs, 7453 blocks]
    • Python 3.8: [18082 refs, 5867 blocks]
    • Python 3.9: [17786 refs, 5766 blocks]
    • Python 3.10: [10725 refs, 3978 blocks]
    • current main branch: [2957 refs, 1240 blocks]

    Yeah, main is way better than Python 3.10! I pushed many changes in
    bpo-46417 to clear most static types, it helps a lot!

    @vstinner
    Copy link
    Member

    My work on bpo-46417 reduces the number of leak references from around 10k to ... minus 4 references :-) Clearing static types and a few "static" objects helped a lot!

    New changeset a1444f4 by Victor Stinner in branch 'main':
    bpo-46417: Fix _PyStaticType_Dealloc() (GH-30810)
    a1444f4

    At commit a1444f4, I get:

        $ ./python -I -X showrefcount -c pass
        [-4 refs, 61 blocks]

    I have to investigate why it's negative. It may be caused by bpo-46449 issue.

    See also: https://bugs.python.org/issue46417#msg411307

    @vstinner
    Copy link
    Member

    Thanks to recent enhancements, epecially in bpo-46417, the last memory blocks are now released at Python exit! The initial issue has been fixed!!! This bug was 15 years old! Current main branch:

    $ ./python -I -X showrefcount -c pass
    [-5 refs, 0 blocks]

    "0 blocks" means that Python no longer leaks memory at exit. You can use Valgrind to check it ;-)

    The negative reference count is being discussed in bpo-46449.

    I close this issue since it has an insane history: too many pull requests, changes, etc.

    While the work is not 100% done (we need to convert remaining static types to heap types, and convert extensions to multi-phase init), I prefer to address remaining issues in (existing or new) separated issues. See for example the bpo-40077: "Convert static types to heap types: use PyType_FromSpec()".

    I would like to thank everybody who helped to fix this issue! It's hard to list all names since this issue is a meta issues made of many sub-issues which also have sub-issues. Authors of PEP-573 and PEP-630 also helped fixing this issue!

    @vstinner vstinner added 3.11 bug and security fixes and removed 3.10 only security fixes labels Jan 27, 2022
    @vstinner vstinner added 3.11 bug and security fixes and removed 3.10 only security fixes labels Jan 27, 2022
    @vstinner
    Copy link
    Member

    I marked bpo-35774 as a duplicate of this issue.

    @vstinner
    Copy link
    Member

    The negative refcount issue has been fixed ;-)

    At commit 9a24127, Python now leaks exactly 0 reference count and 0 memory block at exit:

    $ ./python -I -X showrefcount -c pass
    [0 refs, 0 blocks]

    @vstinner
    Copy link
    Member

    New changeset c9c178f by Victor Stinner in branch 'main':
    bpo-1635741: test_embed cheks that Python does not leak (GH-31555)
    c9c178f

    @vstinner
    Copy link
    Member

    New changeset 4657bf7 by Victor Stinner in branch 'main':
    bpo-1635741: Fix winreg reference leaks (GH-31560)
    4657bf7

    @AlexSoft73
    Copy link

    I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call
    Py_IsInitialized();
    Py_FinalizeEx();

    @vstinner
    Copy link
    Member

    This issue is closed, please open a new issue.

    @vstinner
    Copy link
    Member

    I have teste Python 3.11 "3.11.0rc1 (main, Aug 8 2022, 11:30:54) [MSC v.1932 64 bit (AMD64)]" in Win10 64 bits and it leaks around 2Mb each time you call

    Ah, it seems like it has been reported as a new issue: #96853

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 bug and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage topic-subinterpreters
    Projects
    None yet
    Development

    No branches or pull requests