[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol #87138

vstinner · 2021-01-19T21:54:26Z

BPO	42972
Nosy	@nascheme, @ncoghlan, @vstinner, @tiran, @corona10, @pablogsal, @miss-islington, @shihai1991, @erlend-aasland, @Fidget-Spinner
PRs	bpo-42972: Fully implement GC protocol for sqlite3 heap types #26104 bpo-42972: Fully implement GC protocol for arraymodule types #26114 [3.10] bpo-42972: Fully implement GC protocol for sqlite3 heap types (GH-26104) #26361 [3.10] bpo-42972: Fully implement GC protocol for arraymodule types (GH-26114) #26362 bpo-42972: Fully implement GC protocol for functools keywrapper and partial types #26363 bpo-42972: Fully implement GC protocol for re types #26368 bpo-42972: Fully implement GC protocol for ssl heap types (GH-26370) #26370 bpo-42972: Fully support GC protocol for operator heap types #26371 bpo-42972: Fully support GC protocol for _queue.SimpleQueue #26372 bpo-42972: Fully support GC for mmap heap types #26373 bpo-42972: Fully support GC for hashlib heap types (GH-26374) #26374 bpo-42972: Fully support GC for pyexpat, unicodedata, and dbm/gdbm heap types #26376 bpo-42972: Fully support GC for _winapi.Overlapped #26381 [3.10] bpo-42972: Fully support GC for pyexpat, unicodedata, and dbm/gdbm heap types (GH-26376) #26397 [3.10] bpo-42972: Fully support GC for hashlib heap types (GH-26374) #26398 [3.10] bpo-42972: Fully implement GC protocol for ssl heap types (GH-26370) #26399 [3.10] bpo-42972: Fully support GC protocol for _queue.SimpleQueue (GH-26372) #26406 [3.10] bpo-42972: Fully support GC for mmap heap types (GH-26373) #26407 [3.10] bpo-42972: Fully implement GC protocol for re types (GH-26368) #26411 [3.10] bpo-42972: Fully support GC protocol for _operator heap types (GH-26371) #26413 [3.10] bpo-42972: Fully implement GC protocol for re types (GH-26368) #26414 bpo-42972: Fully implement GC protocol for functools LRU cache #26423 [3.10] bpo-42972: Fully implement GC protocol for functools keywrapper and partial types (GH-26363) #26424 [3.10] bpo-42972: Fully implement GC protocol for functools LRU cache (GH-26423) #26425 [3.10] bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381) #26426 [3.10] bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381) #26427 bpo-42972: Fix GC assertion error in _winapi by untracking Overlapped earlier #26429 [3.10] bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381) #26430 [3.10] bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381) #26431 bpo-42972: Fully implement GC protocol for xxlimited.Xxo #26451 bpo-42972: Fix sqlite3 traverse/clear #26452 [3.10] bpo-42972: Fully implement GC protocol for xxlimited (GH-26451) #26460 [3.10] bpo-42972: Fix sqlite3 traverse/clear functions (GH-26452) #26461 bpo-42972: Track sqlite3 statement objects #26475 [3.10] bpo-42972: Track sqlite3 statement objects (GH-26475) #26515 bpo-42972: _thread.RLock type implements tp_traverse #26734 [3.10] bpo-42972: _thread.RLock implements the GH protocol (GH-26734) #26735

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-01-19.21:54:26.099>
labels = ['expert-C-API', 'type-bug', '3.10']
title = '[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol'
updated_at = <Date 2021-09-29.00:01:08.612>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-09-29.00:01:08.612>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-01-19.21:54:26.099>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42972
keywords = ['patch']
message_count = 56.0
messages = ['385297', '385299', '385883', '393547', '393609', '393668', '394383', '394385', '394386', '394388', '394402', '394403', '394404', '394409', '394416', '394427', '394446', '394450', '394516', '394517', '394518', '394519', '394520', '394532', '394541', '394547', '394554', '394555', '394562', '394564', '394566', '394568', '394574', '394575', '394583', '394594', '394601', '394609', '394622', '394643', '394645', '394646', '394647', '394658', '394659', '394662', '394669', '394676', '394789', '394795', '394797', '394801', '394852', '395015', '395878', '395879']
nosy_count = 11.0
nosy_names = ['nascheme', 'ncoghlan', 'vstinner', 'christian.heimes', 'corona10', 'pablogsal', 'miss-islington', 'shihai1991', 'erlendaasland', 'kj', 'soffieswan015']
pr_nums = ['26104', '26114', '26361', '26362', '26363', '26368', '26370', '26371', '26372', '26373', '26374', '26376', '26381', '26397', '26398', '26399', '26406', '26407', '26411', '26413', '26414', '26423', '26424', '26425', '26426', '26427', '26429', '26430', '26431', '26451', '26452', '26460', '26461', '26475', '26515', '26734', '26735']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42972'
versions = ['Python 3.10']

vstinner · 2021-01-19T21:54:25Z

Copy of my email sent to python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/C4ILXGPKBJQYUN5YDMTJOEOX7RHOD4S3/

Hi,

In the Python stdlib, many heap types currently don't "properly"
(fully?) implement the GC protocol which can prevent to destroy these
types at Python exit. As a side effect, some other Python objects can
also remain alive, and so are not destroyed neither.

There is an on-going effect to destroy all Python objects at exit
(bpo-1635741). This problem is getting worse when subinterpreters are
involved: Refleaks buildbots failures which prevent to spot other
regressions, and so these "leaks" / "GC bugs" must be fixed as soon as
possible. In my experience, many leaks spotted by tests using
subinterpreters were quite old, it's just that they were ignored
previously.

It's an hard problem and I don't see any simple/obvious solution right
now, except of workarounds that I dislike. Maybe the only good
solution is to fix all heap types, one by one.

== Only the Python stdlib should be affected ==

PyType_FromSpec() was added to Python 3.2 by the PEP-384 to define
"heap types" in C, but I'm not sure if it's popular in practice (ex:
Cython doesn't use it, but defines static types). I expect that most
types to still be defined the old style (static types) in a vas
majority of third party extension modules.

To be clear, static types are not affected by this email.

Third party extension modules using the limited C API (to use the
stable ABI) and PyType_FromSpec() can be affected (if they don't fully
implement the GC protocol).

== Heap type instances now stores a strong reference to their type ==

In March 2019, the PyObject_Init() function was modified in bpo-35810
to keep a strong reference (INCREF) to the type if the type is a heap
type. The fixed problem was that heap types could be destroyed before
the last instance is destroyed.

== GC and heap types ==

The new problem is that most heap types don't collaborate well with
the garbage collector. The garbage collector doesn't know anything
about Python objects, types, reference counting or anything. It only
uses the PyGC_Head header and the traverse functions. If an object
holds a strong reference to an object but its type does not define a
traverse function, the GC cannot guess/infer this reference.

A heap type must respect the following 3 conditions to collaborate with the GC:

Have the Py_TPFLAGS_HAVE_GC flag;
Define a traverse function (tp_traverse) which visits the type: Py_VISIT(Py_TYPE(self));
Instances must be tracked by the GC.

If one of these conditions is not met, the GC can fail to destroy a
type during a GC collection. If an instance is kept alive late while a
Python interpreter is being deleted, it's possible that the type is
never deleted, which can keep indirectly many objects alive and so
don't delete them neither.

In practice, when a type is not deleted, a test using subinterpreter
starts to fail on Refleaks buildbot since it leaks references. Without
subinterpreters, such leak is simply ignored, whereas this is an
on-going effect to delete Python objects at exit (bpo-1635741).

== Boring traverse functions ==

Currently, there is no default traverse implementation which visits the type.

For example, I had the implement the following function for _thread.LockType:

static int
lock_traverse(lockobject self, visitproc visit, void arg)
{
    Py_VISIT(Py_TYPE(self));
    return 0;
}

It's a little bit annoying to have to implement the GC protocol
whereas a lock cannot contain other Python objects, it's not a
container. It's just a thin wrapper to a C lock.

There is exactly one strong reference: to the type.

== Workaround: loop on gc.collect() ==

A workaround is to run gc.collect() in a loop until it returns 0 (no
object was collected).

== Traverse automatically? Nope. ==

Pablo Galindo attempts to automatically visit the type in the traverse function:

https://bugs.python.org/issue40217
0169d30...

Moreover, What's New in Python 3.9 contains a long section suggesting
to implement a traverse function for this problem, but it doesn't
suggest to track instances:
https://docs.python.org/dev/whatsnew/3.9.html#changes-in-the-c-api

This solution causes too many troubles, and so instead, traverse
functions were defined on heap types to visit the type.

Currently in the master branch, 89 types are defined as heap types on
a total of 206 types (117 types are defined statically). I don't think
that these 89 heap types respect the 3 conditions to collaborate with
the GC.

== How should we address this issue? ==

I'm not sure what should be done. Working around the issue by
triggering multiple GC collections? Emit a warning in development mode
if a heap type doesn't collaborate well with the GC?

If core developers miss these bugs and have troubles to debug them, I
expect that extension module authors would suffer even more.

== GC+heap type bugs became common ==

I'm fixing such GC issue for 1 year as part as the work on cleaning
Python objects at exit, and also indirectly related to
subinterpreters. The behavior is surprising, it's really hard to dig
into GC internals and understand what's going on. I wrote an article
on this kind of "GC bugs":
https://vstinner.github.io/subinterpreter-leaks.html

Today, I learnt the hard way that defining a traverse is not enough.
The type constructor (tp_new) must also track instances! See my fix
for _multibytecodec related to CJK codecs:

11ef53a...
https://bugs.python.org/issue42866

== Reference cycles are common ==

The GC only serves to break reference cycles. But reference cycles are
rare, right? Well...

First of all, most types create reference cycles involing themselves.
For example, a type __mro__ tuple contains the type which already
creates a ref cycle. Type methods can also contain a reference to the
type.

=> The GC must break the cycle, otherwise the type cannot be destroyed

When a function is defined in a Python module, the function
__globals__ is the module namespace (module.__dict__) which...
contains the function. Defining a function in a Python module also
creates a reference cycle which prevents to delete the module
namespace.

If a function is used as a callback somewhere, the whole module
remains "alive" until the reference to the callback is cleared.
Example. os.register_at_fork() and codecs.register() callbacks are
cleared really late during Python finalization. Currently, it's
basically the last objects which are cleared at Python exit. After
that, there is exactly one final GC collection.

=> The GC

== Debug GC issues ==

gc.get_referents() and gc.get_referrers() can be used to check traverse functions.
gc.is_tracked() can be used to check if the GC tracks an object.
Using the gdb debugger on gc_collect_main() helps to see which objects are collected. See for example the finalize_garbage() functions which calls finalizers on unreachable objects.
The solution is usually a missing traverse functions or a missing Py_VISIT() in an existing traverse function.

== __del__ hack for debugging ==

If you want to play with the issue or if you have to debug a GC issue,
you can use an object which logs a message when it's being deleted:

class VerboseDel:
    def __del__(self):
        print("DELETE OBJECT")
obj = VerboseDel()

Warning: creating such object in a module also prevents to destroy the
module namespace when the last reference to the module is deleted!
__del__.__globals__ contains a reference to the module namespace, and
obj.__class__ contains a reference to the type... Yeah, ref cycle and
GC issues are fun!

== Long email ==

Yeah, I like to put titles in my long emails. Enjoy. Happy hacking!
Victor

--
Night gathers, and now my watch begins. It shall not end until my death

vstinner · 2021-01-19T21:56:39Z

In June 2020, I create PR 20983 to attempt to automatically traverse the type:
"Provide a default tp_traverse implementation for the base object
type for heap types which have no tp_traverse function. The
traverse function visits the type if the type is a heap type."

I abandoned my PR.

I marked bpo-41036 as a duplicate of this issue.

erlend-aasland · 2021-01-28T20:45:55Z

Should we proceed with fixing GC for all heap types before continuing work with bpo-40077?

pablogsal · 2021-05-12T19:51:44Z

I'm marking this as a 3.10 release blocker untill all converted types that are in 3.10 have GC support.

erlend-aasland · 2021-05-13T21:34:43Z

I've added a checkbox for types that fully implement the GC protocol to https://discuss.python.org/t/list-of-built-in-types-converted-to-heap-types/8403/1.

Heap types that fully implement the GC protocol:

_abc._abc_data
_bz2.BZ2Compressor
_bz2.BZ2Decompressor
_csv.Dialect
_csv.reader
_csv.writer
_json.Encoder
_json.Scanner
_lzma.LZMACompressor
_lzma.LZMADecompressor
_multibytecodec.MultibyteCodec
_struct.unpack_iterator
_thread._local
_thread.lock
ast.AST

Heap types that do not fully implement the GC protocol:

_curses_panel.panel
_dbm.dbm
_gdbm.gdbm
_hashlib.HASH
_hashlib.HASHXOF
_lsprof.Profiler
_md5.md5
_multibytecodec.MultibyteIncrementalDecoder
_multibytecodec.MultibyteIncrementalEncoder
_multibytecodec.MultibyteStreamReader
_multibytecodec.MultibyteStreamWriter
_overlapped.Overlapped
_queue.SimpleQueue
_random.Random
_sha1.sha1
_sha256.sha224
_sha256.sha256
_sha512.sha384
_sha512.sha512
_sre.SRE_Scanner
_ssl.MemoryBIO
_ssl.SSLSession
_ssl._SSLContext
_ssl._SSLSocket
_struct.Struct
_thread.RLock
_thread._localdummy
_tkinter.Tcl_Obj
_tkinter.tkapp
_tkinter.tktimertoken
array.array
array.arrayiterator
functools.KeyWrapper
functools._lru_cache_wrapper
functools._lru_list_elem
functools.partial
mmap.mmap
operator.attrgetter
operator.itemgetter
operator.methodcaller
posix.DirEntry
posix.ScandirIterator
pyexpat.xmlparser
re.Match
re.Pattern
select.devpoll
select.epoll
select.kevent
select.kqueue
select.poll
sqlite3.Cache
sqlite3.Connection
sqlite3.Cursor
sqlite3.Node
sqlite3.PrepareProtocol
sqlite3.Row
sqlite3.Statement
ssl.SSLError
unicodedata.UCD
winapi__overlapped.Overlapped
zlib.Compress
zlib.Decompress

erlend-aasland · 2021-05-14T15:35:08Z

Is there a deterministic way to test these changes? Will something a la this be sufficient:

import gc
import sys

gc.collect()
before = sys.gettotalrefcount()

import somemod
del sys.modules['somemod']
del somemod

gc.collect()
after = sys.gettotalrefcount()

assert after == before

pablogsal · 2021-05-25T17:44:06Z

New changeset d3c277a by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully implement GC protocol for sqlite3 heap types (GH-26104)
d3c277a

miss-islington · 2021-05-25T18:08:47Z

New changeset e8d9df0 by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully implement GC protocol for sqlite3 heap types (GH-26104)
e8d9df0

pablogsal · 2021-05-25T18:26:56Z

New changeset bd404cc by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully implement GC protocol for arraymodule types (GH-26114)
bd404cc

miss-islington · 2021-05-25T18:49:27Z

New changeset 534da74 by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully implement GC protocol for arraymodule types (GH-26114)
534da74

erlend-aasland · 2021-05-25T21:22:29Z

Christian, I've got a PR ready for Modules/_ssl.c, but I won't submit it if you'd rather do it yourself. I'll stay off the sha/md5 types unless you approve :)

tiran · 2021-05-25T21:26:27Z

Please open PRs and assign them to me. I'll review them as soon as possible.

erlend-aasland · 2021-05-25T22:06:21Z

Thanks! Hashlib PR comin' up.

pablogsal · 2021-05-26T00:25:39Z

Victor, can you take a look at the opened PRs?

shihai1991 · 2021-05-26T06:11:59Z

functools._lru_list_elem
Looks like this type have performance in issue PR-5008 when supporting GC. I am not sure there have other similar questions or not.

erlend-aasland · 2021-05-26T09:54:21Z

I've opened PR's to fix most of the heap types converted during the 3.10 alpha phase. What's missing (of the 3.10 batch) is:

_thread types (this needs special care)
winapi__overlapped.Overlapped (I currently don't have a Win dev env at hand)

For the types converted during 3.9 dev, should we backport to 3.9 or just 3.10?

Fidget-Spinner · 2021-05-26T16:36:28Z

_winapi is leaky still even with my PR:

>>> import sys,gc
>>> for _ in range(5):
...  print(sys.gettotalrefcount())
...  import _winapi
...  del sys.modules['_winapi']
...  del _winapi
...  gc.collect()
...
50468
51076
51432
51788
52144

I just noticed this, but _winapi doesn't have a m_traverse/m_clear/m_free in the PyModuleDef eventhough it creates nearly 100 objects in m_slot->Py_mod_exec. I'm not a multi phase init expert, but shouldn't there be a cleanup function or am I confusing something here :( ?

Fidget-Spinner · 2021-05-26T17:16:21Z

it creates nearly 100 objects

Oops sorry I think I'm wrong. most of those objects may be borrowed refs. Just 1 type object is causing the leak.

ncoghlan · 2021-05-27T07:29:15Z

New changeset 59af59c by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully support GC for pyexpat, unicodedata, and dbm/gdbm heap types (GH-26376)
59af59c

tiran · 2021-05-27T07:48:36Z

New changeset 6ef5ba3 by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully support GC for hashlib heap types (GH-26374)
6ef5ba3

tiran · 2021-05-27T07:50:13Z

New changeset dcb8786 by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully implement GC protocol for ssl heap types (GH-26370)
dcb8786

miss-islington · 2021-05-27T08:11:02Z

New changeset 4431922 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-42972: Fully support GC for hashlib heap types (GH-26374) (GH-26398)
4431922

tiran · 2021-05-27T08:20:03Z

#70587 is failing with an access violation on Windows. It's failing in one of the flaky tests. I wonder if the segfault is related to flaky tests somehow...

https://dev.azure.com/Python/cpython/_build/results?buildId=81570&view=logs&j=c83831cd-3752-5cc7-2f01-8276919eb334

test_pha_optional (test.test_ssl.TestPostHandshakeAuth) ... ok
test_pha_optional_nocert (test.test_ssl.TestPostHandshakeAuth) ... ok
test_pha_required (test.test_ssl.TestPostHandshakeAuth) ... ok
Windows fatal exception: access violation

Current thread 0x000009e0 (most recent call first):
File "D:\a\1\s\lib\linecache.py", line 63 in checkcache
File "D:\a\1\s\lib\traceback.py", line 375 in extract
File "D:\a\1\s\lib\traceback.py", line 494 in __init__
File "D:\a\1\s\lib\traceback.py", line 132 in format_exception
File "D:\a\1\s\lib\test\test_ssl.py", line 262 in handle_error
File "D:\a\1\s\lib\test\test_ssl.py", line 2530 in run
File "D:\a\1\s\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\1\s\lib\threading.py", line 963 in _bootstrap

Thread 0x000003c4 (most recent call first):
File "D:\a\1\s\lib\threading.py", line 1102 in _wait_for_tstate_lock
File "D:\a\1\s\lib\threading.py", line 1086 in join
File "D:\a\1\s\lib\test\test_ssl.py", line 2604 in run
File "D:\a\1\s\lib\threading.py", line 1006 in _bootstrap_inner
File "D:\a\1\s\lib\threading.py", line 963 in _bootstrap

Thread 0x00001700 (most recent call first):
File "D:\a\1\s\lib\ssl.py", line 1131 in read
File "D:\a\1\s\lib\ssl.py", line 1256 in recv
File "D:\a\1\s\lib\test\test_ssl.py", line 4471 in test_pha_required_nocert
File "D:\a\1\s\lib\unittest\case.py", line 549 in _callTestMethod
File "D:\a\1\s\lib\unittest\case.py", line 592 in run
File "D:\a\1\s\lib\unittest\case.py", line 652 in __call__
File "D:\a\1\s\lib\unittest\suite.py", line 122 in run
File "D:\a\1\s\lib\unittest\suite.py", line 84 in __call__
File "D:\a\1\s\lib\unittest\suite.py", line 122 in run
File "D:\a\1\s\lib\unittest\suite.py", line 84 in __call__
File "D:\a\1\s\lib\unittest\runner.py", line 176 in run
File "D:\a\1\s\lib\test\support\init.py", line 959 in _run_suite
File "D:\a\1\s\lib\test\support\init.py", line 1082 in run_unittest
File "D:\a\1\s\lib\test\test_ssl.py", line 5007 in test_main
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 246 in _runtest_inner2
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 282 in _runtest_inner
File "D:\a\1\s\lib\test\libregrtest\runtest.py", line 154 in _runtest
File "D:\a\1\s\lib\test\main.py", line 2 in <module>
File "D:\a\1\s\lib\runpy.py", line 86 in _run_code
File "D:\a\1\s\lib\runpy.py", line 196 in _run_module_as_main
##[error]Cmd.exe exited with code '-1073741819'.

erlend-aasland · 2021-05-27T10:19:02Z

Hm, I'm unable to reproduce it w/addr sanitiser on macOS (FWIW).

$ ./python.exe -m test test_ssl -F -u all -m test_pha_required_nocert

Passing 1000 successful runs now. I'll see if I can get a Win dev env set up later.

vstinner · 2021-05-27T20:59:15Z

New changeset f4b70c2 by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully support GC protocol for _operator heap types (GH-26371)
f4b70c2

vstinner · 2021-05-27T21:50:04Z

New changeset d1c7329 by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully support GC protocol for _operator heap types (GH-26371) (GH-26413)
d1c7329

vstinner · 2021-05-28T00:23:52Z

New changeset da9e0cb by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully implement GC protocol for re types (GH-26368) (GH-26414)
da9e0cb

vstinner · 2021-05-28T08:41:46Z

New changeset 8994e9c by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully implement GC protocol for functools keywrapper and partial types (GH-26363)
8994e9c

vstinner · 2021-05-28T09:02:51Z

New changeset 3f8d332 by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fully implement GC protocol for functools LRU cache (GH-26423)
3f8d332

vstinner · 2021-05-28T09:06:53Z

New changeset 0fa282c by Ken Jin in branch 'main':
bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381)
0fa282c

vstinner · 2021-05-28T09:08:05Z

New changeset eb8ab04 by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully implement GC protocol for functools keywrapper and partial types (GH-26363) (GH-26424)
eb8ab04

vstinner · 2021-05-28T13:58:47Z

I'm not fully satisfied by the implementation of the two LRU types in functools. See the discussion in these two PRs:

#26363
#26423

The _lru_list_elem doesnt implement the GC protocol for performance reasons:

But I'm not sure if it's ok that _lru_list_elem doesn't implement the GC protocol: it's disucssion in this issue.

vstinner · 2021-05-28T14:01:57Z

I'm not fully satified by the overlapped_dealloc() implementation neither. There is an unpleasant code path for Windows XP but it's no longer needed in Python 3.11. I would prefer to always call PyObject_GC_UnTrack() and call it earlier.

See the dicsussion in the PR:
#26381

But it can be modified later.

miss-islington · 2021-05-28T14:26:24Z

New changeset 1c0106c by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully implement GC protocol for functools LRU cache (GH-26423)
1c0106c

vstinner · 2021-05-28T16:29:26Z

New changeset 490b638 by Ken Jin in branch 'main':
bpo-42972: Fix GC assertion error in _winapi by untracking Overlapped earlier (GH(26429)
490b638

pablogsal · 2021-05-28T17:47:25Z

New changeset 0d39951 by Ken Jin in branch '3.10':
[3.10] bpo-42972: Fully support GC for _winapi.Overlapped (GH-26381) (bpo-26430)
0d39951

vstinner · 2021-05-31T07:51:51Z

New changeset 4b20f25 by Hai Shi in branch 'main':
bpo-42972: Fully implement GC protocol for xxlimited (GH-26451)
4b20f25

vstinner · 2021-05-31T08:25:04Z

New changeset d1124b0 by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Fix sqlite3 traverse/clear functions (GH-26452)
d1124b0

vstinner · 2021-05-31T09:12:42Z

New changeset ff359d7 by Miss Islington (bot) in branch '3.10':
bpo-42972: Fix sqlite3 traverse/clear functions (GH-26452) (GH-26461)
ff359d7

pablogsal · 2021-05-31T11:23:23Z

New changeset f097d23 by Miss Islington (bot) in branch '3.10':
bpo-42972: Fully implement GC protocol for xxlimited (GH-26451) (GH-26460)
f097d23

vstinner · 2021-06-01T10:47:41Z

New changeset fffa0f9 by Erlend Egeberg Aasland in branch 'main':
bpo-42972: Track sqlite3 statement objects (GH-26475)
fffa0f9

vstinner · 2021-06-03T16:38:19Z

New changeset 84d80f5 by Erlend Egeberg Aasland in branch '3.10':
[3.10] bpo-42972: Track sqlite3 statement objects (GH-26475) (GH-26515)
84d80f5

vstinner · 2021-06-15T13:09:32Z

New changeset 1cd3d85 by Victor Stinner in branch 'main':
bpo-42972: _thread.RLock implements the GH protocol (GH-26734)
1cd3d85

miss-islington · 2021-06-15T13:29:52Z

New changeset e30fe27 by Miss Islington (bot) in branch '3.10':
bpo-42972: _thread.RLock implements the GH protocol (GH-26734)
e30fe27

erlend-aasland · 2022-06-09T13:21:01Z

Can we close this, Victor?

vstinner · 2022-06-09T14:38:33Z

Yep, I close it. Thanks to everyone who helped to fix the issue!

vstinner added 3.10 only security fixes topic-C-API labels Jan 19, 2021

pablogsal added release-blocker labels May 12, 2021

soffieswan015 mannequin added type-bug An unexpected behavior, bug, or error labels Jun 1, 2021

pablogsal removed release-blocker labels Jun 4, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

vstinner closed this as completed Jun 9, 2022

sunmy2019 mentioned this issue May 16, 2023

GH-104510: Fix refleaks in _io base types #104516

Merged

vstinner mentioned this issue Mar 18, 2024

Heap types implemented in C must have Py_TPFLAGS_HAVE_GC and implement tp_traverse #116946

Open

[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol #87138

[C API] Heap types (PyType_FromSpec) must fully implement the GC protocol #87138

Comments

vstinner commented Jan 19, 2021

vstinner commented Jan 19, 2021

vstinner commented Jan 19, 2021

erlend-aasland commented Jan 28, 2021

pablogsal commented May 12, 2021

erlend-aasland commented May 13, 2021

erlend-aasland commented May 14, 2021

pablogsal commented May 25, 2021

miss-islington commented May 25, 2021

pablogsal commented May 25, 2021

miss-islington commented May 25, 2021

erlend-aasland commented May 25, 2021

tiran commented May 25, 2021

erlend-aasland commented May 25, 2021

pablogsal commented May 26, 2021

shihai1991 commented May 26, 2021

erlend-aasland commented May 26, 2021

Fidget-Spinner commented May 26, 2021

Fidget-Spinner commented May 26, 2021

ncoghlan commented May 27, 2021

tiran commented May 27, 2021

tiran commented May 27, 2021

miss-islington commented May 27, 2021

tiran commented May 27, 2021

erlend-aasland commented May 27, 2021

vstinner commented May 27, 2021

vstinner commented May 27, 2021

vstinner commented May 28, 2021

vstinner commented May 28, 2021

vstinner commented May 28, 2021

vstinner commented May 28, 2021

vstinner commented May 28, 2021

vstinner commented May 28, 2021

vstinner commented May 28, 2021

miss-islington commented May 28, 2021

vstinner commented May 28, 2021

pablogsal commented May 28, 2021

vstinner commented May 31, 2021

vstinner commented May 31, 2021

vstinner commented May 31, 2021

pablogsal commented May 31, 2021

vstinner commented Jun 1, 2021

vstinner commented Jun 3, 2021

vstinner commented Jun 15, 2021

miss-islington commented Jun 15, 2021

erlend-aasland commented Jun 9, 2022

vstinner commented Jun 9, 2022