classification
Title: Add _PyType_AllocNoTrack() function: allocate without tracking in the GC
Type: Stage: patch review
Components: C API Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2021-06-29 02:02 by vstinner, last changed 2021-07-01 00:30 by vstinner.

Pull Requests
URL Status Linked Edit
PR 26947 merged vstinner, 2021-06-29 02:17
PR 26948 merged vstinner, 2021-06-29 02:31
Messages (8)
msg396695 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 02:02
The PyType_GenericAlloc() function tracks the newly created object in the garbage collector (GC) as soon as memory is initialized, but before all object members are initialized.

If a GC collection happens before the object is fully initialized, the traverse function of the newly created object can crash.

This case is hypothetical for built-in types since their constructor should not trigger a GC collection. It is more likely in third party extensions and subclasses.

Anyway, I propose to add a new _PyType_AllocNoTrack() function which allocates memory without tracking the newly allocated object directly in the GC.

This function can be used to only track explicitly the object in the GC once it is fully initialized.
msg396696 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 02:22
In bpo-40142, I tried to modify _PyObject_GC_TRACK() to visit the object before tracking it, as done by PyObject_GC_Track().

Problem: PyType_GenericAlloc() cannot traverse the object since the object members are not initialized yet. For example, dict_traverse() can only be called at dict_new() exit.

Here I propose a different approach. First, fix built-in types to only track instances when they are fully initialized. Avoid PyType_GenericAlloc().
msg396732 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-06-29 14:20
> Anyway, I propose to add a new _PyType_AllocNoTrack() function which allocates memory without tracking the newly allocated object directly in the GC.

How is that going to help third party extensions and subclasses?
msg396735 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 14:39
New changeset 823460daa9fab3d0cf00ec553d1e35635ef73d40 by Victor Stinner in branch 'main':
bpo-44531: Fix type_repr() if tp_name is NULL (GH-26948)
https://github.com/python/cpython/commit/823460daa9fab3d0cf00ec553d1e35635ef73d40
msg396736 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 14:40
> How is that going to help third party extensions and subclasses?

It doesn't solve bpo-40142 issue for third party extensions.

About subclasses, I'm not sure if it's safe to call the traverse function if a subclass overrides it and visits its own member which are not in the built-in type (like dict or tuple).
msg396738 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-06-29 14:44
> About subclasses, I'm not sure if it's safe to call the traverse function if a subclass overrides it and visits its own member which are not in the built-in type (like dict or tuple).

Technically the traverse should only call visit on the members, but in practice they can do whatever. I agree is not clear is a safe operation
msg396741 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 14:49
For the background of this issue, please see bpo-38392:
https://bugs.python.org/issue38392#msg354074
msg396804 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-07-01 00:30
New changeset 818628c2da99ba0376313971816d472c65c9a9fc by Victor Stinner in branch 'main':
bpo-44531: Add _PyType_AllocNoTrack() function (GH-26947)
https://github.com/python/cpython/commit/818628c2da99ba0376313971816d472c65c9a9fc
History
Date User Action Args
2021-07-01 00:30:52vstinnersetmessages: + msg396804
2021-06-29 14:49:17vstinnersetmessages: + msg396741
2021-06-29 14:44:03pablogsalsetmessages: + msg396738
2021-06-29 14:40:31vstinnersetmessages: + msg396736
2021-06-29 14:39:38vstinnersetmessages: + msg396735
2021-06-29 14:20:50pablogsalsetnosy: + pablogsal
messages: + msg396732
2021-06-29 02:31:24vstinnersetpull_requests: + pull_request25515
2021-06-29 02:22:05vstinnersetmessages: + msg396696
2021-06-29 02:17:49vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request25514
2021-06-29 02:02:58vstinnercreate