classification
Title: Modify _PyObject_GC_TRACK() to ensure that newly tracked object is valid
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords: patch

Created on 2020-04-01 21:55 by vstinner, last changed 2021-06-29 02:25 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
track.patch vstinner, 2020-04-01 21:55
gc_track.patch vstinner, 2021-06-29 02:25
Messages (5)
msg365515 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 21:55
In bpo-38392, I modified PyObject_GC_Track() to ensure that the object newly tracked is valid: call its traverse function.
=> commit 1b1845569539db5c1a6948a5d32daea381f1e35f

I propose to now also attempt to implement the same check in _PyObject_GC_TRACK() which is part of the internal C API.

PyType_GenericAlloc() allocates memory for a type allocated on the heap... and then immediately track it in the GC. Problem: this type is not initialized yet, all fields are set to zero. Calling type_traverse() at this point fails with an assertion error:
---
Objects/typeobject.c:3570: type_traverse: Assertion failed: type_traverse() called on non-heap type '(null)'
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x840860
object refcount : 1
object type     : 0x7e0900
object type name: type
object repr     : 
---

By the way, Python crash in _PyObject_Dump() on PyObject_Repr() call: type_repr() crash when accessing type->tp_name which is NULL.

type_call() should only track the newly created type when it's fully initialized.


Try attached track.patch to reproduce the crash.
msg368873 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-14 23:00
While it might be doable, I don't have the bandwidth to investigate this issue and so I prefer to close it as out of date.
msg396693 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 01:43
PyType_GenericAlloc() cannot traverse the object since the object members are not initialized yet. For example, dict_traverse() can only be called when PyDict_New() completes.

A different approach would be to:

* (1) Add PyType_AllocNoTrack(), use it in built-in types, and call _PyObject_GC_TRACK() on the instance once it is fully initialized.
* (2) Modify PyType_GenericAlloc() to use a new variant of _PyObject_GC_TRACK() which will not traverse the instance.
* (3) Modify _PyObject_GC_TRACK() to traverse the instance.

In short, PyType_GenericAlloc() cannot be modified for backward compatibility.

Moreover, _PyObject_GC_TRACK() should only be used inside Python internals, since it's part the internal C API.
msg396697 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 02:22
I created bpo-44531 "Add _PyType_AllocNoTrack() function: allocate without tracking in the GC".
msg396698 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-29 02:25
Attached gc_track.patch: my latest attempt to implement this idea.
History
Date User Action Args
2021-06-29 02:25:22vstinnersetfiles: + gc_track.patch

messages: + msg396698
2021-06-29 02:22:16vstinnersetmessages: + msg396697
2021-06-29 01:43:49vstinnersetmessages: + msg396693
2020-05-14 23:00:43vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg368873

stage: resolved
2020-04-01 21:55:41vstinnercreate