Leaking the PyTypeObject structure in the C API indirectly causes ABI issue (especially for statically allocated types), cause practical issues when old fields are removed and new fields are added (ex: tp_vectorcall addition and tp_print removal caused a lot of troubles with C code generated by Cython: see bpo-37250), prevents us to add feature and experiment optimization.

I don't expect that we will be able to make PyTypeObject opaque soon. The purpose of this issue is to track the work done towards this goal.

I propose to slowly prepare the Python code base, the C API and third party code (especially Cython) to make PyTypeObject structure opaque.

We have to identify most common code patterns which access directly PyTypeObject fields, provide helper functions, and ease the migration to solutions which don't access directly PyTypeObject.

See also bpo-39573: "Make PyObject an opaque structure in the limited C API" and bpo-39947 "Make the PyThreadState structure opaque (move it to the internal C API)".

Longer rationale about making structures of the C API opaque:

* https://pythoncapi.readthedocs.io/
* https://pythoncapi.readthedocs.io/bad_api.html
* https://pythoncapi.readthedocs.io/optimization_ideas.html

--

Multiple practical issues are preventing us to make PyTypeObject opaque right now.


(*) Many C extension modules are still using statically allocated types: there is an on-going effort in bpo-40077 to convert C extension modules one by one to PyType_FromSpec().


(*) Py_TYPE(obj)->tp_name is commonly accessed to format an error message. Example:

        PyErr_Format(PyExc_TypeError, "exec() globals must be a dict, not %.100s",
                     Py_TYPE(globals)->tp_name);

I worked on bpo-34595 and started a discussion on python-dev to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format():
https://mail.python.org/archives/list/python-dev@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53

Sadly, we failed to reach a consensus and I gave up on this idea. We should reconsider this idea.

We need to agree on how types should be formatted:

* just the name without any dot "type_name",
* qualified name "something.type_name",
* fully qualified name "module.something.type_name"

There is also the question of breaking applications which rely on the current exact error message. And the question of removing legacy "%.100s" which was used before Python was able to allocate a buffer large enough to arbitrary string length. When an error is formatted in pure Python, names are never truncated.


(*) Call the function of the parent type when a method is overriden in a subclass. Example with PyTypeObject.tp_free called in a deallocator:

static void
abc_data_dealloc(_abc_data *self)
{
    PyTypeObject *tp = Py_TYPE(self);
    ...
    tp->tp_free(self);
    Py_DECREF(tp);
}


(*) The PEP 384 provides the most generic PyType_GetSlot() but it's not convenient to use: need to handle error (NULL), need to cast the void* into the expected type (error prone cast), etc.

We should slowly add more and more helper functions for most common use cases. We can try to convert a few C extension modules of the Python stdlib to see which use cases are the most common.

Hopefully, many use cases are already abstracted by widely used functions like PyNumber_Add(), PySequence_Size(), etc.


(*) Likely other issues that I forgot.