This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [subinterpreters] Add a "PyInterpreterState *" field to PyTypeObject.
Type: enhancement Stage: resolved
Components: Subinterpreters Versions: Python 3.9
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, eric.snow, petr.viktorin, scoder, steve.dower, ta1hia, vstinner
Priority: normal Keywords:

Created on 2019-09-13 13:33 by eric.snow, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (10)
msg352325 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-09-13 13:33
A "PyInterpreterState *interp" field on PyTypeObject would allow us to quickly access the originating interpreter from any object.  This will help us avoid more costly lookups.  The additional pointer used for this should not have a significant impact on the memory usage of CPython.
msg352330 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-13 13:41
PEP 573 already proposes adding a pointer to the *module* to heap types (well, those that opt in).
We should rather add a pointer from *modules* to the interpreter state.
msg352408 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2019-09-14 05:27
I think access to the global module state is the most critical for an extension, probably by far. A direct pointer to the interpreter state from every type feels like an optimisation that may be premature at this point. Modules should have that reference, though, as long as we assume a unique interpreter for each module instance (which we currently do).
msg353339 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-26 21:04
How about you add a module state reference to the type object and a interpreter state reference to the module object?
msg353340 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 21:08
> A "PyInterpreterState *interp" field on PyTypeObject would allow us to quickly access the originating interpreter from any object.

Does it mean that a type A created in interperter 1 can be seen/accessed from interpreter 2? First I understood that an interepter should only access objects created by itself?
msg353341 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-26 21:19
> Does it mean that a type A created in interperter 1 can be seen/accessed from interpreter 2?

No, but it means that a type A *knows* that it was created in interpreter 1 without relying on the current thread local storage.

What it does with this information is totally independent :)

> How about you add a module state reference to the type object and a interpreter state reference to the module object?

Module state != module object, IIUC, but a chain like:

instance -> type -> module -> interpreter

would be feasible, if an extra step. Though that's only if PEP 573 makes it *compulsory* for heap types to contain a pointer to their module. Without a pointer to the interpreter state, an object could lose access to its memory (de)allocator (or its lock), depending on the model we eventually use to reduce reliance on the GIL in subinterpreters.

(Yes, this is one step in the direction of reducing the impact of the GIL, as well as reducing the tight coupling between native threads and Python threads, which is a huge pain point for embedders.)
msg353349 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-27 08:46
> only if PEP 573 makes it *compulsory* for heap types to contain a pointer to their module

And that's impossible without either breaking API (or some hackery).
PyType_FromSpec doesn't get the information, and it can be easily outside module initialization.

One possibility is to attach the __main__ module (or sys, or some synthetic module) to "module-less" classes, just to have something interpreter-specific there. But at that point, a dedicated "PyInterpreterState *" field starts looking like a better alternative.
It does complicate the "chain":

instance --> type -?> module --> interpreter
               |                   ^
               '-------------------'

But, one more thing that we need to think about is static types. Those are shared across interpreters; the "PyInterpreterState *" field should only be added to PyHeapTypeObject. ISTM that until we phase out static types entirely, you can't get the interpreter by following pointers from a type/instance anyway.
At that point, the opt-in approach of PEP 573 starts looking more attractive again:

instance --> type -?> module --> interpreter
msg353351 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-27 09:39
> No, but it means that a type A *knows* that it was created in interpreter 1 without relying on the current thread local storage.

What's wrong with _PyInterpreterState_GET_UNSAFE()?

This macro is based on _Py_atomic_load_relaxed(&_PyRuntime.gilstate.tstate_current).

If this atomic variable is an issue, do we need a better solution to get the current thread state and get the current interpreter?
msg370746 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-05 01:21
Which kind of code pattern is supposed to be more efficient if it would be possible to get a "PyInterpreterState *" from a "PyTypeObject*"? I don't understand which problem this issue is supposed to solve.
msg371129 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-09 17:01
I close the issue.
History
Date User Action Args
2022-04-11 14:59:20adminsetgithub: 82341
2020-06-09 17:01:03vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg371129

stage: needs patch -> resolved
2020-06-05 01:21:48vstinnersetmessages: + msg370746
2020-05-15 00:52:49vstinnersetcomponents: + Subinterpreters, - Interpreter Core
title: Add a "PyInterpreterState *" field to PyTypeObject. -> [subinterpreters] Add a "PyInterpreterState *" field to PyTypeObject.
2019-09-27 09:39:27vstinnersetmessages: + msg353351
2019-09-27 08:46:26petr.viktorinsetmessages: + msg353349
2019-09-26 21:19:16steve.dowersetmessages: + msg353341
2019-09-26 21:08:18vstinnersetnosy: + vstinner
messages: + msg353340
2019-09-26 21:04:12christian.heimessetnosy: + christian.heimes
messages: + msg353339
2019-09-26 17:29:11ta1hiasetnosy: + ta1hia
2019-09-14 05:27:34scodersetmessages: + msg352408
2019-09-13 13:41:06petr.viktorinsetmessages: + msg352330
2019-09-13 13:33:29eric.snowcreate