In the limited C-API we expose the following static PyObject variables:

• 5 singletons
• ~70 exception types
• ~70 other types

Since they are part of the limited API, they have a direct effect on the stable ABI.

The problem is that these objects should not be shared between interpreters. There are a number of possible solutions for isolating the objects, but the big constraint is that the solution cannot break the stable ABI.

Here are some solutions that I've considered:

1. immutable objects
   a. make the objects truly immutable/const
   • not trivial, if possible
     b. make the objects effectively immutable
   • (see Segfault when using store-context AST node in a load context #69016) use a really high refcount to make races irrelevant
2. per-interpreter objects
   a. replace them with macros that do a per-interpreter lookup
   b. replace them with simple placeholders and do a per-interpreter lookup internally
   c. replace them with PyObject placeholders and do a per-interpreter lookup internally

As far as I'm aware, only (1b) and (2c) are realistic and won't break the stable ABI (i.e. preserve layout).

(FWIW, I think that even with (1b) we would still have per-interpreter objects.)

-- Regarding (1a) --

See see #69016 for an example implementation. This includes storing some state for the objects in PyInterpreterState and doing a lookup internally.

pros:

• relatively straightforward to implement
• overlaps with other interests (see bpo-40255)
• makes the objects shareable between interpreters (making them more efficient)

cons:

• we have to ensure the objects stay immutable (a tractable problem if the solution is constrained to only the limited API objects)

-- Regarding (2c) --

This involves adding an API to get the per-interpreter object for a given identity value (flag, global, etc.) and then mapping the limited API objects to the corresponding per-interpreter ones.

pros:

• avoids a one-off solution
• extensions can stop using the variables directly (in favor of the new lookup API)

cons:

• effectively couples the C-API to the design (as long as the objects are in the limited API)
• many touches all over the C-API
• any future changes/additions in the C-API must deal with the objects

If the stable ABI weren't an issue then we would probably:

• deprecate using the objects directly
• do something like (2a) in the meantime

It may make sense to do that for "#ifndef Py_LIMITED_API", regardless of how we handle the limited API.

I can never remember what "Py_LIMITED_API" stands for. If it's not defined, does that mean we have the *unlimited* API? Is that a superset or a subset of the limited API?

Regarding 1a *and* 1b, I think it would help to list the specific reasons exceptions and other types are not entirely immutable. Is it just __subclasses__ or is there other state (apart from the refcount) that's mutable and visible to the end user? (Or even if it's visible to C API users.)

• 5 singletons

This issue is discussed in bpo-39511 "[subinterpreters] Per-interpreter singletons (None, True, False, etc.)".

Since they are part of the limited API, they have a direct effect on the stable ABI.

This issue is discussed in bpo-40601: "[C API] Hide static types from the limited C API".

I can never remember what "Py_LIMITED_API" stands for.

Include/README file is being written, have a look ;-)
https://github.com/python/cpython/pull/24884/files

One simple solution is to explicitly state that the limited API does not support subinterpreters. This is already implied by the fact that the multi-phase init API (PEP-489) requires subinterpreter support but is not part of the limited API.

If we establish this constraint then the problem I originally described here goes away (and we can close this issue).

(Note: I'm pretty sure this is something someone suggested to me at some point, rather than my own idea.)

FYI, I posted to capi-sig about this:

https://mail.python.org/archives/list/capi-sig@python.org/thread/INLCGPMTYFLRTWQL7RB4MUQZ37JAFRAU/

I am confused. How can widening the usable number of functions (i.e. using the whole C-API rather than the limited API) help c-extension modules be usable in subinterpreters? Aren't the same singletons, exception types, and other types exposed in the full C-API?

There seems to be much confusion here. Maybe on my side?

PEP-489 is *very much* part of the limited API.

Eric Snow proposes that C extensions which want to be compatible with subinterpreters must use an hypothetical variant of the C API which doesn't inherit flaws of the current C API. For example, static types like "&PyLong_Type" would be excluded.

To be clear, the limited C API does expose (indirectly) "&PyLong_Type". We are talking about a new variant of the C API.

The main interpreter would continue to use its static type "&PyLong_Type", whereas each subinterpreter would get its own "int" type allocated on the heap (heap type).

Someone has to write a PoC to ensure that this idea works in practice.

In bpo-40601, I proposed that all interpreters including the main interpreter only use heap types: remove "&PyLong_Type" from the C API which is a backward incompatible C API change.

I am confused. How can widening the usable number of functions (i.e. using
the whole C-API rather than the limited API) help c-extension modules be
usable in subinterpreters? Aren't the same singletons, exception types, and
other types exposed in the full C-API?

If Py_LIMITED_API is defined then things would stay the same. Otherwise we would replace the names with macros to do the appropriate lookup. (That isn't the whole story since the Py*_Type names are PyTypeObject and not PyObject*.)

PEP-489 is *very much* part of the limited API.

Gah, I missed that. That said, I don't think it matters; I just lose an easy point in the rationale. :)

FYI, I'm going to focus discussion on the capi-sig thread.