We do very little testing of subinterpreters in CPython.  About all I'm aware of is in test_tracemalloc.  I'll be working on improving test coverage as a precursor to fixing some existing bugs.

There are some basic tests in test_capi as well:

* https://hg.python.org/cpython/file/09b223827f63/Lib/test/test_capi.py#l344
* https://hg.python.org/cpython/file/default/Programs/_testembed.c

FTR, subinterpreters are already accessible during testing with _testcapi.run_in_subinterp().

* https://hg.python.org/cpython/file/09b223827f63/Modules/_testcapimodule.c#l2615


That function is used here:

* Lib/test/test_threading.py
* Lib/test/support.__init__.py (run_in_subinterp())

test.support.run_in_subinterp() is used here:

* Lib/test/test_atexit.py
* Lib/test/test_capi.py
* Lib/test/test_threading.py

Also, I was mistaken about test_tracemalloc.

Now I'm wondering what further test coverage we really need...

Sorry about the misleading reference to tracemalloc in my email - it was actually test_atexit I was debugging in the PEP 432 branch. tracemalloc only came up in that context because test.support.run_in_subinterp() automatically skips subinterpreter tests when tracemalloc is running. From the function comments:

# Issue #10915, #15751: PyGILState_*() functions don't work with
# sub-interpreters, the tracemalloc module uses these functions internally

As far as test coverage goes, what I would actually like to do is to run regrtest itself in a subinterpreter, and blacklist tests until it passes. This would double the length of a test run, so hiding it behind a resource option would probably be a good idea.

> Now I'm wondering what further test coverage we really need...

Ideally we'd test every C module with the tests executing "in
parallel" (sort of) in multiple interpreters.

I have done so for _decimal, which is mostly okay due to the
thread-local contexts. However, there are a couple of minor
glitches.


The reason I haven't pursued this is that *if* you run _decimal
in multiple interpreters, you get nasty sounding (but harmless)
warnings from libmpdec, which complains about being initialized
multiple times.


So from the absence of bug reports I concluded that no one is
using the sub-interpreter feature with _decimal.

Adding Petr Viktorin to the nosy list as well, as the kind of issues Stefan mentions there are the kinds of things that the PEP 489 extension module import issues *didn't* address yet, but we'd like to address in the 3.6 iteration of the multi-phase initialisation support for extension modules.

Petr, for context, this issue is about improving our test coverage for subinterpreter support - see the earlier messages for more details.

I think for fast access we need a hybrid solution that allows
static types (heap types slowed down _decimal) *and* cache the
thread local values (like it's currently done for the thread-local
context in _decimal).

Caching the context brought an enormous speedup (about 20%).

I'm not sure if all that can be done in a general API, but it's
clearly the fastest way.

That's jumping ahead a little - this issue is only about running the regression test suite from a subinterpreter, and establishing what *already* works properly. Dealing with the fallout of any quirks and outright failures we discover that way will be a different issue :)

Nick, this may be a misunderstanding, but you mentioned issues
that PEP 489 did not address yet.  The only issue left for _decimal
is the speed issue.  Sub-interpreter tests run fine even with the
existing module state API, *if* one is willing to take a speed hit.

[I get that this is outside the scope of this issue, just for
clarification.]

As a possible starting point for this, I'll point to https://hg.python.org/cpython/file/02b81a82a57d/Lib/test/__main__.py

Now, I'd expected running that in a process that was *already* running regrtest to fail miserably (it would stomp all over itself).

But what we could potentially do is launch a *clean* subprocess, where the only thing it did was:

    from _testcapi import run_in_subinterp

    regrtest_in_subinterpreter = (
    """
    from test import regrtest
    regrtest.main_in_temp_cwd()
    """
    )
    run_in_subinterp(regrtest_in_subinterpreter)

I'd currently expect that to fail as well, but I think the failures might be enlightening :)

I'm also not sure we need to integrate this directly into the main regrtest test runner - it could just be a separate submodule invoked like "python -m test.subinterpretertest".

If we later decided to integrate it, then it could go behind a "-usubinterpreter" resource that invoked "test.subinterpretertest" in a subprocess.

+1 for python -m test.subinterpretertest. Based on my experiments,
most tests will either fail or leak memory at the very least.

Note that bpo-32604 is strongly related.