classification
Title: improve test coverage for subinterpreters
Type: Stage: test needed
Components: Tests Versions: Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: encukou, eric.snow, grahamd, ncoghlan, skrah
Priority: normal Keywords:

Created on 2015-07-03 00:30 by eric.snow, last changed 2015-07-13 17:12 by skrah.

Messages (13)
msg246107 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-07-03 00:30
We do very little testing of subinterpreters in CPython.  About all I'm aware of is in test_tracemalloc.  I'll be working on improving test coverage as a precursor to fixing some existing bugs.
msg246149 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-07-03 08:10
There are some basic tests in test_capi as well:

* https://hg.python.org/cpython/file/09b223827f63/Lib/test/test_capi.py#l344
* https://hg.python.org/cpython/file/default/Programs/_testembed.c
msg246207 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-07-03 19:17
FTR, subinterpreters are already accessible during testing with _testcapi.run_in_subinterp().

* https://hg.python.org/cpython/file/09b223827f63/Modules/_testcapimodule.c#l2615


That function is used here:

* Lib/test/test_threading.py
* Lib/test/support.__init__.py (run_in_subinterp())

test.support.run_in_subinterp() is used here:

* Lib/test/test_atexit.py
* Lib/test/test_capi.py
* Lib/test/test_threading.py
msg246208 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-07-03 19:21
Also, I was mistaken about test_tracemalloc.
msg246237 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-07-04 01:22
Now I'm wondering what further test coverage we really need...
msg246238 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-07-04 04:10
Sorry about the misleading reference to tracemalloc in my email - it was actually test_atexit I was debugging in the PEP 432 branch. tracemalloc only came up in that context because test.support.run_in_subinterp() automatically skips subinterpreter tests when tracemalloc is running. From the function comments:

# Issue #10915, #15751: PyGILState_*() functions don't work with
# sub-interpreters, the tracemalloc module uses these functions internally

As far as test coverage goes, what I would actually like to do is to run regrtest itself in a subinterpreter, and blacklist tests until it passes. This would double the length of a test run, so hiding it behind a resource option would probably be a good idea.
msg246253 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-07-04 12:33
> Now I'm wondering what further test coverage we really need...

Ideally we'd test every C module with the tests executing "in
parallel" (sort of) in multiple interpreters.

I have done so for _decimal, which is mostly okay due to the
thread-local contexts. However, there are a couple of minor
glitches.


The reason I haven't pursued this is that *if* you run _decimal
in multiple interpreters, you get nasty sounding (but harmless)
warnings from libmpdec, which complains about being initialized
multiple times.


So from the absence of bug reports I concluded that no one is
using the sub-interpreter feature with _decimal.
msg246255 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-07-04 13:20
Adding Petr Viktorin to the nosy list as well, as the kind of issues Stefan mentions there are the kinds of things that the PEP 489 extension module import issues *didn't* address yet, but we'd like to address in the 3.6 iteration of the multi-phase initialisation support for extension modules.

Petr, for context, this issue is about improving our test coverage for subinterpreter support - see the earlier messages for more details.
msg246256 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-07-04 13:30
I think for fast access we need a hybrid solution that allows
static types (heap types slowed down _decimal) *and* cache the
thread local values (like it's currently done for the thread-local
context in _decimal).

Caching the context brought an enormous speedup (about 20%).

I'm not sure if all that can be done in a general API, but it's
clearly the fastest way.
msg246257 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-07-04 13:51
That's jumping ahead a little - this issue is only about running the regression test suite from a subinterpreter, and establishing what *already* works properly. Dealing with the fallout of any quirks and outright failures we discover that way will be a different issue :)
msg246298 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-07-05 09:53
Nick, this may be a misunderstanding, but you mentioned issues
that PEP 489 did not address yet.  The only issue left for _decimal
is the speed issue.  Sub-interpreter tests run fine even with the
existing module state API, *if* one is willing to take a speed hit.

[I get that this is outside the scope of this issue, just for
clarification.]
msg246677 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-07-13 01:52
As a possible starting point for this, I'll point to https://hg.python.org/cpython/file/02b81a82a57d/Lib/test/__main__.py

Now, I'd expected running that in a process that was *already* running regrtest to fail miserably (it would stomp all over itself).

But what we could potentially do is launch a *clean* subprocess, where the only thing it did was:

    from _testcapi import run_in_subinterp

    regrtest_in_subinterpreter = (
    """
    from test import regrtest
    regrtest.main_in_temp_cwd()
    """
    )
    run_in_subinterp(regrtest_in_subinterpreter)

I'd currently expect that to fail as well, but I think the failures might be enlightening :)

I'm also not sure we need to integrate this directly into the main regrtest test runner - it could just be a separate submodule invoked like "python -m test.subinterpretertest".

If we later decided to integrate it, then it could go behind a "-usubinterpreter" resource that invoked "test.subinterpretertest" in a subprocess.
msg246695 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-07-13 17:12
+1 for python -m test.subinterpretertest. Based on my experiments,
most tests will either fail or leak memory at the very least.
History
Date User Action Args
2015-07-13 17:12:41skrahsetmessages: + msg246695
2015-07-13 01:52:52ncoghlansetmessages: + msg246677
2015-07-05 09:53:01skrahsetmessages: + msg246298
2015-07-04 13:51:12ncoghlansetmessages: + msg246257
2015-07-04 13:30:48skrahsetmessages: + msg246256
2015-07-04 13:20:03ncoghlansetnosy: + encukou
messages: + msg246255
2015-07-04 12:33:05skrahsetnosy: + skrah
messages: + msg246253
2015-07-04 04:10:16ncoghlansetstatus: pending -> open
nosy: + grahamd
messages: + msg246238

2015-07-04 01:22:27eric.snowsetstatus: open -> pending

messages: + msg246237
2015-07-03 19:21:27eric.snowsetmessages: + msg246208
2015-07-03 19:17:20eric.snowsetmessages: + msg246207
2015-07-03 08:10:52ncoghlansetmessages: + msg246149
2015-07-03 00:30:03eric.snowcreate