In the interest of getting something landed for 3.7, so we can start using it in tests, I'm putting up a patch for a low-level interpreters module.  In some ways this is a precursor for issue #30439, which will add a proper public stdlib module in 3.8.

The module I'm adding draws from the ideas in PEP 554 (particularly for channels).  Consequently, this will also give us an opportunity to try out some of the semantics from the PEP to give us better ideas for 3.8.

I expect to have some follow-on patches to facilitate simpler use in tests.  This patch is big enough already. :)

@Ned, it may be a little tight to land this given the time left before beta 1.  However, this is meant as a tool for us to use in the test suite (particularly to test the subinterpreter C-API).  So I'm arguing that, if necessary, it would still be okay to land this after the feature freeze.  (I'm still hoping to get this in before the cutoff.)  What do you think?

@Nick, I may make the name change you suggested in issue #30439 ("_subinterpreters").

FYI, there are a few things I need to clean up in the PR.  However, I expect that those changes will be minor relative to the the whole patch, so I wanted to get the ball rolling on a review. :)

@Eric, given the breadth of change introduced in the PR (including adding a new extension), I think it would be best if at all possible to get it in for beta 1 if we can resolve the review comments in time.  If necessary and if there are no objections from other core developers, I would be willing to consider making an exception and allowing it into beta 2 as long as it remains a private interface. If it looks like it won't be in releasable shape by then, I think you should hold off for 3.8; doing otherwise would be unfair to others and to our downstream beta users / testers, for example, even if it is private, adding a new extension and setup.py changes potentially affect downstream packagers.

Sounds good, Ned.  Thanks for taking a look.  I should have everything finished up by Friday, so I'm hopeful for landing the change before the deadline.  I may have a few minor tweaks to make after that, but I'll discuss that with you before making any changes if that happens.

New changeset 7f8bfc9b9a8381ddb768421b5dd5cbd970266190 by Eric Snow in branch 'master':
bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. (gh-1748)
https://github.com/python/cpython/commit/7f8bfc9b9a8381ddb768421b5dd5cbd970266190

I've merged the patch without Windows support, which shouldn't be a problem given the purpose of the extension module.  I've also added a PR for get the module building under Windows.  I'd like to get that resolved ASAP.

Eric, looks like some buildbots are unhappy, for instance:

http://buildbot.python.org/all/#builders/13/builds/648

Yeah, I'm looking into it.  Also, I noticed some refleaks that I'll be sorting out.

On 4 of the buildbots:


======================================================================
ERROR: test_drop_multiple_times (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 890, in test_drop_multiple_times
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)
======================================================================
ERROR: test_drop_single_user (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 848, in test_drop_single_user
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)
======================================================================
ERROR: test_drop_used_multiple_times_by_single_user (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 957, in test_drop_used_multiple_times_by_single_user
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)
======================================================================
ERROR: test_drop_with_unused_items (test.test__xxsubinterpreters.ChannelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test__xxsubinterpreters.py", line 899, in test_drop_with_unused_items
    interpreters.channel_drop_interpreter(cid, send=True, recv=True)
SystemError: More keyword list entries (7) than format specifiers (3)

On the PPC64 AIX 3.x buildbot:

======================================================================
FAIL: test_repr (test.test__xxsubinterpreters.ChannelIDTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test__xxsubinterpreters.py", line 784, in test_repr
    self.assertEqual(repr(cid), 'ChannelID(10)')
AssertionError: 'ChannelID(0)' != 'ChannelID(10)'
- ChannelID(0)
+ ChannelID(10)
?           +

I just put up a PR that should fix the 4 buildbots.

New changeset 83e64c8a544028ae677af2a0bc268dbe1c11cc3a by Eric Snow in branch 'master':
bpo-32604: NULL-terminate kwlist in channel_drop_interpreter(). (gh-5437)
https://github.com/python/cpython/commit/83e64c8a544028ae677af2a0bc268dbe1c11cc3a

The buildbots should be happier now.  I'll keep an eye on them.

A couple defects reported by coverity:



** CID 1428758: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 45 in _coerce_id()
________________________________________________________________________________________________________ *** CID 1428758: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 45 in _coerce_id()
39 }
40 if (cid < 0) {
41 PyErr_SetString(PyExc_ValueError, 42 "'id' must be a non-negative int"); 43 return -1;
44 }
45 if (cid > INT64_MAX) {
46 PyErr_SetString(PyExc_ValueError, 47 "'id' too large (must be 64-bit int)"); 48 return -1;
49 }
50 return cid;
** CID 1428757: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 1215 in channelid_richcompare()
________________________________________________________________________________________________________ *** CID 1428757: Integer handling issues (CONSTANT_EXPRESSION_RESULT) /Modules/_xxsubinterpretersmodule.c: 1215 in channelid_richcompare()
1209 }
1210 int64_t othercid = PyLong_AsLongLong(other); 1211 // XXX decref other here?
1212 if (othercid == -1 && PyErr_Occurred() != NULL) { 1213 return NULL;
1214 }
1215 if (othercid < 0 || othercid > INT64_MAX) { 1216 equal = 0;
 
1217 }
1218 else {
1219 equal = (cid->id == othercid); 1220 }
** CID 1428756: Error handling issues (CHECKED_RETURN) /Modules/gcmodule.c: 1071 in gc_enable_impl()

New changeset 4e9da0d163731caa79811c723c703ee416c31826 by Eric Snow in branch 'master':
bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. (#5507)
https://github.com/python/cpython/commit/4e9da0d163731caa79811c723c703ee416c31826

I've landed a PR that fixes all the memory leaks in the module.  It also fixes the 2 defects reported by coverity.  The only thing left here is to get the module building under Windows.

New changeset f33ecedcad5a001735fa4ded5d21caa2cbf27911 by Eric Snow (Miss Islington (bot)) in branch '3.7':
bpo-32604: Fix memory leaks in the new _xxsubinterpreters module. (GH-5507)
https://github.com/python/cpython/commit/f33ecedcad5a001735fa4ded5d21caa2cbf27911

FYI, out of 2389 source lines in the C extension, 1563 are the channel-related code.  That means the non-channel code is 826 lines (about a third).  That non-channel code does not depend on the channel code at all and I considered splitting the source out, but figured there wasn't enough benefit.  However, I might revisit the matter later when I circle back to PEP 554. :)

New changeset 310b05289b5d9550040f469e60b5e8e77f1022b6 by Zachary Ware in branch 'master':
bpo-32604: Make _xxsubinterpreters build on Windows (GH-5516)
https://github.com/python/cpython/commit/310b05289b5d9550040f469e60b5e8e77f1022b6

New changeset fe61e8d8c7cd1f6fb0ce7e9b8f0460b47b5f29a5 by Zachary Ware (Miss Islington (bot)) in branch '3.7':
bpo-32604: Make _xxsubinterpreters build on Windows (GH-5624)
https://github.com/python/cpython/commit/fe61e8d8c7cd1f6fb0ce7e9b8f0460b47b5f29a5

New changeset 4c6955e2b0ccf88c705f8d1fac685a8e65f9699e by Eric Snow in branch 'master':
bpo-32604: Clean up created subinterpreters before runtime finalization. (gh-5709)
https://github.com/python/cpython/commit/4c6955e2b0ccf88c705f8d1fac685a8e65f9699e

New changeset 3db05a3a9c109f49d54b4965852e41c657e9b117 by Eric Snow (Miss Islington (bot)) in branch '3.7':
bpo-32604: Clean up created subinterpreters before runtime finalization. (gh-5710)
https://github.com/python/cpython/commit/3db05a3a9c109f49d54b4965852e41c657e9b117

Eric, it looks like your recent commit introduced a refleak.  We need to fix it before beta2.

~/d/p/cpython (master $) » ./python.exe -m test -R3:3 test_multiprocessing_fork
Run tests sequentially
0:00:00 load avg: 2.52 [1/1] test_multiprocessing_fork
beginning 6 repetitions
123456
......
test_multiprocessing_fork leaked [21, 2, 1] memory blocks, sum=24
test_multiprocessing_fork leaked [2, 0, 0] file descriptors, sum=2
test_multiprocessing_fork failed in 9 min 48 sec

1 test failed:
    test_multiprocessing_fork


And just before it:


~/d/p/cpython ((bd093355…) $) » ./python.exe -m test -R3:3 test_multiprocessing_fork
Run tests sequentially
0:00:00 load avg: 3.70 [1/1] test_multiprocessing_fork
beginning 6 repetitions
123456
......
test_multiprocessing_fork passed in 9 min 12 sec
1 test OK.

Total duration: 9 min 12 sec
Tests result: SUCCESS

@Yury, thanks!  I thought I had checked before I made the PR, but apparently not.  I'll git it fixed.

I'm not seeing any refleak (on linux/clang).  I'm guessing this is Windows-specific (based on use of "./python.exe").  How does test_multiprocessing_fork even run on Windows?  I thought "fork" is an unsupported start method on Windows (see https://docs.python.org/3.7/library/multiprocessing.html#contexts-and-start-methods).

Also, I'm not sure how my change might cause a refleak outside of code using the _xxsubinterpreters module.

Davin, any thoughts on how my most recent commit (for this issue) might be causing the leaks Yury found?

Looks like macOS rather than Windows, and I can't reproduce it locally with current master:

14:28 $ ./python.exe -m test -R3:3 test_multiprocessing_fork
Run tests sequentially
0:00:00 load avg: 2.68 [1/1] test_multiprocessing_fork
beginning 6 repetitions
123456
......
test_multiprocessing_fork passed in 9 min 20 sec
1 test OK.

Total duration: 9 min 20 sec
Tests result: SUCCESS



The Windows and Linux refleak buildbots [1] are currently unhappy about other things that appear to be unrelated.

[1] http://buildbot.python.org/all/#/builders?tags=%2Brefleak

FYI I found out about this refleak from https://mail.python.org/pipermail/python-checkins/2018-February/153907.html

So it's definitely not Mac OS X specific thing.

I'm pretty sure that report is still running on the last changeset on hg.python.org; can you confirm that Antoine?

New changeset f53d9f2778a87bdd48eb9030f782a4ebf9e7622f by Eric Snow in branch 'master':
bpo-32604: Swap threads only if the interpreter is different. (gh-5778)
https://github.com/python/cpython/commit/f53d9f2778a87bdd48eb9030f782a4ebf9e7622f

New changeset eed3c7af4ec520d324a55e0357f41589a66906f7 by Eric Snow (Miss Islington (bot)) in branch '3.7':
bpo-32604: Swap threads only if the interpreter is different. (gh-5783)
https://github.com/python/cpython/commit/eed3c7af4ec520d324a55e0357f41589a66906f7

What's the status of this issue?  3.7.0b2 is tagging in 48 hours or so.

_xxsubinterpreters has been added, and we'll use PEP 554 as the interim documentation while it's only exposed for testing purposes.

As far as I know, everything related to this issue is in a stable state.  I have not been able to determine any way in which the reported memory leak (in test_multiprocessing_fork) could be related.  Sorry, I didn't notice the release blocker priority.

I do not plan on landing any further changes here for beta2.  However, I do plan on a few more additions under Lib/test before we go final.

Still planned for 3.7:

1. add more tests in Lib/test/test__xxsubinterpreters.py
2. add test.support.interpreters (high-level module)
3. add more tests in Lib/test/test_interpreters.py

At that point I'll start using the high-level module in tests for subinterpreter functionality and for other open issues (e.g. #10915, #15751), aiming to improve stability of the feature.

> Still planned for 3.7: [...]

Eric, please try to get everything you want done and into master as soon as possible, then produce a PR for 3.7, and then we can review it and make a decision if it is appropriate for 3.7.  As I've mentioned in msg310448 and stated elsewhere, the expectation is that the few post-b1 feature freeze extensions granted were only for b2.  Even changes limited to Lib/test can break buildbots and downstream users' testing and take multiple releases to stabilize.  Thanks for your work on this!

Sounds good, Ned.  Sorry for my confusion.

New changeset 6bd0c476c58ca0046969f70dc2a4e4dfb3268062 by Eric Snow in branch '3.7':
bpo-32604: Remove xid registry. (#6813)
https://github.com/python/cpython/commit/6bd0c476c58ca0046969f70dc2a4e4dfb3268062

As agreed upon at the PyCon dev sprints, 6bd0c476c58ca0046969f70dc2a4e4dfb3268062 removes _xxsubinterpreters from the 3.7 branch and 3.7.0 and retargets for 3.8.  Since it was intended for internal use, no user documentation or APIs should be affected.

New changeset 6d2cd9036c0ab78a83de43d1511befb7a7fc0ade by Eric Snow in branch 'master':
bpo-32604: Improve subinterpreter tests. (#6914)
https://github.com/python/cpython/commit/6d2cd9036c0ab78a83de43d1511befb7a7fc0ade

New changeset 3ab0136ac5d6059ce96d4debca89c5f5ab0356f5 by Eric Snow in branch 'master':
bpo-32604: Implement force-closing channels. (gh-6937)
https://github.com/python/cpython/commit/3ab0136ac5d6059ce96d4debca89c5f5ab0356f5

New changeset 110bc01407ac8c75545d0386577c6e17254d97d9 by Eric Snow in branch 'master':
bpo-33615: Temporarily disable a test that is triggering crashes on a few buildbots. (gh-7288)
https://github.com/python/cpython/commit/110bc01407ac8c75545d0386577c6e17254d97d9

New changeset 63799136e6c0491bb5d6f4a234d5a775db3458db by Eric Snow in branch 'master':
bpo-33615: Re-enable a subinterpreter test. (gh-7251)
https://github.com/python/cpython/commit/63799136e6c0491bb5d6f4a234d5a775db3458db

Also see #24553.

New changeset 5e8c691594d68925213d36296ce7c4b3e90bcb1d by Eric Snow in branch 'master':
bpo-32604: Add support for a "default" arg in channel_recv(). (GH-19770)
https://github.com/python/cpython/commit/5e8c691594d68925213d36296ce7c4b3e90bcb1d

New changeset a1d9e0accd33af1d8e90fc48b34c13d7b07dcf57 by Eric Snow in branch 'master':
bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)
https://github.com/python/cpython/commit/a1d9e0accd33af1d8e90fc48b34c13d7b07dcf57

The commit a1d9e0accd33af1d8e90fc48b34c13d7b07dcf57 introduced reference leaks:

$ ./python -m test -R 3:3 test__xxsubinterpreters -m test.test__xxsubinterpreters.RunFailedTests.test_traceback_propagated
(...)
test__xxsubinterpreters leaked [1863, 1861, 1863] references, sum=5587
test__xxsubinterpreters leaked [559, 558, 559] memory blocks, sum=1676
(...)

The buildbots are broken for one whole week and nobody is available to investigate the reference leak regression. Following the CI policy, I wrote PR 20089 to revert the change which broke the CI:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

If someone wants to push again the change, make sure that "./python -m test -R 3:3 test__xxsubinterpreters" does not leak.

New changeset f2c3b6823bc4777d4a14eb0c3615b719521f763a by Victor Stinner in branch 'master':
Revert "bpo-32604: [_xxsubinterpreters] Propagate exceptions. (GH-19768)" (GH-20089)
https://github.com/python/cpython/commit/f2c3b6823bc4777d4a14eb0c3615b719521f763a

The revert fixed the test:

$ ./python -m test -R 3:3 test__xxsubinterpreters 
(...)
Tests result: SUCCESS

I marked bpo-30439 and bpo-31375 as duplicates of this issue.

I marked bpo-39881 as a duplicate of this issue. It has two open PRs:

* PR 18817
* PR 19985

I changed their title to use bpo-32604.

New changeset 9d17cbf33df7cfb67ca0f37f6463ba5c18676641 by Joannah Nanjekye in branch 'master':
bpo-32604: PEP 554 for use in test suite (GH-19985)
https://github.com/python/cpython/commit/9d17cbf33df7cfb67ca0f37f6463ba5c18676641

The new test leaks references:

https://buildbot.python.org/all/#/builders/563/builds/105

test_interpreters leaked [216, 216, 216] references, sum=648
test_interpreters leaked [84, 84, 84] memory blocks, sum=252

Use "./python -m test.bisect_cmd -R 3:3 test_interpreters" to find tests which leak. Example:

$ ./python -m test test_interpreters -R 3:3 -m test.test_interpreters.TestInterpreterDestroy.test_from_current
0:00:00 load avg: 0.82 Run tests sequentially
0:00:00 load avg: 0.82 [1/1] test_interpreters
beginning 6 repetitions
123456
......
test_interpreters leaked [36, 36, 36] references, sum=108
test_interpreters leaked [14, 14, 14] memory blocks, sum=42
test_interpreters failed

== Tests result: FAILURE ==

1 test failed:
    test_interpreters

Total duration: 1.3 sec
Tests result: FAILURE

I will look at this tommorrow. Am abit busy today. Thanks Victor

Commit 9d17cbf33df7cfb67ca0f37f6463ba5c18676641 is causing all refleak buildbots to fail and is masking other issues and causing some confusion already in other PRs (like https://github.com/python/cpython/pull/20433) so I opened a revert in PR 20465 following the buildbot revert policy (https://discuss.python.org/t/policy-to-revert-commits-on-buildbot-failure/404).

Please, open a new PR with this change again once the refleaks have been resolved.

New changeset 7d80b35af1ee03834ae4af83e920dee89c2bc273 by Pablo Galindo in branch 'master':
Revert "bpo-32604: PEP 554 for use in test suite (GH-19985)" (#20465)
https://github.com/python/cpython/commit/7d80b35af1ee03834ae4af83e920dee89c2bc273

The following test is enough to reproduce the leak:
---
import unittest
import _xxsubinterpreters as _interpreters

class TestInterpreterDestroy(unittest.TestCase):

    def tearDown(self):
        for interp_id in _interpreters.list_all():
            if interp_id == 0:  # main
                continue
            _interpreters.destroy(interp_id)

    def test_from_current(self):
        interp_id = _interpreters.create(isolated=True)
        _interpreters.run_string(interp_id, "import select")
---

New changeset 18a90248fdd92b27098cc4db773686a2d10a4d24 by Victor Stinner in branch 'master':
bpo-32604: Fix reference leak in select module (GH-20600)
https://github.com/python/cpython/commit/18a90248fdd92b27098cc4db773686a2d10a4d24

New changeset bae872f1fe9b3a0d3e3b8800a2ac8d6b440d6e4d by Joannah Nanjekye in branch 'master':
bpo-32604: Recommit "bpo-32604: PEP 554 for use in test suite (GH-19985)" (GH-20611)
https://github.com/python/cpython/commit/bae872f1fe9b3a0d3e3b8800a2ac8d6b440d6e4d

test__xxsubinterpreters and test_interpreters failed on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/152/builds/973

See also bpo-37224: "[subinterpreters] test__xxsubinterpreters fails randomly".

IMO there are multiple race conditions in these tests.

======================================================================
FAIL: test_subinterpreter (test.test__xxsubinterpreters.IsRunningTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test__xxsubinterpreters.py", line 478, in test_subinterpreter
    self.assertTrue(interpreters.is_running(interp))
AssertionError: False is not true

======================================================================
FAIL: test_still_running (test.test_interpreters.TestInterpreterDestroy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_interpreters.py", line 309, in test_still_running
    interp.close()
AssertionError: RuntimeError not raised

======================================================================
FAIL: test_subinterpreter (test.test_interpreters.TestInterpreterIsRunning)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_interpreters.py", line 207, in test_subinterpreter
    self.assertTrue(interp.is_running())
AssertionError: False is not true

======================================================================
FAIL: test_already_running (test.test_interpreters.TestInterpreterRun)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_interpreters.py", line 383, in test_already_running
    self.interp.run('print("spam")')
AssertionError: RuntimeError not raised

Joannah, Eric, could you look into this? Having random failures in the builbots can make debugging more challenging, hide other issues and can also spam the buildbot list :(

New changeset 818f5b597ae93411cc44e404544247d436026a00 by Eric Snow in branch 'master':
bpo-32604: Clean up test.support.interpreters. (gh-20926)
https://github.com/python/cpython/commit/818f5b597ae93411cc44e404544247d436026a00

@Pablo, yeah, I'll try to take a look this week.  Are these new failures?

Keep in mind that these tests can sometimes expose existing bugs in our runtime (as we fix runtime issues elsewhere) rather than regressions, though that isn't necessarily the case here. :)