classification
Title: Create a way to always run tests in subprocesses within regrtest
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eli.bendersky Nosy List: Arfrever, eli.bendersky, ezio.melotti, ncoghlan, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2013-09-02 17:17 by eli.bendersky, last changed 2019-06-25 10:23 by vstinner. This issue is now closed.

Messages (10)
msg196793 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-02 17:17
There were numerous discussions and issues in the past about the cross-test dependencies that sometimes exist because some tests need to muck with import caches (both on the Python and C level). Some examples:

http://mail.python.org/pipermail/python-dev/2013-January/123409.html
http://mail.python.org/pipermail/python-dev/2013-August/127766.html

Issue #15651 is an example of a hack that has to stay in the code to allow tests to pass. Other things are also difficult - such as simulating situations where some optional modules don't exist (like pyexpat).

----

I suggest to add a capability to regrtest that will allow certain tests to request always being run inside a subprocess. Most of the infrastructure is already in place, since this is what regrtest uses to run with -jN today.

An alternative would be to add some test.support utility to help with this, but then test execution would not be uniform between -j1 and -jN and also some things would be difficult to track, such as -R executions (currently all -R runs of a single test have to be within a single process for the refcount tracking to work).
msg196810 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-02 21:43
A question that comes up is how should a module signal to regrtest that it needs to be run in a subprocess?

The most natural approach is to have a special attribute set in the module's global dict (for example: __REGRTEST_SUBPROCESS__ = True); however, there's a slight problem with this approach - regrtest has to import the module to see this attribute, and the module may do some work in its top-level code (commonly, imports) that already needs to be done within a subprocess.

One solution is to use a different approach, such as a separate file alongside the test file (i.e. test_foo.regrtest_subprocess alongside test_foo.py). This is quite ugly but "safe".

A different solution is to require modules that want to be executed in a subprocess to not do any top-level work; modules can already define a custom test_main function that regrtest discovers - we can require for this feature to work that modules do all their work through this function, rather than in the global scope.
msg196811 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-02 21:48
An easier hack is likely just a new "always run in subprocess" container
with submodule names in regrtest.py. It's not elegant, but it will work.
msg196812 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-02 21:50
Although the "well, don't do that then" alternative also sounds reasonable,
and better localises the information about how the test should run.
msg196814 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-02 21:54
> An easier hack is likely just a new "always run in subprocess" container
> with submodule names in regrtest.py. It's not elegant, but it will work.
>

True, that's also an option. I had it in mind in the beginning, but it's
too hacky for my tastes :-) Not doing state-changing stuff in the global
scope is a good practice anyway, so the restriction should not be too horrible.

Do we have some sort of conventions of outside-discoverable module
attributes (like the __REGRTEST_SUBPROCESS__ proposed above)? I.e. in terms
of naming, type, expected values?
msg196833 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-03 09:32
> The most natural approach is to have a special attribute set in the module's global dict (for example: __REGRTEST_SUBPROCESS__ = True); however, there's a slight problem with this approach - regrtest has to import the module to see this attribute, and the module may do some work in its top-level code (commonly, imports) that already needs to be done within a subprocess.

The main regrtest process can run auxilary child process which imports all test modules and says main process which of them have __REGRTEST_SUBPROCESS__=True.

It will be even better if the main process runs child process for testing all tests so when any test crashes it is possible to report this and respawn child process to continue testing other tests.
msg196844 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-03 13:47
On Tue, Sep 3, 2013 at 2:32 AM, Serhiy Storchaka <report@bugs.python.org>wrote:

>
> Serhiy Storchaka added the comment:
>
> > The most natural approach is to have a special attribute set in the
> module's global dict (for example: __REGRTEST_SUBPROCESS__ = True);
> however, there's a slight problem with this approach - regrtest has to
> import the module to see this attribute, and the module may do some work in
> its top-level code (commonly, imports) that already needs to be done within
> a subprocess.
>
> The main regrtest process can run auxilary child process which imports all
> test modules and says main process which of them have
> __REGRTEST_SUBPROCESS__=True.
>
> It will be even better if the main process runs child process for testing
> all tests so when any test crashes it is possible to report this and
> respawn child process to continue testing other tests.
>

Well, if we go *that* way, my initial proposal would be to just always run
every test in a subprocess. Kind of what happens today with -jN, just also
for -j1. Since most people, and I assume bots, run -jN anyway, they already
see each test executed in a subprocess. Some folks didn't feel good about
it because the stress testing "all in one process" provides is apparently
desired.

Your proposal complicates the flow significantly, IMHO. I'd just run each
test in its own subprocess and be done with it.
msg221991 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-30 23:20
I believe that #9517 may be relevant if we decide to take the subprocess route.
msg251963 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-30 20:39
I created the issue #25285: "regrtest: run tests in subprocesses with -j1 on buildbots".
msg346501 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 10:23
Since 2013, the Python test suite evolved a lot. First, we fixed hundreds of bugs where tests leaked various kinds of resources: processes, threads, file descriptors, memory, etc. Second, python3 -m test -j0 now runs each test file in a subprocess. Maybe we should emit a warning when tests are run sequentially, without -j0. Maybe it should be an opt-in option, rather than the default. But I prefer to close this old issue. Please open a new issue if you want to make further changes in regrtest.
History
Date User Action Args
2019-06-25 10:23:03vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg346501

stage: resolved
2019-02-24 22:35:56BreamoreBoysetnosy: - BreamoreBoy
2015-09-30 20:39:05vstinnersetnosy: + vstinner
messages: + msg251963
2014-06-30 23:20:44BreamoreBoysetversions: + Python 3.5, - Python 3.4
nosy: + BreamoreBoy

messages: + msg221991

type: enhancement
2013-10-08 13:16:19eli.benderskylinkissue16817 superseder
2013-09-03 16:56:45Arfreversetnosy: + Arfrever
2013-09-03 13:47:17eli.benderskysetmessages: + msg196844
2013-09-03 09:32:01serhiy.storchakasetmessages: + msg196833
2013-09-02 21:54:46eli.benderskysetmessages: + msg196814
2013-09-02 21:54:31eli.benderskysetmessages: - msg196813
2013-09-02 21:52:53eli.benderskysetmessages: + msg196813
2013-09-02 21:50:36ncoghlansetmessages: + msg196812
2013-09-02 21:48:27ncoghlansetmessages: + msg196811
2013-09-02 21:43:01eli.benderskysettype: enhancement -> (no value)
messages: + msg196810
stage: needs patch -> (no value)
2013-09-02 17:21:19ezio.melottisetnosy: + ezio.melotti

type: enhancement
stage: needs patch
2013-09-02 17:17:17eli.benderskycreate