New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create a way to always run tests in subprocesses within regrtest #63106
Comments
There were numerous discussions and issues in the past about the cross-test dependencies that sometimes exist because some tests need to muck with import caches (both on the Python and C level). Some examples: http://mail.python.org/pipermail/python-dev/2013-January/123409.html Issue bpo-15651 is an example of a hack that has to stay in the code to allow tests to pass. Other things are also difficult - such as simulating situations where some optional modules don't exist (like pyexpat). ---- I suggest to add a capability to regrtest that will allow certain tests to request always being run inside a subprocess. Most of the infrastructure is already in place, since this is what regrtest uses to run with -jN today. An alternative would be to add some test.support utility to help with this, but then test execution would not be uniform between -j1 and -jN and also some things would be difficult to track, such as -R executions (currently all -R runs of a single test have to be within a single process for the refcount tracking to work). |
A question that comes up is how should a module signal to regrtest that it needs to be run in a subprocess? The most natural approach is to have a special attribute set in the module's global dict (for example: __REGRTEST_SUBPROCESS__ = True); however, there's a slight problem with this approach - regrtest has to import the module to see this attribute, and the module may do some work in its top-level code (commonly, imports) that already needs to be done within a subprocess. One solution is to use a different approach, such as a separate file alongside the test file (i.e. test_foo.regrtest_subprocess alongside test_foo.py). This is quite ugly but "safe". A different solution is to require modules that want to be executed in a subprocess to not do any top-level work; modules can already define a custom test_main function that regrtest discovers - we can require for this feature to work that modules do all their work through this function, rather than in the global scope. |
An easier hack is likely just a new "always run in subprocess" container |
Although the "well, don't do that then" alternative also sounds reasonable, |
True, that's also an option. I had it in mind in the beginning, but it's Do we have some sort of conventions of outside-discoverable module |
The main regrtest process can run auxilary child process which imports all test modules and says main process which of them have __REGRTEST_SUBPROCESS__=True. It will be even better if the main process runs child process for testing all tests so when any test crashes it is possible to report this and respawn child process to continue testing other tests. |
On Tue, Sep 3, 2013 at 2:32 AM, Serhiy Storchaka <report@bugs.python.org>wrote:
Well, if we go *that* way, my initial proposal would be to just always run Your proposal complicates the flow significantly, IMHO. I'd just run each |
I believe that bpo-9517 may be relevant if we decide to take the subprocess route. |
I created the issue bpo-25285: "regrtest: run tests in subprocesses with -j1 on buildbots". |
Since 2013, the Python test suite evolved a lot. First, we fixed hundreds of bugs where tests leaked various kinds of resources: processes, threads, file descriptors, memory, etc. Second, python3 -m test -j0 now runs each test file in a subprocess. Maybe we should emit a warning when tests are run sequentially, without -j0. Maybe it should be an opt-in option, rather than the default. But I prefer to close this old issue. Please open a new issue if you want to make further changes in regrtest. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: