New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Windows: Running the Python test suite sequentially is interrupted by (Pdb) prompt #88121
Comments
Recent test failures have been noticed on Windows 10. The full test log has been attached. The tests test_distutils, test_pdb, test_signal and test_threading require a little bit of attention since there seems to be some kind of error. |
I wonder if these four tests do something that affects all threads? The test driver module runs tests in parallel (if you ask it to run all or most tests, like the default). |
Pdb error source - Line 1270 in 9aea31d
This is where the error in Pdb is originating from. |
The test expects the <built-in default_int_handler> but gets sigint_handler method of a pdb.Pdb instance instead. |
Can you submit a PR that fixes it? |
I expect parallelism is a red herring: early in the test output attached to this report: 0:00:04 Run tests sequentially and there's no other evidence in the output that multiple tests are running simultaneously. Also on Win10, the 4 failing tests here pass for me if I run them one at a time, so it's not obvious. I _suspect_ that what's going wrong with test_pdb is the root cause: every now & again, for some weeks now, when I try to run tests on Windows I come back to the cmd.exe window and see that it's just sitting there, waiting at a pdb prompt. In the output attached to this report, note that after test_threading starts, (Pdb) continue appears out of the blue. But test_pdb is long finished by that time. But I'm clueless about current pdb internals. |
Yes you're right. Running the failed tests individually does not trigger the errors. And the (Pdb) continue actually is manually entered by me. My tests also stop at that prompt but somehow entering these commands again allow us to continue. Why are the errors triggering at all? The tests that are failing when ran alongside the test suite haven't been messed with for a very long time now and still somehow out of the blue they are failing. Any thoughts? |
Yes, I also see that pdb prompt. I recommend looking at what different |
File "C:\github\cpython\lib\test\test_distutils.py", line 15, in <module> Ah, test_distutils fails if distutils was already imported previously. The test should be fixed. """
test test_pdb failed -- Traceback (most recent call last):
File "C:\github\cpython\lib\doctest.py", line 2205, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_pdb.test_pdb_issue_20766
File "C:\github\cpython\lib\test\test_pdb.py", line 1270, in test_pdb_issue_20766
""" This is a surprising failure. No clue why test_pdb fails when other tests are run previously (when tests are run sequentially). See also bpo-43960 |
Shreyan Avigyan:
Do you mean that you modified the Python source code? Does the issue go away if you revert your change or if you test a newly installed Python? |
I think the test failures cascade, and some state is not being cleaned up. I'm trying to work through them now, but if someone else is as well, do say so and I'll find something else to do. |
Shreyan Avigyan:
Victor Stinner: Me:
For me, I was using Win10 x64 CPython built from yesterday's github master/main. And thanks for the distutils clue! I bet that one is a distinct issue. To make it more confusing, all 4 tests passed when I ran with |
bpo-37387 "test_compileall fails randomly on Windows when tests are run in parallel" about random PermissionError. |
For test_logging test_namer_rotator_inheritance() error: I created bpo-43961 "[Windows] test_logging.test_namer_rotator_inheritance() logs a logging error". It doesn't like like an error and it only happens on Windows. |
Oh. I also got this surprising (Pdb) prompt when running "python -m test" on Windows. The prompt doesn't display any output, I only get the prompt display (...) *But* I managed to get more info about that issue: (Pdb) f=open("debug.txt", "w") C:\vstinner\python\master\build\test_python_6260æ\debug.txt content: (...) Using debug.txt, I also see that in test_signal, sys.gettrace() = <bound method Bdb.trace_dispatch of <pdb.Pdb object at 0x0000029D85AB3570>>. Lib/test/libregrtest/save_env.py is supposed to mark a test as "env changed" (ENV_CHANGED) if sys.gettrace() is changed between two tests. |
Doesn't the pdb prompt use the builtin input()? |
See also bpo-43963: "test_interpreters has side effects on test_signal". |
Running these 3 tests is enough to reproduce the issue: vstinner@DESKTOP-DK7VBIL C:\vstinner\python\master>python -m test test_interpreters test_pdb test_threading
Running Debug|x64 interpreter...
0:00:00 Run tests sequentially
0:00:00 [1/3] test_interpreters
0:00:04 [2/3] test_pdb
test test_pdb failed -- Traceback (most recent call last):
File "C:\vstinner\python\master\lib\doctest.py", line 2205, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_pdb.test_pdb_issue_20766
File "C:\vstinner\python\master\lib\test\test_pdb.py", line 1270, in test_pdb_issue_20766
(...)
0:00:10 load avg: 0.03 [3/3/1] test_threading -- test_pdb failed
(Pdb) |
FYI I used the following patch to debug this issue: diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py def _callTestMethod(self, method):
+ if sys.gettrace() is not None:
+ raise Exception("sys.gettrace() is not None")
method()
def _callTearDown(self): And I used the following script: import random
import os.path
import sys
with open("tests") as fp:
lines = list(filter(bool, (line.strip() for line in fp)))
print(len(lines))
filename = "test"
loop = 2
while os.path.exists(filename):
filename = "test%s" % loop
loop += 1
with open(filename, "w") as fp:
for _ in range(5):
test = random.choice(lines)
print(test, file=fp)
print("test_signal", file=fp)
with open(filename) as fp:
for line in fp:
print(line.rstrip())
args = [sys.executable, "-m", "test", "--fromfile=" + filename]
print("RUN", args)
os.execv(args[0], args) I created the "tests" file using the command: python -m test --list-tests > tests Then I removed all tests starting at test_signal, including test_signal. I removed slow tests like asyncio and multiprocessing tests. |
Another issue with the tests is that it has become terribly slow. It's taking about 45 minutes to complete 1 run of the test suite. A few days ago, it took only 10 - 15 minutes. And moreover I think test_threading is also modifying sys.gettrace. The logs have the full error message. --Snippet-- 0:33:17 load avg: 0.80 [307/426/2] test_signal
(Pdb) continue
Warning -- sys.gettrace was modified by test_signal
Before: None
After: <bound method Bdb.trace_dispatch of <pdb.Pdb object at 0x0000021A5D2DCD70>>
test test_signal failed -- Traceback (most recent call last):
File "C:\github\cpython\lib\test\test_signal.py", line 1345, in test_sigint
signal.raise_signal(signal.SIGINT)
AssertionError: KeyboardInterrupt not raised --Snippet-- 0:38:22 load avg: 0.04 [353/426/3] test_threading --Snippet-- |
Shreyan Avigyan: please try an up to date version of the master branch, I fixed the Pdb issue with commit a09766d. |
Same errors. (Recompilation was done 1 or 2 hours ago against the then updated master branch. It may not include the latest commit) |
FYI test_signal is no longer modifying sys.gettrace thanks to your fix. But test_threading still does. Do you have a fix for that? |
I ran the test again and test_signal still modifies sys.gettrace() |
All the test failures are side effects of other tests because when they are ran individually they do not trigger errors. Running sequentially causes the error. |
I ran "python -m test" on an up to date master branch (commit fe52eb6), there is a single remaining issue. test_distutils failed because the deprecation warning was already emited: issue fixed by PR 25675. The other issues are gone. Shreyan Avigyan: "Same errors." IMO it's a mistake on your side. You ran an outdated version or you didn't rebuild Python cleanly. "FYI test_signal is no longer modifying sys.gettrace thanks to your fix. But test_threading still does. (...) I ran the test again and test_signal still modifies sys.gettrace()" It's not my case on my machine, again it sounds like an issue on your side. |
It took 49 minutes on my Windows 10 VM. Are you sure that you ran the test suite sequentially when it was slower? I advice you run using -j0 option to run tests in parallel: it's much faster, and it's also more reliable. == Tests result: FAILURE == 384 tests OK. 1 test failed: 41 tests skipped: Total duration: 49 min 12 sec |
Thanks everyone for fixing these issues and thanks Shreyan for the bug report. The main initial is now fixed, so I close the issue. Let's continue the discussions in remaining open issues: |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: