some test_idle tests are not re-runnable, producing false failures with regrtest -w option #71901
Comments
|
If you run test_idle using the standard Python regression test runner, regrtest, and use regrtest's -w option (to re-run failure test verbosely) *without* using regrtest's -j option (to run tests in separate processes), any real test failure triggering a rerun will cause a number of additional false IDLE test case failures because some test cases modify global state in such a way as to be not re-runable. An example: at the moment there is a real IDLE test case failure (see bpo-27830). If I run test_idle with both -w and -j options: $ /usr/local/bin/python3.6 -m test -w -j2 -uall test_idle
Run tests in parallel using 2 child processes
0:00:01 [1/1/1] test_idle failed
can't invoke "event" command: application has been destroyed
while executing
"event generate $w <<ThemeChanged>>"
(procedure "ttk::ThemeChanged" line 6)
invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')First differing element 1: ('__test__',
+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py') ====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')First differing element 1: ('__test__',
+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py') ---------------------------------------------------------------------- FAILED (failures=1) only the 1 valid test case failure shows up. But if I run test_idle with just -w (no -j) options: $ /usr/local/bin/python3.6 -m test -w -uall test_idle
Run tests sequentially
0:00:00 [1/1] test_idle
can't invoke "event" command: application has been destroyed
while executing
"event generate $w <<ThemeChanged>>"
(procedure "ttk::ThemeChanged" line 6)
invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')First differing element 1: ('__test__',
+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py') test_idle failed ====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
TV.transient.__init__()
NameError: name 'TV' is not defined====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
TV.transient.__init__()
NameError: name 'TV' is not defined====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
TV.transient.__init__()
NameError: name 'TV' is not defined====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 26, in tearDownModule
del TV
NameError: name 'TV' is not defined====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')First differing element 1: ('__test__',
+ '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py') ---------------------------------------------------------------------- FAILED (failures=1, errors=4) now there are 4 or 5 errors in addition to the 1 failure. The fifth false error is somewhat timing dependent, e.g. it shows up some of the time while the other 4 always show up: ====================================================================== Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
macosx.setupApp(cls.root, None)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
overrideRootMenu(root, flist)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of rangeIt's not a critical problem but it can be very confusing if you have not run into the problem before and can cause lost time trying to track down the false errors. |
|
Mark, see questions about Mac above. Basically, do we still need to add 'file://'? The test_textview repeat errors are eliminated by removing 'del TV' in tearDownModule. I added it in what seems to be an excess of caution when chasing down tk warnings about trying to do things after app destroyed. Re-reading the code, deleting it or not seems like it should have no effect and deleting it now has none that I can see on Windows. If either of you test deletion and get no new warnings on Mac, I will patch the deletion away. Macosx assumes that it is run once on IDLE startup. It is not intended to be idempotent. We could catch the IndexError and ignore it, but that could mask a real error arising from modifying the initial menu structure. We could catch the error and add a note to the message: "Ignore this if this only appears when regrtest re-runs test_idle after a failure." Another approach would be to break the triple indexing into three statements, so we would know which indexing fails. (I suspect the first.) query and test_query are 3.6 only. test_textview is all 3 versions. I presume the macosx issue is also. |
|
Mark, the 'question above' was actually posted to bpo-27380, the original query issue. |
1 similar comment
|
Mark, the 'question above' was actually posted to bpo-27380, the original query issue. |
|
For test_textview, I triggered the symptom -- 4 'TV' NameErrors on re-run, by (temporarily) adding 1/0 to tearDownModule and running with -w. After removing the TV deletion, only the ZeroDivisionError remained. This experiment also resulted in a '''can't invoke "event" command: application has been destroyed''' message. For me it only happens on the re-run, not in the first run. In verbose mode, it follows the one test in test_config_key. Converting the test to 'pass' proved it to be the source. Looking at the code, I could not find any obvious callback that would be left behind. Adding 'root.update() stops the message. (Adding root.update_idletasks() did not.) In the output quoted above, the same message is shown for the non-verbose initial run, If from the same cause, the same fix might work on Mac also. |
|
New changeset b10a312f6d00 by Terry Jan Reedy in branch 'default': |
|
Terry, b10a312f6d00 appears to fix most of the issue, that is, if I insert a deliberate failure into another IDLE test and use -w without -j, the four repeatable errors I identified above no longer occur. Thanks. However, the fifth, timing-dependent error still occurs during the -w rerun perhaps ever 5th run or so. It might be worth pursuing to help squash other intermittent results: ====================================================================== Traceback (most recent call last):
File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
macosx.setupApp(cls.root, None)
File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
overrideRootMenu(root, flist)
File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of range |
|
New changeset dc31fa499359 by Terry Jan Reedy in branch '3.5': |
|
I agree. The issue may come up with other tests in the future. I just decided to do the part I knew how to do first ;-). For bpo-18409, I added the mac call to Phil's original patch with the vague comment "now needed to run without raising". But it is not needed on Windows. I suspect that the then new IDLE GSOC student, S.H., had emailed me after reviewing existing work. He mostly worked on Linux, but I believe he had access to a Mac. setupApp calls 4 fixup functions. I am pretty sure that that the test itself does not need the failing menu fix overrideRootMenu, at least not now. (The case might be different if I ever add live EditorWindow tests.) Three days after the initial patch, the patch by S.H. for bpo-21682 replaced EditorWindow with a mock. The reason was to stop menu leaks. But it should have also eliminated any need to patch the EditorWindow menu. Two of the functions, addOpenEventSupport and hideTkConsole don't seem like they should ever be needed. The last, should only matter for tests that generate the events involved (and only on 8.5). So what happens if you comment out line 34? If deleting the call is not possible, maybe we can replace it with something more specific. If not, then the following should work, but at the cost of masking a possible IDLE bug (unless a new _utest=False parameter is added to the signature -- but I dislike adding such complications until really needed.) first_time = True # new
def setupApp(root, flist):
""
if isAquaTk() and first_time: # modified
first_time = False # new
... |
|
New changeset 0de509a79181 by Terry Jan Reedy in branch '2.7': |
|
The issue with test_textview in 3.x is that tearDownModule deleted a module attribute, 'TV', that was not created in setUpModule. For 2.7, 'TV' is a class attribute that is created in setUpClass and deleted in tearDownClass, so it does not have the same problem. On the other hand, 'TV' is the original textView.TextViewer class, as modified with mocks in setUpClass. The modifications are not reverted in tearDownClass. This is the original code, later modified in 3.x. This is safe both for re-runs and likely future tests, but this is not immediately obvious. In any case, it violates the idea/rule that tests should not make permanent changes. So I backported the local subclass of TextViewer used in 3.x. |
That seems to eliminate the intermittent "IndexError: list assignment index out of range" and doesn't seem to cause any new errors. |
|
I just noticed another similar test dependency, this time between test_idle and test_tix. If run with -j set, the problem is not seen. This is still present in 3.6.0a4: $ python3.6 -m test -uall -w test_idle test_tix
Run tests sequentially
0:00:00 [1/2] test_idle
can't invoke "event" command: application has been destroyed
while executing
"event generate $w <<ThemeChanged>>"
(procedure "ttk::ThemeChanged" line 6)
invoked from within
"ttk::ThemeChanged"
0:00:02 [2/2] test_tix
test test_tix crashed -- Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/libregrtest/runtest.py", line 167, in runtest_inner
the_module = importlib.import_module(abstest)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 996, in _gcd_import
File "<frozen importlib._bootstrap>", line 979, in _find_and_load
File "<frozen importlib._bootstrap>", line 968, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 667, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/test_tix.py", line 11, in <module>
from tkinter import tix, TclError
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/tkinter/tix.py", line 30, in <module>
from tkinter import _cnfmerge, _default_root
ImportError: cannot import name '_default_root'
test_tix failed
1 test OK.
1 test failed:
test_tix
Re-running failed tests in verbose mode
Re-running test 'test_tix' in verbose mode
test test_tix crashed -- Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/libregrtest/runtest.py", line 167, in runtest_inner
the_module = importlib.import_module(abstest)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 996, in _gcd_import
File "<frozen importlib._bootstrap>", line 979, in _find_and_load
File "<frozen importlib._bootstrap>", line 968, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 667, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/test_tix.py", line 11, in <module>
from tkinter import tix, TclError
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/tkinter/tix.py", line 30, in <module>
from tkinter import _cnfmerge, _default_root
ImportError: cannot import name '_default_root'1 test failed again: |
|
Ned, test_tix failing on some systems (but not Windows) after test_idle, because test_idle calls tk.NoDefaultRoot() (added June 21 in bpo-24137) is bpo-27611. Serhiy says there is a bug in tix, but no one has fix that yet. Zach says I should restore tkinter module. I will probably revert changing it in the first place except when running test_idle alone as main. |
|
"Ned, test_tix failing on some systems [...]" Sorry, Terry! I had forgotten about bpo-27611 and didn't do a proper search. |
|
I fixed bpo-27611. Could you try the comment out test of macosx call for test_autocomplete (around line 30 to 35) in 2.7 and 3.5? If those also pass, I will patch all three and close this. |
|
"Could you try the comment out test of macosx call for test_autocomplete (around line 30 to 35) in 2.7 and 3.5?" Commenting out the mac.setupApp call in the test setup class did not appear to affect the running of the tests for either 2.7 or 3.5. |
|
New changeset 1c4cb01723a0 by Terry Jan Reedy in branch '2.7': New changeset 0e4bea7c8c00 by Terry Jan Reedy in branch '3.5': New changeset e5db20ad0328 by Terry Jan Reedy in branch 'default': |
|
Terry, in revision 6c8dd4cb4ee7 you wrote “Fix warning tom test_config.” Perhaps this is meant to say “. . . from test_config_key” (taken from b10a312f6d00)? |
|
Fixed twice in all 3 versions. Thanks for the heads-up. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: