classification
Title: some test_idle tests are not re-runnable, producing false failures with regrtest -w option
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: markroseman, martin.panter, ned.deily, python-dev, terry.reedy
Priority: normal Keywords:

Created on 2016-08-08 23:14 by ned.deily, last changed 2016-08-21 04:21 by terry.reedy. This issue is now closed.

Messages (20)
msg272200 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-08 23:14
If you run test_idle using the standard Python regression test runner, regrtest, and use regrtest's -w option (to re-run failure test verbosely) *without* using regrtest's -j option (to run tests in separate processes), any real test failure triggering a rerun will cause a number of additional false IDLE test case failures because some test cases modify global state in such a way as to be not re-runable.

An example: at the moment there is a real IDLE test case failure (see Issue27830).  If I run test_idle with both -w and -j options:

$ /usr/local/bin/python3.6 -m test -w -j2 -uall test_idle
Run tests in parallel using 2 child processes
0:00:01 [1/1/1] test_idle failed
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
1 test failed:
    test_idle
Re-running failed tests in verbose mode
Re-running test 'test_idle' in verbose mode
test_autocomplete_event (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
test_delayed_open_completions (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
[...]
test_shell_show (idlelib.idle_test.test_warning.ShellWarnTest) ... ok
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok

======================================================================
FAIL: test_click_help_source (idlelib.idle_test.test_query.HelpsourceGuiTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

----------------------------------------------------------------------
Ran 219 tests in 0.977s

FAILED (failures=1)

only the 1 valid test case failure shows up.

But if I run test_idle with just -w (no -j) options:

$ /usr/local/bin/python3.6 -m test -w -uall test_idle
Run tests sequentially
0:00:00 [1/1] test_idle
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

test_idle failed
1 test failed:
    test_idle
Re-running failed tests in verbose mode
Re-running test 'test_idle' in verbose mode
test_autocomplete_event (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
test_delayed_open_completions (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
[...]
test_no_delete (idlelib.idle_test.test_text.TkTextTest) ... ok
test_init_modal (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_init_nonmodal (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_ok (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_view_file (idlelib.idle_test.test_textview.ViewFunctionTest) ... ok
test_view_text (idlelib.idle_test.test_textview.ViewFunctionTest) ... ok
ERROR
test_init (idlelib.idle_test.test_tree.TreeTest) ... ok
[...]
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok

======================================================================
ERROR: test_init_modal (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: test_init_nonmodal (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: test_ok (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: tearDownModule (idlelib.idle_test.test_textview)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 26, in tearDownModule
    del TV
NameError: name 'TV' is not defined

======================================================================
FAIL: test_click_help_source (idlelib.idle_test.test_query.HelpsourceGuiTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

----------------------------------------------------------------------
Ran 219 tests in 0.998s

FAILED (failures=1, errors=4)

now there are 4 or 5 errors in addition to the 1 failure.

The fifth false error is somewhat timing dependent, e.g. it shows up some of the time while the other 4 always show up:

======================================================================
ERROR: setUpClass (idlelib.idle_test.test_autocomplete.AutoCompleteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
    macosx.setupApp(cls.root, None)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
    overrideRootMenu(root, flist)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
    del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of range


It's not a critical problem but it can be very confusing if you have not run into the problem before and can cause lost time trying to track down the false errors.
msg272219 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-09 07:14
Mark, see questions about Mac above.  Basically, do we still need to add 'file://'?

The test_textview repeat errors are eliminated by removing 'del TV' in tearDownModule.  I added it in what seems to be an excess of caution when chasing down tk warnings about trying to do things after app destroyed.  Re-reading the code, deleting it or not seems like it should have no effect and deleting it now has none that I can see on Windows.  If either of you test deletion and get no new warnings on Mac, I will patch the deletion away.

Macosx assumes that it is run once on IDLE startup.  It is not intended to be idempotent.  We could catch the IndexError and ignore it, but that could mask a real error arising from modifying the initial menu structure.  We could catch the error and add a note to the message: "Ignore this if this only appears when regrtest re-runs test_idle after a failure."  Another approach would be to break the triple indexing into three statements, so we would know which indexing fails.  (I suspect the first.)

query and test_query are 3.6 only.  test_textview is all 3 versions.  I presume the macosx issue is also.
msg272222 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-09 07:20
Mark, the 'question above' was actually posted to #27380, the original query issue.
msg272349 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-10 17:31
Mark, the 'question above' was actually posted to #27380, the original query issue.
msg272361 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-10 19:14
For test_textview, I triggered the symptom -- 4 'TV' NameErrors on re-run, by (temporarily) adding 1/0 to tearDownModule and running with -w.  After removing the TV deletion, only the ZeroDivisionError remained.

This experiment also resulted in a '''can't invoke "event" command:  application has been destroyed''' message.  For me it only happens on the re-run, not in the first run.  In verbose mode, it follows the one test in test_config_key.  Converting the test to 'pass' proved it to be the source.  Looking at the code, I could not find any obvious callback that would be left behind.  Adding 'root.update() stops the message.  (Adding root.update_idletasks() did not.)

In the output quoted above, the same message is shown for the non-verbose initial run,  If from the same cause, the same fix might work on Mac also.
msg272362 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-10 19:15
New changeset b10a312f6d00 by Terry Jan Reedy in branch 'default':
Issue #27714: text_textview now passes when re-run in the same process
https://hg.python.org/cpython/rev/b10a312f6d00
msg272369 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-10 19:51
Terry, b10a312f6d00 appears to fix most of the issue, that is, if I insert a deliberate failure into another IDLE test and use -w without -j, the four repeatable errors I identified above no longer occur.  Thanks.  However, the fifth, timing-dependent error still occurs during the -w rerun perhaps ever 5th run or so.  It might be worth pursuing to help squash other intermittent results:

======================================================================
ERROR: setUpClass (idlelib.idle_test.test_autocomplete.AutoCompleteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
    macosx.setupApp(cls.root, None)
  File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
    overrideRootMenu(root, flist)
  File "/py/dev/3x/root/fwd_macports/Library/Frameworks/pytest_10.12.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
    del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of range
msg272378 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-10 20:54
New changeset dc31fa499359 by Terry Jan Reedy in branch '3.5':
Issue #27714: text_textview now passes when re-run in the same process
https://hg.python.org/cpython/rev/dc31fa499359
msg272379 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-10 22:18
I agree.  The issue may come up with other tests in the future.  I just decided to do the part I knew how to do first ;-).

For #18409, I added the mac call to Phil's original patch with the vague comment "now needed to run without raising".  But it is not needed on Windows.  I suspect that the then new IDLE GSOC student, S.H., had emailed me after reviewing existing work.  He mostly worked on Linux, but I believe he had access to a Mac.

setupApp calls 4 fixup functions.  I am pretty sure that that the test itself does not need the failing menu fix overrideRootMenu, at least not now. (The case might be different if I ever add live EditorWindow tests.) Three days after the initial patch, the patch by S.H. for #21682 replaced EditorWindow with a mock.  The reason was to stop menu leaks.  But it should have also eliminated any need to patch the EditorWindow menu.

Two of the functions, addOpenEventSupport and hideTkConsole don't seem like they should ever be needed.  The last, should only matter for tests that generate the events involved (and only on 8.5).
 
So what happens if you comment out line 34?
  macosx.setupApp(cls.root, None)

If deleting the call is not possible, maybe we can replace it with something more specific.  If not, then the following should work, but at the cost of masking a possible IDLE bug (unless a new _utest=False parameter is added to the signature -- but I dislike adding such complications until really needed.)

first_time = True  # new
def setupApp(root, flist):
    ""
    if isAquaTk() and first_time:  # modified
        first_time = False  # new
        ...
msg272387 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-10 23:41
New changeset 0de509a79181 by Terry Jan Reedy in branch '2.7':
Issue #27714: For IDLE's test_textview, backport 3.x subclass with mocks
https://hg.python.org/cpython/rev/0de509a79181
msg272388 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-10 23:42
The issue with test_textview in 3.x is that tearDownModule deleted a module attribute, 'TV', that was not created in setUpModule.  For 2.7, 'TV' is a class attribute that is created in setUpClass and deleted in tearDownClass, so it does not have the same problem.  On the other hand, 'TV' is the original textView.TextViewer class, as modified with mocks in setUpClass.  The modifications are not reverted in tearDownClass.  This is the original code, later modified in 3.x.  This is safe both for re-runs and likely future tests, but this is not immediately obvious.  In any case, it violates the idea/rule that tests should not make permanent changes.  So I backported the local subclass of TextViewer used in 3.x.
msg272567 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-12 21:48
>So what happens if you comment out line 34?
>  macosx.setupApp(cls.root, None)

That seems to eliminate the intermittent "IndexError: list assignment index out of range" and doesn't seem to cause any new errors.
msg272805 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-15 22:25
I just noticed another similar test dependency, this time between test_idle and test_tix.  If run with -j set, the problem is not seen.  This is still present in 3.6.0a4:

$ python3.6 -m test -uall -w test_idle test_tix
Run tests sequentially
0:00:00 [1/2] test_idle
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
0:00:02 [2/2] test_tix
test test_tix crashed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/libregrtest/runtest.py", line 167, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 996, in _gcd_import
  File "<frozen importlib._bootstrap>", line 979, in _find_and_load
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 667, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/test_tix.py", line 11, in <module>
    from tkinter import tix, TclError
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/tkinter/tix.py", line 30, in <module>
    from tkinter import _cnfmerge, _default_root
ImportError: cannot import name '_default_root'

test_tix failed
1 test OK.
1 test failed:
    test_tix
Re-running failed tests in verbose mode
Re-running test 'test_tix' in verbose mode
test test_tix crashed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/libregrtest/runtest.py", line 167, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 996, in _gcd_import
  File "<frozen importlib._bootstrap>", line 979, in _find_and_load
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 667, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/test/test_tix.py", line 11, in <module>
    from tkinter import tix, TclError
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/tkinter/tix.py", line 30, in <module>
    from tkinter import _cnfmerge, _default_root
ImportError: cannot import name '_default_root'

1 test failed again:
    test_tix
Total duration: 0:00:03
msg272816 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-16 01:54
Ned, test_tix failing on some systems (but not Windows) after test_idle, because test_idle calls tk.NoDefaultRoot() (added June 21 in #24137) is #27611.  Serhiy says there is a bug in tix, but no one has fix that yet.  Zach says I should restore tkinter module.  I will probably revert changing it in the first place except when running test_idle alone as main.
msg272819 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-16 02:11
"Ned, test_tix failing on some systems [...]"

Sorry, Terry! I had forgotten about Issue27611 and didn't do a proper search.
msg272840 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-16 06:00
I fixed #27611.  Could you try the comment out test of macosx call for test_autocomplete (around line 30 to 35) in 2.7 and 3.5?  If those also pass, I will patch all three and close this.
msg273010 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-18 02:26
"Could you try the comment out test of macosx call for test_autocomplete (around line 30 to 35) in 2.7 and 3.5?"

Commenting out the mac.setupApp call in the test setup class did not appear to affect the running of the tests for either 2.7 or 3.5.
msg273257 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-21 01:09
New changeset 1c4cb01723a0 by Terry Jan Reedy in branch '2.7':
Issue #27714: Remove unneeded non-idempotent call that fails on retest.
https://hg.python.org/cpython/rev/1c4cb01723a0

New changeset 0e4bea7c8c00 by Terry Jan Reedy in branch '3.5':
Issue #27714: Remove unneeded non-idempotent call that fails on retest.
https://hg.python.org/cpython/rev/0e4bea7c8c00

New changeset e5db20ad0328 by Terry Jan Reedy in branch 'default':
Issue #27714: Remove unneeded non-idempotent call that fails on retest.
https://hg.python.org/cpython/rev/e5db20ad0328
msg273263 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-21 03:20
Terry, in revision 6c8dd4cb4ee7 you wrote “Fix warning tom test_config.” Perhaps this is meant to say “. . . from test_config_key” (taken from b10a312f6d00)?
msg273264 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-08-21 04:21
Fixed twice in all 3 versions.  Thanks for the heads-up.
History
Date User Action Args
2016-08-21 04:21:40terry.reedysetmessages: + msg273264
2016-08-21 03:20:17martin.pantersetnosy: + martin.panter
messages: + msg273263
2016-08-21 01:10:27terry.reedysetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2016-08-21 01:09:13python-devsetmessages: + msg273257
2016-08-18 02:26:32ned.deilysetmessages: + msg273010
2016-08-16 06:00:48terry.reedysetmessages: + msg272840
2016-08-16 02:11:07ned.deilysetmessages: + msg272819
2016-08-16 01:54:27terry.reedysetmessages: + msg272816
2016-08-15 22:25:51ned.deilysetmessages: + msg272805
2016-08-12 21:48:28ned.deilysetmessages: + msg272567
2016-08-10 23:42:51terry.reedysetmessages: + msg272388
2016-08-10 23:41:56python-devsetmessages: + msg272387
2016-08-10 22:18:55terry.reedysetpriority: low -> normal
versions: + Python 2.7
messages: + msg272379

type: behavior
stage: needs patch
2016-08-10 20:54:51python-devsetmessages: + msg272378
2016-08-10 19:51:20ned.deilysetmessages: + msg272369
2016-08-10 19:15:43python-devsetnosy: + python-dev
messages: + msg272362
2016-08-10 19:14:53terry.reedysetmessages: + msg272361
2016-08-10 17:31:40terry.reedysetmessages: + msg272349
2016-08-09 07:20:57terry.reedysetmessages: + msg272222
2016-08-09 07:14:39terry.reedysetnosy: + markroseman
messages: + msg272219
2016-08-08 23:14:56ned.deilycreate