Author ned.deily
Recipients ned.deily, terry.reedy
Date 2016-08-08.23:14:56
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1470698096.85.0.204645941326.issue27714@psf.upfronthosting.co.za>
In-reply-to
Content
If you run test_idle using the standard Python regression test runner, regrtest, and use regrtest's -w option (to re-run failure test verbosely) *without* using regrtest's -j option (to run tests in separate processes), any real test failure triggering a rerun will cause a number of additional false IDLE test case failures because some test cases modify global state in such a way as to be not re-runable.

An example: at the moment there is a real IDLE test case failure (see Issue27830).  If I run test_idle with both -w and -j options:

$ /usr/local/bin/python3.6 -m test -w -j2 -uall test_idle
Run tests in parallel using 2 child processes
0:00:01 [1/1/1] test_idle failed
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
1 test failed:
    test_idle
Re-running failed tests in verbose mode
Re-running test 'test_idle' in verbose mode
test_autocomplete_event (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
test_delayed_open_completions (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
[...]
test_shell_show (idlelib.idle_test.test_warning.ShellWarnTest) ... ok
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok

======================================================================
FAIL: test_click_help_source (idlelib.idle_test.test_query.HelpsourceGuiTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

----------------------------------------------------------------------
Ran 219 tests in 0.977s

FAILED (failures=1)

only the 1 valid test case failure shows up.

But if I run test_idle with just -w (no -j) options:

$ /usr/local/bin/python3.6 -m test -w -uall test_idle
Run tests sequentially
0:00:00 [1/1] test_idle
can't invoke "event" command:  application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"
test test_idle failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

test_idle failed
1 test failed:
    test_idle
Re-running failed tests in verbose mode
Re-running test 'test_idle' in verbose mode
test_autocomplete_event (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
test_delayed_open_completions (idlelib.idle_test.test_autocomplete.AutoCompleteTest) ... ok
[...]
test_no_delete (idlelib.idle_test.test_text.TkTextTest) ... ok
test_init_modal (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_init_nonmodal (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_ok (idlelib.idle_test.test_textview.TextViewTest) ... ERROR
test_view_file (idlelib.idle_test.test_textview.ViewFunctionTest) ... ok
test_view_text (idlelib.idle_test.test_textview.ViewFunctionTest) ... ok
ERROR
test_init (idlelib.idle_test.test_tree.TreeTest) ... ok
[...]
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok

======================================================================
ERROR: test_init_modal (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: test_init_nonmodal (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: test_ok (idlelib.idle_test.test_textview.TextViewTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 40, in setUp
    TV.transient.__init__()
NameError: name 'TV' is not defined

======================================================================
ERROR: tearDownModule (idlelib.idle_test.test_textview)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_textview.py", line 26, in tearDownModule
    del TV
NameError: name 'TV' is not defined

======================================================================
FAIL: test_click_help_source (idlelib.idle_test.test_query.HelpsourceGuiTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py", line 389, in test_click_help_source
    Equal(dialog.result, ('__test__', __file__))
AssertionError: Tuples differ: ('__test__', 'file:///Library/Frameworks/Python.framewo[58 chars].py') != ('__test__', '/Library/Frameworks/Python.framework/Vers[51 chars].py')

First differing element 1:
'file:///Library/Frameworks/Python.framewo[57 chars]y.py'
'/Library/Frameworks/Python.framework/Vers[50 chars]y.py'

  ('__test__',
-  'file:///Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')
?   -------

+  '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_query.py')

----------------------------------------------------------------------
Ran 219 tests in 0.998s

FAILED (failures=1, errors=4)

now there are 4 or 5 errors in addition to the 1 failure.

The fifth false error is somewhat timing dependent, e.g. it shows up some of the time while the other 4 always show up:

======================================================================
ERROR: setUpClass (idlelib.idle_test.test_autocomplete.AutoCompleteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/idle_test/test_autocomplete.py", line 34, in setUpClass
    macosx.setupApp(cls.root, None)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 245, in setupApp
    overrideRootMenu(root, flist)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/idlelib/macosx.py", line 215, in overrideRootMenu
    del mainmenu.menudefs[-1][1][0]
IndexError: list assignment index out of range


It's not a critical problem but it can be very confusing if you have not run into the problem before and can cause lost time trying to track down the false errors.
History
Date User Action Args
2016-08-08 23:14:56ned.deilysetrecipients: + ned.deily, terry.reedy
2016-08-08 23:14:56ned.deilysetmessageid: <1470698096.85.0.204645941326.issue27714@psf.upfronthosting.co.za>
2016-08-08 23:14:56ned.deilylinkissue27714 messages
2016-08-08 23:14:56ned.deilycreate