classification
Title: IDLE: test_browser is failed when run twice
Type: behavior Stage: resolved
Components: IDLE, Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: serhiy.storchaka, terry.reedy
Priority: high Keywords: patch

Created on 2017-09-23 12:06 by serhiy.storchaka, last changed 2017-09-23 19:24 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3708 merged terry.reedy, 2017-09-23 17:59
PR 3709 merged python-dev, 2017-09-23 18:19
Messages (5)
msg302786 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-23 12:06
Buildbots failures:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/101/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.x/builds/116/steps/test/logs/stdio

I don't know why test_browser fails first time, but this provokes running it second time why always fails.

$ ./python -m test -vuall -m test_browser test_idle test_idle
...
======================================================================
FAIL: test_gettext (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 181, in test_gettext
    self.assertEqual(self.cbt_C1.GetText(), 'class C1')
AssertionError: 'class C1()' != 'class C1'
- class C1()
?         --
+ class C1


======================================================================
FAIL: test_init (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 175, in test_init
    eq(self.cbt_C1.name, 'C1')
AssertionError: 'C1()' != 'C1'
- C1()
?   --
+ C1


======================================================================
FAIL: test_getsublist (idlelib.idle_test.test_browser.ModuleBrowserTreeItemTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 141, in test_getsublist
    self.assertEqual(sub1.name, 'C0')
AssertionError: 'C0(base)' != 'C0'
- C0(base)
+ C0


======================================================================
FAIL: test_nested (idlelib.idle_test.test_browser.NestedChildrenTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.6/Lib/idlelib/idle_test/test_browser.py", line 238, in test_nested
    self.assertEqual(actual_names, expected_names)
AssertionError: Lists differ: ['f0', 'C0(base)', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3'] != ['f0', 'C0', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3']

First differing element 1:
'C0(base)'
'C0'

- ['f0', 'C0(base)', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3']
?           ------

+ ['f0', 'C0', 'f1', 'c1', 'F1', 'C1()', 'f2', 'C2', 'F3']

----------------------------------------------------------------------

The test is passed if run it once on my computer.
msg302794 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-09-23 17:52
browser.transform_children, when called the first time on a dict of children nodes, rebinds the name attribute of Classes with bases*. Tests that called the real browser.transform_children the first time on a node mutate the global mock_pyclbr_tree.  When the tests are re-run in the same process, tests initially run before the mutation are also run after the mutation.

*This is not an issue in current real use.  IDLE gets a new tree each time a module browser is opened.  The current tree widget calls GetSublist and hence transform_children at most once per node as it caches the result for each node.  But don't know what ttk.Treeview does and do know that tests can make multiple calls.  

Restoring the tree with
def tearDownModule():
    C0.name = 'C0'
    C1.name = 'C1'
fixes the failures found when rerunning the test module in the same process, but not the underlying problem of order dependence.

Instead, I added immediate calls to transform_children so that the mock tree is never mutated thereafter by any of the tests.  The subsequent class checks that the initial name replacements are correct.  All tests see the tree in the same state, so there should not be any order dependence left.

Testing global function calls in a testcase is slightly awkward, but the testcase follows immediately, and making the tree and its nodes global simplifies the tests that use the tree.

I checked that running test_pyclbr and test_idle twice each succeeds.
msg302795 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-09-23 18:07
> I don't know why test_browser fails first time,

On both buildbots, the initial rerun passes:
Ran 371 tests in 12.265s  OK (skipped=3) Gentoo
Ran 371 tests in 12.047s  OK (skipped=1) Windows8
msg302796 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-09-23 18:19
New changeset 99167f85b7373c8082b30a74211f009627bdedfa by Terry Jan Reedy in branch 'master':
bpo-31559: Remove test order dependence in idle_test.test_browser. (#3708)
https://github.com/python/cpython/commit/99167f85b7373c8082b30a74211f009627bdedfa
msg302800 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-09-23 19:24
New changeset 429b3b1188cfac654677e8aeed494bb6067c475f by Terry Jan Reedy (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31559: Remove test order dependence in idle_test.test_browser. (GH-3708) (#3709)
https://github.com/python/cpython/commit/429b3b1188cfac654677e8aeed494bb6067c475f
History
Date User Action Args
2017-09-23 19:24:53terry.reedysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-09-23 19:24:21terry.reedysetmessages: + msg302800
2017-09-23 18:19:43python-devsetstage: patch review
pull_requests: + pull_request3695
2017-09-23 18:19:25terry.reedysetmessages: + msg302796
2017-09-23 18:07:20terry.reedysetmessages: + msg302795
stage: patch review -> (no value)
2017-09-23 17:59:00terry.reedysetkeywords: + patch
stage: patch review
pull_requests: + pull_request3694
2017-09-23 17:52:59terry.reedysetmessages: + msg302794
2017-09-23 12:06:23serhiy.storchakacreate