classification
Title: Fix leaks in idlelib
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: louielu, terry.reedy
Priority: normal Keywords:

Created on 2017-06-13 05:17 by terry.reedy, last changed 2017-08-07 18:07 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2147 merged louielu, 2017-06-13 05:49
PR 2161 merged terry.reedy, 2017-06-13 12:51
PR 2163 merged terry.reedy, 2017-06-13 14:35
PR 2165 merged terry.reedy, 2017-06-13 14:56
Messages (8)
msg295846 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-13 05:17
When Louie Lu posted a link to
https://blog.louie.lu/2017/06/12/diagnosing-and-fixing-reference-leaks-in-cpython/
on core-mentorship list, I tested idlelib.
python -m test -ugui test_idle  # SUCCESS, no extraneous output
python -m test -R: test_idle  # SUCCESS, no extraneous output
python -m test -R: -ugui test_idle  # error output, FAILURE
[So people who leaktest without a screen see nothing in idlelib.]

Error output is about 20 copies of the following:
can't invoke "event" command: application has been destroyed
    while executing
"event generate $w <<ThemeChanged>>"
    (procedure "ttk::ThemeChanged" line 6)
    invoked from within
"ttk::ThemeChanged"

At the end:
test_idle leaked [471, 471, 471, 471] references, sum=1884
test_idle leaked [209, 211, 211, 211] memory blocks, sum=842
[similar for python 3.6]

In a response email, I noted that test_idle gathers tests from idlelib.idle_test.test_* and that something extra is needed to pin leaks to specific test modules.

I don't know whether the absence of 'invoke event' error messages when not running -R means that there are also no refleaks, or not.
---
import os
import subprocess

os.chdir('f:/dev/3x/Lib/idlelib/idle_test')
testfiles = [name for name in os.listdir() if name.startswith('test_')]
for name in testfiles:
    os.rename(name, 'x'+name)
for name in testfiles:
    os.rename('x'+name, name)
    try:
        res = subprocess.run(
            ['f:/dev/3x/python.bat', '-m', 'test', '-R:', '-ugui', 'test_idle'],
             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        if res.returncode:
            print(name)
            print(res.stderr)
    except Exception as err:
        print(name, err)
    os.rename(name, 'x'+name)
for name in testfiles:
    os.rename('x'+name, name)
---
reports
test_macosx.py
b'beginning 9 repetitions\r\n123456789\r\n\r\ntest_idle leaked [31, 31, 31, 31] references, sum=124\r\ntest_idle leaked [19, 21, 21, 21] memory blocks, sum=82\r\n'
test_query.py
b'beginning 9 repetitions\r\n123456789\r\n\r\ntest_idle leaked [429, 429, 429, 429] references, sum=1716\r\ntest_idle leaked [190, 192, 192, 192] memory blocks, sum=766\r\n'
There are no 'invoke event' messages.

For further testing within each file, by commenting out code, as suggested in the link above, I replaced 'testfiles' in the middle loop with ['testmacosx.py'] or ['test_query.py'].  For test_macosx, the culprit is class SetupTest.  For test_query, the culprit is class QueryGuiTest.  Adding cls.root.update_idletasks did not solve the problem by itself (as it has in other cases).  I plan to continue another time.
msg295852 - (view) Author: Louie Lu (louielu) * Date: 2017-06-13 06:11
test_query were fixed in PR 2147, which is leak by not removing mock.Mock() in dialog.
msg295911 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-13 12:29
New changeset b070fd275b68df5c5ba9f6f43197b8d7066f0b18 by terryjreedy (mlouielu) in branch 'master':
bpo-30642: IDLE: Fix test_query refleak (#2147)
https://github.com/python/cpython/commit/b070fd275b68df5c5ba9f6f43197b8d7066f0b18
msg295918 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-13 14:11
New changeset 2bfb45d447c445b3c3afc19d16b4cd4773975993 by terryjreedy in branch '3.6':
bpo-30642: IDLE: Fix test_query refleak (#2147) (#2161)
https://github.com/python/cpython/commit/2bfb45d447c445b3c3afc19d16b4cd4773975993
msg295923 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-13 14:44
The macosx leak came from repeated calls to
    root.createcommand("::tk::mac::OpenDocument", doOpenFile)
in macosx.addOpenEventSupport(root, flist), which is called from macosx.setupApp, which is called in test_macosx.SetupTest.test_setupapp.

Attached is an improved version of findleak.py, which made it easy to isolate one test file by renaming, repeatedly determine whether the leak remained after modifying either macosx.py or test_macosx.py, and revert all the other names when done.
msg295932 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-13 15:52
New changeset b0efd493b6af24a6ae744e7e02f4b69c70e88f3d by terryjreedy in branch '3.6':
[3.6]bpo-30642: Fix ref leak in idle_test.test_macosx (#2163) (#2165)
https://github.com/python/cpython/commit/b0efd493b6af24a6ae744e7e02f4b69c70e88f3d
msg295981 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-14 04:15
f:\dev\36>python -m test -R: -ugui test_idle
gives about 40 invoke event messages, but the test passes.  So the messages and leaks are not connected.
msg299856 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-08-07 18:07
Unlinked old findleak.py after uploading much improved version to #31130.
History
Date User Action Args
2017-08-07 18:07:39terry.reedysetmessages: + msg299856
2017-08-07 18:06:46terry.reedysetfiles: - findleak.py
2017-06-16 05:12:51terry.reedysetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2017-06-14 04:15:18terry.reedysetmessages: + msg295981
2017-06-13 15:52:12terry.reedysetmessages: + msg295932
2017-06-13 14:56:52terry.reedysetpull_requests: + pull_request2216
2017-06-13 14:44:54terry.reedysetfiles: + findleak.py

messages: + msg295923
2017-06-13 14:35:11terry.reedysetpull_requests: + pull_request2214
2017-06-13 14:11:04terry.reedysetmessages: + msg295918
2017-06-13 12:51:43terry.reedysetpull_requests: + pull_request2212
2017-06-13 12:29:47terry.reedysetmessages: + msg295911
2017-06-13 06:11:17louielusetmessages: + msg295852
2017-06-13 05:49:07louielusetpull_requests: + pull_request2197
2017-06-13 05:17:26terry.reedycreate