classification
Title: Improve test coverage for idlelib
Type: enhancement Stage: patch review
Components: IDLE, Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: anthony shaw, cheryl.sabella, taleinat, terry.reedy
Priority: normal Keywords: patch, patch, patch, patch

Created on 2019-01-05 21:44 by anthony shaw, last changed 2019-01-09 00:54 by anthonypjshaw.

Pull Requests
URL Status Linked Edit
PR 11451 anthonypjshaw, 2019-01-09 00:54
PR 11451 anthonypjshaw, 2019-01-09 00:54
Messages (10)
msg333077 - (view) Author: anthony shaw (anthony shaw) Date: 2019-01-05 21:44
idlelib is one of the lesser-tested libraries in cpython:
https://codecov.io/gh/python/cpython/tree/master/Lib/idlelib

Raising this issue and also volunteering to extend the test module to get coverage across major behaviours and functions that are missing tests.
msg333083 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-05 23:47
I (we) agree that idlelib needs even better test coverage.  Some history:
1. I added idle_test/ 5-1/2 years ago.  Only the calltip module had automated tests easily converted to unittests (test_calltip).
2. Other exiting tests, requiring human judgement, were debugged, completed, and converted to 'htests', driven by idle_test.htest.  Properly measuring idlelib coverage requires excluding htest code (up to 20% of a module).
3. idle_test/README.txt has information on testing IDLE and local conventions.  Perhaps more should be added about mocking.
4. We should now be using ttk widgets whenever possible.  I would like to add an up-to-date widget-testing doc.

Some modules need doctests and a bit of refactoring before adding tests.  Tests for a specific module should be a separate issue.

Testing tkinter is hard. Can you tell us a bit about your python, tkinter, and testing experience?


Cheryl Sabella wrote the majority of tests added in the last year+, and Tal Einat wrote test_squeezer.  I suggest you review some of their recent work.  They should be able to answer most questions as well as me.
msg333084 - (view) Author: anthony shaw (anthony shaw) Date: 2019-01-06 01:22
thanks terry, 

Some great pointers there, I'll review the exiting work and the README doc.

With regards to my experience, I have quite extensive experience with python testing. Most of which would be open-source on my Github profile https://github.com/tonybaloney
Some of the larger Python projects I've contributed test suites to would be SaltStack, Apache Libcloud and StackStorm.
I've contributed to the tox project and pytest.
I also write tutorials on Python testing (mostly for beginners) like this one https://realpython.com/python-testing/

I admit I don't have much experience with tkinter. 

It does sound like a challenge, but definitely one that I'm willing to research and approach responsibly.
msg333085 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-06 02:17
Great.  I suggest then that you start with untested, normal, non-GUI (non-tkinter) code, assuming that you can find some.  This can mean segregating functional code from gui code if they are currently intertwined.  As point 1 above suggested, IDLE, which dates from about 2000, was originally written without automated testing in mind.

I should have added above
5. 7 months ago, finished #33855 'Minimally test every implementation module'.
In a few cases, that meant import the file, create an instance of the main module, and make a couple of minimal assertions.  The 'coverage' of such files mostly means no syntax errors and X% ran without an exception.

Such code still needs 'does the right thing' tests.  Part of my intention was to make adding those easier by removing the initial boilerplate as a barrier.
msg333086 - (view) Author: anthony shaw (anthony shaw) Date: 2019-01-06 02:24
I was looking at the debugger.py module as being a good place the start, writing test cases for the Idb, Debugger, StackViewer and NamespaceViewer by patching out the dependant components (bdb, Idb, etc.

I might start there, raise a PR against it and do a module at a time, then work my way up to some of the trickier, tkinter-based modules.
msg333089 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-06 03:26
I like that choice.  There are 15 open issues for debugger and I have notes on a few possible enhancements.
msg333113 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2019-01-06 13:47
Welcome Anthony!

Please ask me any questions you may have.  My only suggestion at this point would be for you to feel free to add any docstrings/comments to the module as you go.  Having the docstrings really helps in understanding the tests and vice versa.
msg333124 - (view) Author: anthony shaw (anthony shaw) Date: 2019-01-06 23:19
thanks Cheryl,

here's my branch https://github.com/tonybaloney/cpython/tree/idlelib_tests

I've already seen some code which is 17 years old!
msg333138 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-07 04:27
I like having multiple commits with explanatory messages.  They get squashed when merging into master.  Unless I get distracted and forget, I will squash the messages.
msg333269 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-01-09 00:33
I moved the debugger tests to #35690.  I want to keep this issue for general discussion of testing IDLE, and possibly related PRs improving the documentation thereof.

Serhiy Storchaka, the current tkinter maintainer, and I, have decided that IDLE should run with the implicit default root mechanism disabled.  (It allows bugs unless there are no explicit Tk() calls in the process.)

Therefore, tests should run the same way.  But when I put 'tk.NoDefaultRoot()' in the always-run part of test.test_idle, which is usually run with test.regrtest, there were problems.  So I moved it to the 'if main' clause that runs the IDLE suite directly with unittest.  But this clause only executes when test_idle is the main module, such as when one runs 'python -m test.test_idle' instead of 'python -m test -ugui', with or without 'test_idle added to restrict testing to test_idle.

I will add the consequence for the debugger test to the PR.
History
Date User Action Args
2019-01-09 00:54:15anthonypjshawsetpull_requests: + pull_request10973
2019-01-09 00:54:10anthonypjshawsetpull_requests: + pull_request10972
2019-01-09 00:33:26terry.reedysetkeywords: patch, patch, patch, patch

messages: + msg333269
2019-01-09 00:15:08terry.reedysetpull_requests: - pull_request10913
2019-01-07 04:27:37terry.reedysetkeywords: patch, patch, patch, patch

messages: + msg333138
2019-01-07 04:06:50terry.reedysetpull_requests: - pull_request10914
2019-01-07 04:06:38terry.reedysetpull_requests: - pull_request10915
2019-01-07 04:06:19terry.reedysetpull_requests: - pull_request10916
2019-01-06 23:49:12anthonypjshawsetkeywords: + patch
stage: patch review
pull_requests: + pull_request10915
2019-01-06 23:49:06anthonypjshawsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10916
2019-01-06 23:49:00anthonypjshawsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10914
2019-01-06 23:48:54anthonypjshawsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10913
2019-01-06 23:19:08anthony shawsetmessages: + msg333124
2019-01-06 13:47:26cheryl.sabellasetmessages: + msg333113
2019-01-06 03:26:47terry.reedysetmessages: + msg333089
2019-01-06 02:24:09anthony shawsetmessages: + msg333086
2019-01-06 02:17:40terry.reedysetmessages: + msg333085
2019-01-06 01:22:50anthony shawsetmessages: + msg333084
2019-01-05 23:47:31terry.reedysetnosy: + taleinat, cheryl.sabella
title: low test coverage for idlelib -> Improve test coverage for idlelib
messages: + msg333083

versions: + Python 3.7, Python 3.8
2019-01-05 21:44:20anthony shawcreate