Title: Create a GUI test framework for Idle
Type: enhancement Stage: resolved
Components: Versions: Python 3.3, Python 3.4, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: JayKrish, Todd.Rovito, ezio.melotti, ncoghlan, ned.deily, philwebster, python-dev, r.david.murray, terry.reedy
Priority: normal Keywords:

Created on 2013-05-30 21:06 by terry.reedy, last changed 2013-06-30 00:09 by terry.reedy. This issue is now closed.

Messages (17)
msg190381 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-05-30 21:06
This is a follow-up to #15392, which created the Idle test framework. The similar endpoint for this issue will be a framework for tests that require a gui resource, at least one prototype test file, and documentation of Idle practice. At the moment, completely automated tests that can run on buildbots with a gui, but without human intervention, are a lower priority to me than adding text-only tests.

The tkinter test framework should be a starting point, but some things should be modernized, as they were for the basic framework. This means using unittest rather than regrtest. One thing to copy may be to put gui-tests in a separate directory. I presume load_tests() could conditionally load tests in such a directory.

It is not clear to me to what extent tk widgets can be created and manipulated without being shown on a screen or needing a 'gui' resource.

Idle tests are not intended to test tkinter and tk. A mock_tk could replace at least some gui use. Creating a mock-tk framework would be a different issue from this one. A generic tk mock might possibly go in a tkinter subdirectory. Mocks of Idle classes would belong somewhere below idle_test.

As noted in #15392, about 20 idlelib files have primitive non-automated 'main' tests requiring unspecified human action and observation, some of which do not even run. The 'test' is to see whether the action produced the expected response (which one has to know) or to check the appearance of a text or dialog widget. A first step might be to upgrade and semi-automate these instructions with unittests in idle_test/human. Running them all with one documented command or import would be much nicer pleasant than having to discover and run each individually. I might make this a separate sub-issue.
msg190389 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-05-31 00:53
I opened #18104 for the human-required gui tests described in the last paragraph above.
msg190576 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-04 01:35
My experiments and some web postings indicate that if a tkinter class has a master or parent option, it may not really be an option, regardless of what our docs imply. If tkinter.Tk is called either directly or indirectly, the graphics system is initiated and something is displayed. This appears to includes the seemingly non-graphics Variable (Var) classes such as IntVar (#18131). I started idle_lib/ as part of #18130. This will allow non-gui unit testing of any method whose tkinter use is limited to Vars and message boxes.
msg190611 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-06-04 20:56
Terry: FYI, I just noticed the languishing patches in Issue4343 which *might* be of some use here.
msg190636 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-05 01:28
Thanks Ned. I commented on the issue. The code in the proposed function will fix one of the two problems with tkinter.Variable that I noted in #18131. I can imagine there might be situations in which it would also be useful for testing, as GP claims.
msg191244 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-15 22:49
The basic question for this issue is whether to segregate gui tests in a separate directory (idle_test_gui?) and run them with a separate or to sprinkle gui test cases throughout the test suite, perhaps one to each test_idle/ file?

For development, I am pretty sure I would prefer the latter. I have started test_grep for the 3 methods that can be gui free, but if (when) I do a gui-requiring test of the dialog itself, I would prefer to have it follow in the same file.

However, my impression is that each requires('gui') call will result in a skip message on the test output. (Correct?) So my question is: will non-idle developers tolerate over 50 skip warnings from Idle tests? or is there a way to suppress multiple warnings and consolidate them into just one if there is at least one?
msg191246 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-06-15 22:59
Individual test cases are only seen if you run regrtest (-m test) in verbose mode (-v).  There are many test cases that are akipped in the standard library, depending on platform and -u options, and you normally don't see them (without -v).  So mixing gui-dependent test cases in with others shouldn't be a problem.
msg191255 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-16 02:44
Thanks. That still leaves the problem of getting all tests to run during development when running them with unittest. Calling requires('gui') passes when (_is_gui_available() and (caller is __main__ or 'gui' in use_resources)). Use_resources is normally set by regrtest. As far as I saw, most other support functions work fine with unittest and are not dependent on regrtest.

The doc for unittest.main say that the argv argument can be a list of options passed to the program. Which program? Can that be used to set a resource?

I plan to try
    from test import support; support.use_resources = ['gui']
in the 'if __name__ ...' sections of and, but that will not work from the command line ('python -m unittest whatever').

An alternative is to have requires() check whether the test is running under regrtest (which supposedly changes support.verbose from 1 (which it starts as) to 0) and passing, at least for 'gui', if not. (Is there any other way for a test to detect the test runner?)
msg191259 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-06-16 04:53
Why not just run the tests under regrtest like all other standard library tests?  From the build directory, you could use something like (Windows may be a bit different):

./python.exe -m test -h  # to get regrtest options help
./python.exe -m test -ugui -v test_idle  # to run just the idle tests with verbose on

For 2.7, you'll need to use '-m test.regrtest'.
msg191261 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-06-16 05:20
Indeed, I think the most sensible course here is to require the use of regrtest to run the GUI tests. Our other resource-dependent tests all fall into that category already.

If the stdlib unittest provided a resource system, we'd rewrite the regrtest resource system to be based on that instead, but that's not currently the case.
msg191307 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-17 01:54
support.requires is smart.  If the caller's module is __main__, it treats the resources as set.  So you don't have to do anything special in your __main__ to make the tests runnable directly.

The tests are also run if unittest is called.  I don't know why that is, but I tested it on test_urllibnet, which is protected by the 'net' resource, and the tests ran.
msg191308 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-17 02:02
Oh, actually I do know why that is.  It is because requires is called from test_main in test_urllibnet, which unittest bypasses.  So if you are calling it in particular tests, then I presume that will be an issue for the unittest case.
msg191312 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-17 03:20
I should have explicitly said 'all test methods in one test file'.

I included ' caller is __main__' in the requires 'pass' condition. My first question was only about, for instance, running test_grep from the 'if __name__ ...' part of GrepDialog, For that, I verified as part of #18081 that setting support.use_resources is possible.

My second question was about running all tests in one test_idle file from the command line. If not possible, so be it. I just need to document the limitation.

Since multiple skips are acceptable, the 'framework' patch for this issue will just be a change to README explaining the use of the gui resource and when it is needed.
msg191670 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-22 20:12
Must mention that 'support' was 'test_support' in 2.7.
msg192055 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-06-29 22:30
New changeset c818c215f1a4 by Terry Jan Reedy in branch '3.3':
Issue #18103: Update README.txt and test_idle to describe and run gui tests.
msg192056 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-06-29 22:41
New changeset 0767363a0393 by Terry Jan Reedy in branch '2.7':
Issue #18103: Update README.txt and test_idle to describe and run gui tests.
msg192059 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-30 00:09
While I have not committed any gui tests, the patch is based on experiments with a couple of temporary files.
Date User Action Args
2013-06-30 00:09:26terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg192059

stage: needs patch -> resolved
2013-06-29 22:41:17python-devsetmessages: + msg192056
2013-06-29 22:30:45python-devsetnosy: + python-dev
messages: + msg192055
2013-06-22 20:12:00terry.reedysetmessages: + msg191670
2013-06-18 02:17:30philwebstersetnosy: + philwebster
2013-06-17 17:57:26JayKrishsetnosy: + JayKrish
2013-06-17 03:20:49terry.reedysetmessages: + msg191312
2013-06-17 02:02:49r.david.murraysetmessages: + msg191308
2013-06-17 01:54:02r.david.murraysetmessages: + msg191307
2013-06-16 05:20:56ncoghlansetmessages: + msg191261
2013-06-16 04:53:35ned.deilysetmessages: + msg191259
2013-06-16 02:44:18terry.reedysetmessages: + msg191255
2013-06-15 22:59:35ned.deilysetmessages: + msg191246
2013-06-15 22:49:23terry.reedysetnosy: + ncoghlan, r.david.murray
messages: + msg191244
2013-06-05 01:28:21terry.reedysetmessages: + msg190636
2013-06-04 20:56:53ned.deilysetnosy: + ned.deily
messages: + msg190611
2013-06-04 01:35:56terry.reedysetmessages: + msg190576
2013-05-31 16:25:06Todd.Rovitosetnosy: + Todd.Rovito
2013-05-31 00:53:15terry.reedysetmessages: + msg190389
2013-05-30 21:08:03ezio.melottisetnosy: + ezio.melotti
2013-05-30 21:06:56terry.reedycreate