This is a follow-up to bpo-15392, which created the Idle test framework. The similar endpoint for this issue will be a framework for tests that require a gui resource, at least one prototype test file, and documentation of Idle practice. At the moment, completely automated tests that can run on buildbots with a gui, but without human intervention, are a lower priority to me than adding text-only tests.

The tkinter test framework should be a starting point, but some things should be modernized, as they were for the basic framework. This means using unittest rather than regrtest. One thing to copy may be to put gui-tests in a separate directory. I presume load_tests() could conditionally load tests in such a directory.

It is not clear to me to what extent tk widgets can be created and manipulated without being shown on a screen or needing a 'gui' resource.

Idle tests are not intended to test tkinter and tk. A mock_tk could replace at least some gui use. Creating a mock-tk framework would be a different issue from this one. A generic tk mock might possibly go in a tkinter subdirectory. Mocks of Idle classes would belong somewhere below idle_test.

As noted in bpo-15392, about 20 idlelib files have primitive non-automated 'main' tests requiring unspecified human action and observation, some of which do not even run. The 'test' is to see whether the action produced the expected response (which one has to know) or to check the appearance of a text or dialog widget. A first step might be to upgrade and semi-automate these instructions with unittests in idle_test/human. Running them all with one documented command or import would be much nicer pleasant than having to discover and run each individually. I might make this a separate sub-issue.

I opened bpo-18104 for the human-required gui tests described in the last paragraph above.

My experiments and some web postings indicate that if a tkinter class has a master or parent option, it may not really be an option, regardless of what our docs imply. If tkinter.Tk is called either directly or indirectly, the graphics system is initiated and something is displayed. This appears to includes the seemingly non-graphics Variable (Var) classes such as IntVar (bpo-18131). I started idle_lib/mock_tk.py as part of bpo-18130. This will allow non-gui unit testing of any method whose tkinter use is limited to Vars and message boxes.

Terry: FYI, I just noticed the languishing patches in bpo-4343 which *might* be of some use here.

Thanks Ned. I commented on the issue. The code in the proposed function will fix one of the two problems with tkinter.Variable that I noted in bpo-18131. I can imagine there might be situations in which it would also be useful for testing, as GP claims.

The basic question for this issue is whether to segregate gui tests in a separate directory (idle_test_gui?) and run them with a separate test_idle_gui.py? or to sprinkle gui test cases throughout the test suite, perhaps one to each test_idle/text_xxx.py file?

For development, I am pretty sure I would prefer the latter. I have started test_grep for the 3 methods that can be gui free, but if (when) I do a gui-requiring test of the dialog itself, I would prefer to have it follow in the same file.

However, my impression is that each requires('gui') call will result in a skip message on the test output. (Correct?) So my question is: will non-idle developers tolerate over 50 skip warnings from Idle tests? or is there a way to suppress multiple warnings and consolidate them into just one if there is at least one?

Individual test cases are only seen if you run regrtest (-m test) in verbose mode (-v). There are many test cases that are akipped in the standard library, depending on platform and -u options, and you normally don't see them (without -v). So mixing gui-dependent test cases in with others shouldn't be a problem.

Thanks. That still leaves the problem of getting all tests to run during development when running them with unittest. Calling requires('gui') passes when (is_gui_available() and (caller is __main_ or 'gui' in use_resources)). Use_resources is normally set by regrtest. As far as I saw, most other support functions work fine with unittest and are not dependent on regrtest.

The doc for unittest.main say that the argv argument can be a list of options passed to the program. Which program? Can that be used to set a resource?

I plan to try
from test import support; support.use_resources = ['gui']
in the 'if __name__ ...' sections of xyz.py and test_idle.py, but that will not work from the command line ('python -m unittest whatever').

An alternative is to have requires() check whether the test is running under regrtest (which supposedly changes support.verbose from 1 (which it starts as) to 0) and passing, at least for 'gui', if not. (Is there any other way for a test to detect the test runner?)

Why not just run the tests under regrtest like all other standard library tests? From the build directory, you could use something like (Windows may be a bit different):

./python.exe -m test -h # to get regrtest options help
./python.exe -m test -ugui -v test_idle # to run just the idle tests with verbose on

For 2.7, you'll need to use '-m test.regrtest'.

Indeed, I think the most sensible course here is to require the use of regrtest to run the GUI tests. Our other resource-dependent tests all fall into that category already.

If the stdlib unittest provided a resource system, we'd rewrite the regrtest resource system to be based on that instead, but that's not currently the case.

support.requires is smart. If the caller's module is __main__, it treats the resources as set. So you don't have to do anything special in your __main__ to make the tests runnable directly.

The tests are also run if unittest is called. I don't know why that is, but I tested it on test_urllibnet, which is protected by the 'net' resource, and the tests ran.

Oh, actually I do know why that is. It is because requires is called from test_main in test_urllibnet, which unittest bypasses. So if you are calling it in particular tests, then I presume that will be an issue for the unittest case.

I should have explicitly said 'all test methods in one test file'.

I included ' caller is __main__' in the requires 'pass' condition. My first question was only about, for instance, running test_grep from the 'if __name__ ...' part of GrepDialog, For that, I verified as part of bpo-18081 that setting support.use_resources is possible.

My second question was about running all tests in one test_idle file from the command line. If not possible, so be it. I just need to document the limitation.

Since multiple skips are acceptable, the 'framework' patch for this issue will just be a change to README explaining the use of the gui resource and when it is needed.

Must mention that 'support' was 'test_support' in 2.7.

New changeset c818c215f1a4 by Terry Jan Reedy in branch '3.3':
Issue bpo-18103: Update README.txt and test_idle to describe and run gui tests.
http://hg.python.org/cpython/rev/c818c215f1a4

New changeset 0767363a0393 by Terry Jan Reedy in branch '2.7':
Issue bpo-18103: Update README.txt and test_idle to describe and run gui tests.
http://hg.python.org/cpython/rev/0767363a0393

While I have not committed any gui tests, the patch is based on experiments with a couple of temporary files.