23 of the 62 idlelib/*.py files have an 'if __name__ ...' test that brings up a tkinter gui widget and waits for the human tester to do ... something. Problems:

0. They are a bit of a nuisance to run individually: either type
python -m idlelib.SomeLongNameYoumightMistype
or open in an editor and hit F5.

Running all is much worse given that I found no list; doing the above with all 62 files to find the 23 is something few would do.

1. About 5 do not run; at least 1 is a simple incomplete 2to3 fix (which means it has never been run with 3.x). I will post separately any that are not obvious.

2. They were written by and for the module authors; some are a puzzle. For example: python -m idlelib.FileList. I do not know what to do to perform the test.

3. About 4 of those that do run do not stop properly, depending on how run. Example: after python -m idlelib.ColorDelegator, the puzzling window (blank on 3.3, not 2.7) seems to close correctly, but open the same file and run with F5 (2.7 or 3.3, Windows) and pressing [x] in the window does nothing. But most do close so this is something different with a few.

4. Even when the test is fairly obvious, (python -m idlelib.MultiStatusBar) there is no way to indicate failure so that an  automated record of failures is collected.

My overall idea expands on the example of the dialog tests, such as
python -m idlelib.aboutDialog
Each opens a master test window with a start button that opens a window with the dialog be tested. Make a standardized master window for all tests, add an explanation of what to do (or a button to display it) and buttons to record the result: OK, NothingHappened, or Error (explain in text widget). A test method would set up the explanation and test item, connect both to a master window, run, retrieve result (I have no idea how to do this ;-), and end with AssertTrue(result, explanation).

I am thinking to suffix human-only tests with '_H' and put 'test_xyy_H' in a subdirectory such as idle_test/human. Some test files might test several different modules, such as 'test_dialog_H'.

Since you *don't* want these to be autodiscovered by testing frameworks, perhaps h_test_xxxx.py would be a better naming scheme.

What I want is for the tests to be discovered when they should be and not when they should not ;-) -- without jumping through too many hoops. I probably once read 'recursing into subdirectories' but forgot. I like 'h_test...' for 'human test ...'.

Thinking about it more, unittest does not not add much to this type of test. If dialogs maps dialog to test information used by h_test_dialog, then a first draft of testing all (there are 6 that I know of, 3 with display tests) could amount to
def h_test_dialogs():
  for dialog in dialogs:
    h_test_dialog(dialog)
New entries to dialogs would be 'discovered' as added.

While improving the self-test for confixSectionNameDialog.py after fixing the error that stopped it from running, I discovered an error in the NameOk method: names were not really stripped before being tested. so duplicate section names were erroneously accepted if initially surrounded by whitespace. After fixing that, I removed unneeded code, including 5 pairs of parentheses in one statement, and added recommended spaces. I also centered the action buttons in the dialog.

Attached is a drop-in replacement for the file (produced with Idle on my non-developement  Win7 machine, so probably with the extra line end char). I think it is good enough to commit, but I am curious if the instructions added to the test are clear enough. (The dialog is the one seen if Save as New Custom Theme (Highlighting tab) or Save as New Custom Key Set (Key tab) are selected on the Options / Preferences dialog.

For new custom test files, the root window should be open farther from the edge of the screen.

I opened #18130 for an updated configSectionNameDialog.py patch. It also adds to idle_tests two files, mock_tk.py and test_config_name.py. The latter uses the former for gui-free automated tests of some of the dialog methods. I plan to commit in a couple of days. The fixed human test is required for this issue.

Some follow-up after revising the test for GetCfgSectionNameDialog in  configSectionNameDialog.py (#18130):

1. I am no longer concerned about automated discovery. Anyone adding a widget can just as easily add something to a master list as add a file to be discovered. I meant to type up and post my handwritten list of files with widget tests, with the status of each. I will try to find it or recreate it.

2. I am not so much concerned about an automated summary. Once the tests work correctly, errors due to editing the underlying code should be caught by immediate testing.  The model I used is

if __name__...:
  <run unittests, as usual>
  <if file defines visible widget, also run human test>

where <run human test> currently means to open a tk window with a button to start the test, but which can instead just be closed [x] to skip the test. My idea is that after editing, one should always run the automated test (if there is one, which there eventually will be) and maybe run the visual test.

3. For #18130, I left the test code in the 'if name' clause. To run all such tests from a separate master htest file, it needs to be moved into a function. I am thinking of '_htest'. (Some files already have the visual test code pulling into a 'main' function, but to me that is too similar to 'unitest.main' and similar.) Or perhaps the code should be moved to an idle_test/htest.py file, but with a means to run just the one test from the module file.

4. For this test, the widget has a message parameter, which I used for test hints. Other widgets do not have such, and the test hints will have to be in the driver window.

I need to pick, say, 3 tests and do some experiments.

I thought through my design criteria a bit, and in the process, decided on a specific concrete proposal to test.

1. It should be possible to run one test. It should easy to run (or not) the test (or tests) for a module when editing its file. It should be easy to tell when editing that there is a human test. These would both be served the a couple of lines in the 'if name' block (after the unittest lines) such as

    from idlelib.idle_test.htest import run
    run(GetCfgSectionNameDialog)  # or other class

2. It should be possible to see which modules have a human test. But I do not want a directory of 20+ 10-line files.

3. I want to factor out as much boilerplate code as possible. This will both make it easier to add tests and to modify the behavior of all tests at once.

Something like the following may work for both criteria.

In idle_test/htest.py

GetCfgSectionNameDialog_spec = {
  'kwds': {'title':'Get Name',
         'message':'Enter something',
         'used-names': {'abc'}},
  'msg': "After the text entered with [Ok] is stripped, <nothing>, "
         "'abc', or more that 30 chars are errors. "
         "Close with a valid entry (printed), [Cancel], or [X]"}

def run(klas):
  root = tk.Tk()
  klas_spec = globals[klas.__name__+'_arg']
  klas_spec.kwds['parent'] = root # does Idle use 'parent' consistently?
  def run_klas():
    widget = klas(**klas_spec.kwds)
    try:
      print(widget.result)
    except AttributeError:
      pass
  Message(root, text=klas_spec.msg).pack()
  Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
  root.mainloop()

4. It should be possible to discover and run all tests. With one object per widget in htest, each with a marked name, it is easy to filter globals for marked objects.

  filter(lambda ob: hasattr(ob, '__name__') and '_args' in ob.__names__, globals())

After correcting bugs in the 'explaining with code' posted before, and making a few other changes, the attached 3.3 diff works. The main substantive change is that I added a parameter to the class to move the tested dialog under the toplevel driver so the test message would be visible. The default for this dialog is centered over the parent (normally the config dialog).

If I still like this design after adding a couple more tests and a run_all function, I  will commit it.

The new patch adds a docstring with spec template, a second example, and a crude runall(). I am inclined to push this as a base for further patches -- at least one to improve runall and others to test more widget classes. These might be separate issues.

New patch. I refined the definition of the parameter for run() in the process of adding a test for EditorWindow.HelpDialog. This cannot be tested directly. This example is also a reminder that some files define more than one display widget.

I should try template and framework with non-dialog, but the example above pretty well convinces me that the framework should be flexible enough.

Defective patches removed and replaced with one that includes current htest.py and should have proper (unix) line endings.

Here are my notes on modules that should probably be included in htest. Most have some end-of-module test now, many with errors.

New changeset 460203eaf731 by Terry Jan Reedy in branch '2.7':
Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
http://hg.python.org/cpython/rev/460203eaf731

New changeset 617656f7b538 by Terry Jan Reedy in branch '3.4':
Issue #18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
http://hg.python.org/cpython/rev/617656f7b538

New changeset aea4f427902f by Terry Jan Reedy in branch '3.4':
Issue #18104: News for 3.4 (which will not merge forward).
http://hg.python.org/cpython/rev/aea4f427902f

New changeset b7bc43e96041 by Terry Jan Reedy in branch 'default':
#18104: null merge of 3.4 News entry
http://hg.python.org/cpython/rev/b7bc43e96041

New changeset 7cdb38fa191e by Terry Jan Reedy in branch 'default':
Issue #18104: News for 3.4, in proper place.
http://hg.python.org/cpython/rev/7cdb38fa191e

The revised patch, as pushed, will serve as a base for this portion of Saimadhav's GSOC project. #21477 proposes changes to htest itself. Adding tests from other modules will be dependencies of that issue.

New changeset 0a6d51ccff54 by Terry Jan Reedy in branch '2.7':
Issue #18104: revise docstrings, remove obsolete comments.
http://hg.python.org/cpython/rev/0a6d51ccff54

New changeset 6d2982ff441f by Terry Jan Reedy in branch '3.4':
Issue #18104: revise docstrings, remove obsolete comments.
http://hg.python.org/cpython/rev/6d2982ff441f