23 of the 62 idlelib/*.py files have an 'if __name__ ...' test that brings up a tkinter gui widget and waits for the human tester to do ... something. Problems:

0. They are a bit of a nuisance to run individually: either type
python -m idlelib.SomeLongNameYoumightMistype
or open in an editor and hit F5.

Running all is much worse given that I found no list; doing the above with all 62 files to find the 23 is something few would do.

1. About 5 do not run; at least 1 is a simple incomplete 2to3 fix (which means it has never been run with 3.x). I will post separately any that are not obvious.

2. They were written by and for the module authors; some are a puzzle. For example: python -m idlelib.FileList. I do not know what to do to perform the test.

3. About 4 of those that do run do not stop properly, depending on how run. Example: after python -m idlelib.ColorDelegator, the puzzling window (blank on 3.3, not 2.7) seems to close correctly, but open the same file and run with F5 (2.7 or 3.3, Windows) and pressing [x] in the window does nothing. But most do close so this is something different with a few.

4. Even when the test is fairly obvious, (python -m idlelib.MultiStatusBar) there is no way to indicate failure so that an  automated record of failures is collected.

My overall idea expands on the example of the dialog tests, such as
python -m idlelib.aboutDialog
Each opens a master test window with a start button that opens a window with the dialog be tested. Make a standardized master window for all tests, add an explanation of what to do (or a button to display it) and buttons to record the result: OK, NothingHappened, or Error (explain in text widget). A test method would set up the explanation and test item, connect both to a master window, run, retrieve result (I have no idea how to do this ;-), and end with AssertTrue(result, explanation).

I am thinking to suffix human-only tests with '_H' and put 'test_xyy_H' in a subdirectory such as idle_test/human. Some test files might test several different modules, such as 'test_dialog_H'.