Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idle: make human-mediated GUI tests usable #62304

Closed
terryjreedy opened this issue May 31, 2013 · 16 comments
Closed

Idle: make human-mediated GUI tests usable #62304

terryjreedy opened this issue May 31, 2013 · 16 comments
Assignees
Labels
topic-IDLE type-bug An unexpected behavior, bug, or error

Comments

@terryjreedy
Copy link
Member

BPO 18104
Nosy @terryjreedy, @ezio-melotti, @serwy, @bitdancer, @rovitotv
Dependencies
  • bpo-18130: idlelib.configSectionNameDialog: fix and add tests and mocks
  • Files
  • 18104-htest1.diff
  • 18104-htest4.diff
  • 18104-htest.txt: Notes from a year ago.
  • htest5.diff: The patch pushed (except News entry).
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/terryjreedy'
    closed_at = <Date 2014-05-12.04:17:46.277>
    created_at = <Date 2013-05-31.00:52:01.157>
    labels = ['expert-IDLE', 'type-bug']
    title = 'Idle: make human-mediated GUI tests usable'
    updated_at = <Date 2019-03-25.08:04:31.943>
    user = 'https://github.com/terryjreedy'

    bugs.python.org fields:

    activity = <Date 2019-03-25.08:04:31.943>
    actor = 'terry.reedy'
    assignee = 'terry.reedy'
    closed = True
    closed_date = <Date 2014-05-12.04:17:46.277>
    closer = 'terry.reedy'
    components = ['IDLE']
    creation = <Date 2013-05-31.00:52:01.157>
    creator = 'terry.reedy'
    dependencies = ['18130']
    files = ['34410', '35116', '35119', '35222']
    hgrepos = []
    issue_num = 18104
    keywords = ['patch']
    message_count = 16.0
    messages = ['190388', '190391', '190430', '190438', '190573', '213319', '213499', '213514', '217548', '217560', '217632', '217640', '218310', '218311', '218313', '218637']
    nosy_count = 8.0
    nosy_names = ['terry.reedy', 'ezio.melotti', 'roger.serwy', 'r.david.murray', 'Todd.Rovito', 'python-dev', 'JayKrish', 'Saimadhav.Heblikar']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue18104'
    versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

    @terryjreedy
    Copy link
    Member Author

    23 of the 62 idlelib/*.py files have an 'if __name__ ...' test that brings up a tkinter gui widget and waits for the human tester to do ... something. Problems:

    1. They are a bit of a nuisance to run individually: either type
      python -m idlelib.SomeLongNameYoumightMistype
      or open in an editor and hit F5.

    Running all is much worse given that I found no list; doing the above with all 62 files to find the 23 is something few would do.

    1. About 5 do not run; at least 1 is a simple incomplete 2to3 fix (which means it has never been run with 3.x). I will post separately any that are not obvious.

    2. They were written by and for the module authors; some are a puzzle. For example: python -m idlelib.FileList. I do not know what to do to perform the test.

    3. About 4 of those that do run do not stop properly, depending on how run. Example: after python -m idlelib.ColorDelegator, the puzzling window (blank on 3.3, not 2.7) seems to close correctly, but open the same file and run with F5 (2.7 or 3.3, Windows) and pressing [x] in the window does nothing. But most do close so this is something different with a few.

    4. Even when the test is fairly obvious, (python -m idlelib.MultiStatusBar) there is no way to indicate failure so that an automated record of failures is collected.

    My overall idea expands on the example of the dialog tests, such as
    python -m idlelib.aboutDialog
    Each opens a master test window with a start button that opens a window with the dialog be tested. Make a standardized master window for all tests, add an explanation of what to do (or a button to display it) and buttons to record the result: OK, NothingHappened, or Error (explain in text widget). A test method would set up the explanation and test item, connect both to a master window, run, retrieve result (I have no idea how to do this ;-), and end with AssertTrue(result, explanation).

    I am thinking to suffix human-only tests with '_H' and put 'test_xyy_H' in a subdirectory such as idle_test/human. Some test files might test several different modules, such as 'test_dialog_H'.

    @terryjreedy terryjreedy self-assigned this May 31, 2013
    @terryjreedy terryjreedy added the type-bug An unexpected behavior, bug, or error label May 31, 2013
    @bitdancer
    Copy link
    Member

    Since you *don't* want these to be autodiscovered by testing frameworks, perhaps h_test_xxxx.py would be a better naming scheme.

    @terryjreedy
    Copy link
    Member Author

    What I want is for the tests to be discovered when they should be and not when they should not ;-) -- without jumping through too many hoops. I probably once read 'recursing into subdirectories' but forgot. I like 'h_test...' for 'human test ...'.

    Thinking about it more, unittest does not not add much to this type of test. If dialogs maps dialog to test information used by h_test_dialog, then a first draft of testing all (there are 6 that I know of, 3 with display tests) could amount to
    def h_test_dialogs():
    for dialog in dialogs:
    h_test_dialog(dialog)
    New entries to dialogs would be 'discovered' as added.

    @terryjreedy
    Copy link
    Member Author

    While improving the self-test for confixSectionNameDialog.py after fixing the error that stopped it from running, I discovered an error in the NameOk method: names were not really stripped before being tested. so duplicate section names were erroneously accepted if initially surrounded by whitespace. After fixing that, I removed unneeded code, including 5 pairs of parentheses in one statement, and added recommended spaces. I also centered the action buttons in the dialog.

    Attached is a drop-in replacement for the file (produced with Idle on my non-developement Win7 machine, so probably with the extra line end char). I think it is good enough to commit, but I am curious if the instructions added to the test are clear enough. (The dialog is the one seen if Save as New Custom Theme (Highlighting tab) or Save as New Custom Key Set (Key tab) are selected on the Options / Preferences dialog.

    For new custom test files, the root window should be open farther from the edge of the screen.

    @terryjreedy
    Copy link
    Member Author

    I opened bpo-18130 for an updated configSectionNameDialog.py patch. It also adds to idle_tests two files, mock_tk.py and test_config_name.py. The latter uses the former for gui-free automated tests of some of the dialog methods. I plan to commit in a couple of days. The fixed human test is required for this issue.

    @terryjreedy
    Copy link
    Member Author

    Some follow-up after revising the test for GetCfgSectionNameDialog in configSectionNameDialog.py (bpo-18130):

    1. I am no longer concerned about automated discovery. Anyone adding a widget can just as easily add something to a master list as add a file to be discovered. I meant to type up and post my handwritten list of files with widget tests, with the status of each. I will try to find it or recreate it.

    2. I am not so much concerned about an automated summary. Once the tests work correctly, errors due to editing the underlying code should be caught by immediate testing. The model I used is

    if __name__...:
      <run unittests, as usual>
      <if file defines visible widget, also run human test>

    where <run human test> currently means to open a tk window with a button to start the test, but which can instead just be closed [x] to skip the test. My idea is that after editing, one should always run the automated test (if there is one, which there eventually will be) and maybe run the visual test.

    1. For bpo-18130, I left the test code in the 'if name' clause. To run all such tests from a separate master htest file, it needs to be moved into a function. I am thinking of '_htest'. (Some files already have the visual test code pulling into a 'main' function, but to me that is too similar to 'unitest.main' and similar.) Or perhaps the code should be moved to an idle_test/htest.py file, but with a means to run just the one test from the module file.

    2. For this test, the widget has a message parameter, which I used for test hints. Other widgets do not have such, and the test hints will have to be in the driver window.

    I need to pick, say, 3 tests and do some experiments.

    @terryjreedy
    Copy link
    Member Author

    I thought through my design criteria a bit, and in the process, decided on a specific concrete proposal to test.

    1. It should be possible to run one test. It should easy to run (or not) the test (or tests) for a module when editing its file. It should be easy to tell when editing that there is a human test. These would both be served the a couple of lines in the 'if name' block (after the unittest lines) such as
        from idlelib.idle_test.htest import run
        run(GetCfgSectionNameDialog)  # or other class
    1. It should be possible to see which modules have a human test. But I do not want a directory of 20+ 10-line files.

    2. I want to factor out as much boilerplate code as possible. This will both make it easier to add tests and to modify the behavior of all tests at once.

    Something like the following may work for both criteria.

    In idle_test/htest.py

    GetCfgSectionNameDialog_spec = {
      'kwds': {'title':'Get Name',
             'message':'Enter something',
             'used-names': {'abc'}},
      'msg': "After the text entered with [Ok] is stripped, <nothing>, "
             "'abc', or more that 30 chars are errors. "
             "Close with a valid entry (printed), [Cancel], or [X]"}
    
    def run(klas):
      root = tk.Tk()
      klas_spec = globals[klas.__name__+'_arg']
      klas_spec.kwds['parent'] = root # does Idle use 'parent' consistently?
      def run_klas():
        widget = klas(**klas_spec.kwds)
        try:
          print(widget.result)
        except AttributeError:
          pass
      Message(root, text=klas_spec.msg).pack()
      Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
      root.mainloop()
    1. It should be possible to discover and run all tests. With one object per widget in htest, each with a marked name, it is easy to filter globals for marked objects.

    filter(lambda ob: hasattr(ob, '__name__') and '_args' in ob.__names__, globals())

    @terryjreedy
    Copy link
    Member Author

    After correcting bugs in the 'explaining with code' posted before, and making a few other changes, the attached 3.3 diff works. The main substantive change is that I added a parameter to the class to move the tested dialog under the toplevel driver so the test message would be visible. The default for this dialog is centered over the parent (normally the config dialog).

    If I still like this design after adding a couple more tests and a run_all function, I will commit it.

    @terryjreedy
    Copy link
    Member Author

    The new patch adds a docstring with spec template, a second example, and a crude runall(). I am inclined to push this as a base for further patches -- at least one to improve runall and others to test more widget classes. These might be separate issues.

    @terryjreedy
    Copy link
    Member Author

    New patch. I refined the definition of the parameter for run() in the process of adding a test for EditorWindow.HelpDialog. This cannot be tested directly. This example is also a reminder that some files define more than one display widget.

    I should try template and framework with non-dialog, but the example above pretty well convinces me that the framework should be flexible enough.

    @terryjreedy
    Copy link
    Member Author

    Defective patches removed and replaced with one that includes current htest.py and should have proper (unix) line endings.

    @terryjreedy
    Copy link
    Member Author

    Here are my notes on modules that should probably be included in htest. Most have some end-of-module test now, many with errors.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 12, 2014

    New changeset 460203eaf731 by Terry Jan Reedy in branch '2.7':
    Issue bpo-18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
    http://hg.python.org/cpython/rev/460203eaf731

    New changeset 617656f7b538 by Terry Jan Reedy in branch '3.4':
    Issue bpo-18104: Add idlelib/idle_test/htest.py with a few sample tests to begin
    http://hg.python.org/cpython/rev/617656f7b538

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 12, 2014

    New changeset aea4f427902f by Terry Jan Reedy in branch '3.4':
    Issue bpo-18104: News for 3.4 (which will not merge forward).
    http://hg.python.org/cpython/rev/aea4f427902f

    New changeset b7bc43e96041 by Terry Jan Reedy in branch 'default':
    bpo-18104: null merge of 3.4 News entry
    http://hg.python.org/cpython/rev/b7bc43e96041

    New changeset 7cdb38fa191e by Terry Jan Reedy in branch 'default':
    Issue bpo-18104: News for 3.4, in proper place.
    http://hg.python.org/cpython/rev/7cdb38fa191e

    @terryjreedy
    Copy link
    Member Author

    The revised patch, as pushed, will serve as a base for this portion of Saimadhav's GSOC project. bpo-21477 proposes changes to htest itself. Adding tests from other modules will be dependencies of that issue.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 16, 2014

    New changeset 0a6d51ccff54 by Terry Jan Reedy in branch '2.7':
    Issue bpo-18104: revise docstrings, remove obsolete comments.
    http://hg.python.org/cpython/rev/0a6d51ccff54

    New changeset 6d2982ff441f by Terry Jan Reedy in branch '3.4':
    Issue bpo-18104: revise docstrings, remove obsolete comments.
    http://hg.python.org/cpython/rev/6d2982ff441f

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    topic-IDLE type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants