classification
Title: IDLE: fix sidebar click-drag bug and macOS test failures
Type: Stage: resolved
Components: IDLE Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: miss-islington, ned.deily, steve.dower, taleinat, terry.reedy
Priority: normal Keywords: patch

Created on 2019-07-29 14:41 by ned.deily, last changed 2020-06-28 16:41 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15010 merged terry.reedy, 2019-07-29 21:30
PR 15012 merged miss-islington, 2019-07-29 21:57
PR 15013 merged miss-islington, 2019-07-29 21:57
PR 15103 merged taleinat, 2019-08-04 10:19
PR 15107 merged miss-islington, 2019-08-04 16:25
PR 15108 merged miss-islington, 2019-08-04 16:25
Messages (24)
msg348666 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-07-29 14:41
On macOS, there are two test failure seen with test_sidebar (tested on 10.14 with Tk 8.6.8 and 8.6.0 - the failures should be reproducible with the python.org 3.8.0b3 installer for macOS which will be available soon.)

======================================================================
FAIL: test_click_selection (idlelib.idle_test.test_sidebar.LineNumbersTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/idlelib/idle_test/test_sidebar.py", line 253, in test_click_selection
    self.assertEqual(self.get_selection(), ('2.0', '3.0'))
AssertionError: Tuples differ: ('1.0', '2.0') != ('2.0', '3.0')

First differing element 0:
'1.0'
'2.0'

- ('1.0', '2.0')
+ ('2.0', '3.0')

======================================================================
FAIL: test_drag_selection (idlelib.idle_test.test_sidebar.LineNumbersTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/idlelib/idle_test/test_sidebar.py", line 273, in test_drag_selection
    self.assertEqual(self.get_selection(), ('1.0', '4.0'))
AssertionError: Tuples differ: ('1.0', '3.0') != ('1.0', '4.0')

First differing element 1:
'3.0'
'4.0'

- ('1.0', '3.0')
?          ^

+ ('1.0', '4.0')
?          ^


----------------------------------------------------------------------
msg348667 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-07-29 14:45
> tested on 10.14 with Tk 8.6.8 and 8.6.0
er, make that "8.6.9" not "8.6.0"
msg348670 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-07-29 15:26
Unfortunately, my Mac is out of order, so I am unable to diagnose and fix this. I hope to be able to do so again in a few weeks.
msg348685 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-29 21:36
IDLE tests passed the Azure macOS test (when it does not timeout) before and since #17535 was merged.  So this failure report initially surprised me.  But then I checked Display Build Info on
https://dev.azure.com/Python/cpython/_build/results?buildId=47634&view=logs&j=18d1a34d-6940-5fc1-f55b-405e2fba32b1

tkinter.TCL_VERSION: 8.5
tkinter.TK_VERSION: 8.5
tkinter.info_patchlevel: 8.5.9

Oh dear. Azure is testing Tkinter, Turtle, and IDLE with Apple's ancient, buggy, and somewhat obsolete system tcl/tk, which we do not support. 

Steve, for years we have closed issues reporting failures and crashes with 8.5.9 and told users to install something more recent.
https://www.python.org/download/mac/tcltk/
We now directly supply something current in the Mac installer.  We need tkinter, etc, tested on macOS with the tcl/tk that we support and supply, just like Azure does on Windows.  

tkinter.TCL_VERSION: 8.6
tkinter.TK_VERSION: 8.6
tkinter.info_patchlevel: 8.6.9

Ned, for the present, we should skip the tests on Mac. If the code and test were made to work on 8.6.x, they might well fail on 8.5.9.  PR 15010.  You will have to un-comment the skips to test for real.

What behavior do users see.  Can you check?  The test says that clicking on line 2 selects line 1.  Does it?  That would be awful. What does click and drag do?  Given the off-by-one reports, I made the text five lines so that a 3-line drag should have unselected lines above and below.  I added a test for drag up as well as down.
msg348687 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-29 21:57
New changeset e8874b85b4e3bbb735467b0beaa933dcef362004 by Terry Jan Reedy in branch 'master':
bpo-37706: Disable 3 IDLE scrollbar tests on Mac. (#15010)
https://github.com/python/cpython/commit/e8874b85b4e3bbb735467b0beaa933dcef362004
msg348690 - (view) Author: miss-islington (miss-islington) Date: 2019-07-29 22:15
New changeset 8513b90e114a815ebf4b389bcdfb08b49207a751 by Miss Islington (bot) in branch '3.8':
bpo-37706: Disable 3 IDLE scrollbar tests on Mac. (GH-15010)
https://github.com/python/cpython/commit/8513b90e114a815ebf4b389bcdfb08b49207a751
msg348691 - (view) Author: miss-islington (miss-islington) Date: 2019-07-29 22:23
New changeset ad36d21223fd8118af8dbcf55ead03ee5905e7c3 by Miss Islington (bot) in branch '3.7':
bpo-37706: Disable 3 IDLE scrollbar tests on Mac. (GH-15010)
https://github.com/python/cpython/commit/ad36d21223fd8118af8dbcf55ead03ee5905e7c3
msg348692 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-29 22:29
Tests disabled, leaving open for live behavior report.
msg348873 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-01 19:22
I can't reproduce this on a Mac now with the recent 3.8.0b3 installer: After removing the skipIf() decorators, the tests all pass as expected.

Ned, can you check with this installer? If it still fails for you, can you give some details about your Mac, e.g. the screen resolution you are using and whether you are using any kind of scaling?
msg348968 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-03 20:44
I can reproduce this at will on all of the real and virtual macOS systems I've tried (10.12 through 10.14) even building with ActiveTcl 8.5.18.  Tal, make sure the appropriate tests cases are actually running.  If you don't have a proper Gui installation of your Python, many of the GUI-related test cases in test_idle are silently skipped unless you add -v or something:

/usr/local/bin/python3.8 -m test -v -uall test_idle
msg348979 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-04 10:25
I was even able to reproduce this on Windows, though not on my Mac. A sneaky bug.

See PR GH-15103 with a fix.
msg348986 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-04 14:58
With PR 15103. the tests appear to no longer fail on macOS.  Thanks!
msg348987 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-04 15:09
Terry, with regard to Azure testing and elsewhere: presumably most of the CI runs are run from processes without a GUI-capable "console" so that test failures like this one would never be seen as test cases requiring a GUI interface would be silently skipped by test_idle (likewise for some tk tests).  Perhaps the tests should issue a message when a GUI isn't present even when -v isn't selected?  Or perhaps a message in test.pythoninfo?
msg348989 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-04 16:25
New changeset 86f1a18abfee5939452f468d80de998918e2afd2 by Terry Jan Reedy (Tal Einat) in branch 'master':
bpo-37706: IDLE - fix sidebar code bug and drag tests (GH-15103)
https://github.com/python/cpython/commit/86f1a18abfee5939452f468d80de998918e2afd2
msg348990 - (view) Author: miss-islington (miss-islington) Date: 2019-08-04 16:43
New changeset 1d2b4dba6ccbeb90ef797034b775c0f1ed17d1a0 by Miss Islington (bot) in branch '3.8':
bpo-37706: IDLE - fix sidebar code bug and drag tests (GH-15103)
https://github.com/python/cpython/commit/1d2b4dba6ccbeb90ef797034b775c0f1ed17d1a0
msg348991 - (view) Author: miss-islington (miss-islington) Date: 2019-08-04 16:47
New changeset 9e0c48a002c144d910a46a74b76b02179502b9aa by Miss Islington (bot) in branch '3.7':
bpo-37706: IDLE - fix sidebar code bug and drag tests (GH-15103)
https://github.com/python/cpython/commit/9e0c48a002c144d910a46a74b76b02179502b9aa
msg348992 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-04 16:56
Tal, thanks for the fix.  Ned, thanks for the quick confirmation.  Closing.

Ned, I don't understand you last comment.  The only CI testing on macOS is done by Azure.  My understanding is that Azure runs a virtual console on Mac as well as linux, so it can run GUI (tk) tests.  But it is using the 'wrong' tcl/tk on Mac.

Steve, can you comment?
msg349056 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-08-05 16:54
All I can add is these are the steps: https://github.com/python/cpython/blob/master/.azure-pipelines/macos-steps.yml

The Linux steps use xvfb, but the macOS steps do not. And we don't do anything at all to install Tcl/Tk on the build agent.
msg349077 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-08-05 21:13
Terry, using an out-of-date version of Tcl/Tk is the least of the problems here.  My point was that, the way things are now, IDLE GUI tests are not being run at all on macOS Azure CI runs regardless of the Tk version in use and, AFAIK, there is no easy way to tell that from the default output from the tests.  The particular test failures reported here *do* fail when properly configured to actually run using the Apple-supplied Tk 8.5.9 (in fact, with the latest fixes to the tests, all of test_idle passes with Tk 8.5.9).  And I'm guessing that most times when you or Tal or other developers are running the tests from your own local builds, you aren't aware that the GUI tests are being skipped.

There is a long-standing issue with Aqua Tk on macOS that, if a process tries to call Tk to draw objects on a screen but the process is not running under a username that is logged in through the normal macOS GUI (i.e. does not have a macOS Finder window, does not have a macOS "console"), Tk will segfault crashing Python along with it, which is very bad when running tests.  There have been various issues about this going back a long time (see, for example, Issue22770 but there are older ones).  To avoid those segfaults, we take a conservative approach in test.support._is_gui_available()  (Lib/test/support/__init__.py) which is effectively (I believe) only allows GUI tests on macOS to run if they are being run under an application bundle which effectively means the Python under test must be a framework build (./configure ... --enable-framework=... ...).  I believe that test is overly stringent but we haven't up to now found a reliable way to test and prevent the segfaults without depending on the framework approach.

So (1) if you don't build Python with an --enable-framework= option, Tk and IDLE GUI tests are not going to be run.

(2) If the GUI tests are skipped, there is no indication of this in the default log output.  You only see this if you run, for example, with the regrtest -v (verbose) option:

$ ./python.exe -m test -uall -j2 test_idle
Run tests in parallel using 2 child processes
0:00:01 load avg: 2.56 [1/1] test_idle passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1 sec 753 ms
Tests result: SUCCESS

$ ./python.exe -m test -v -uall -j2 test_idle
[...]
Run tests in parallel using 2 child processes
0:00:01 load avg: 2.22 [1/1] test_idle passed
skipped 'cannot run without OS X gui process'
skipped 'cannot run without OS X gui process'
skipped 'cannot run without OS X gui process'
test_geticonname (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest) ... ok
test_getsublist (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest) ... ok
test_gettext (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest) ... ok
test_init (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest) ... ok
test_isexpandable (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest) ... ok
test_ondoubleclick (idlelib.idle_test.test_browser.ChildBrowserTreeItemTest) ... ok
skipped 'cannot run without OS X gui process'
[...]
skipped 'cannot run without OS X gui process'
idlelib.idle_test.test_textview (unittest.loader.ModuleSkipped) ... skipped 'cannot run without OS X gui process'
idlelib.idle_test.test_tooltip (unittest.loader.ModuleSkipped) ... skipped 'cannot run without OS X gui process'
idlelib.idle_test.test_tree (unittest.loader.ModuleSkipped) ... skipped 'cannot run without OS X gui process'
idlelib.idle_test.test_undo (unittest.loader.ModuleSkipped) ... skipped 'cannot run without OS X gui process'
test_run_show (idlelib.idle_test.test_warning.RunWarnTest) ... ok
test_showwarnings (idlelib.idle_test.test_warning.RunWarnTest) ... ok
test_idle_formatter (idlelib.idle_test.test_warning.ShellWarnTest) ... ok
test_shell_show (idlelib.idle_test.test_warning.ShellWarnTest) ... ok
test_showwarnings (idlelib.idle_test.test_warning.ShellWarnTest) ... ok
skipped 'cannot run without OS X gui process'
test_init (idlelib.idle_test.test_window.WindowListTest) ... ok
skipped 'cannot run without OS X gui process'

----------------------------------------------------------------------

Ran 257 tests in 1.001s

OK (skipped=70)

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1 sec 836 ms
Tests result: SUCCESS

So there is a false sense of security that tests are being run when they aren't and those skipped GUI test might be failing if they were actually run as was the case in this issue.

Possible followup actions:

1. Have test_idle (and test_ttk_guionly?) issue a warning message by default if GUI tests are being skipped.

2. Investigate if it is practical to run GUI tests under Azure CI (I'm guessing it is not) or one of the other CI runners we use (Appveyor maybe?).  (Note I personally will not have time to look into this until at least the Sep dev sprint so it would be good for someone else to look into it).  That would also require changing the CI to make and test framework builds.

3. It *should* be possible to run GUI tests on a macOS buildbot where the console is logged into the buildbot user.  That will also take some investigation and reconfiguration as a framework build.

4. Find an easier way to run macOS tests with an up-to-date Tk.  I have plans to provide something in the near future.

5. It may be possible to loosen the test for GUI environment so that a framework build is not necessary but having a logged-in "console" user would need to happen anyway so that's much less important than the other items.
msg349080 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-05 22:09
I remembered this morning that we need to check test_tkk_guionly or (easier, but less obviously also all gui) test_tk either in test suite output, or run by itself with
  python -m test.test_tk  # or test -ugui test_tk
because idle it has non-gui tests that always run. From a random Azure Mac output:
  test_ttk_guionly skipped -- cannot run without OS X gui process
whereas Azure Ubuntu gives
  [201/419] test_ttk_guionly passed
and ditto for test_tk.  Also, "test_tix test_tk test_ttk_guionly" under 'tests skipped'.

The skips for the only current Mac buildbot I found,
https://buildbot.python.org/all/#/workers/20,
Matt Billenstein <matt@vazor.com> High Sierra 10.13.6,
also include "test_tix test_tk test_ttk_guionly"

Perhaps you can ask him if his machine is suitable for adding gui tests and offer to help.

Tal, when you can build on Mac, are you making framework builds?  Does test_tk run?
msg372516 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-06-28 13:48
I had been running the GUI tests, apparently until the change Ned mentioned whereby they are only run on framework builds of Python.

With Ned's help, I've finally managed to create a framework build on macOS where the GUI tests run properly.

Indeed, running test_tk without such a build causes it to be skipped. With a framework build and -ugui (or -uall), test_tk runs (and fails!).
msg372517 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-06-28 13:52
While we're on the topic, it seems wise to have our macOS buildbot workers create proper framework builds and run the GUI tests, if possible.

We should also try to get a macOS 10.15 ("Catalina") machine into our buildbot fleet. Perhaps there's a company with macOS machines that would be willing to run one?

Ned, would you like to bring this up on the buildbots mailing list, or should I?
msg372518 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-28 14:06
Running the GUI test on macOS buildbots is not straightforward, AFAIK. We'd need to have either the console logged in to the username that the tests are running under or, better, the buildbot would need to run in a virtual machine. We have had a chronic shortage of macOS buildbots anyway. Plus the automated tests aren't all that good at finding the nasty problems that pop up in the intersection of the user, IDLE, and macOS Tk; at the moment, there's no substitute for manual testing.
msg372525 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-28 16:41
The htests are for manual testing and they include one for sidebar, with instructions to click and drag.  We just need to remember to run them occasionally on the various systems.
History
Date User Action Args
2020-06-28 16:41:15terry.reedysetmessages: + msg372525
2020-06-28 14:06:12ned.deilysetmessages: + msg372518
2020-06-28 13:52:18taleinatsetmessages: + msg372517
2020-06-28 13:48:27taleinatsetmessages: + msg372516
2019-08-05 22:09:50terry.reedysetmessages: + msg349080
2019-08-05 21:13:28ned.deilysetmessages: + msg349077
2019-08-05 16:54:08steve.dowersetmessages: + msg349056
2019-08-04 16:56:03terry.reedysetstatus: open -> closed
title: IDLE test_sidebar test_(click/drag)_selection fail on macOS 8.6 -> IDLE: fix sidebar click-drag bug and macOS test failures
messages: + msg348992

resolution: fixed
stage: patch review -> resolved
2019-08-04 16:47:14miss-islingtonsetmessages: + msg348991
2019-08-04 16:43:59miss-islingtonsetnosy: + miss-islington
messages: + msg348990
2019-08-04 16:25:45miss-islingtonsetpull_requests: + pull_request14850
2019-08-04 16:25:39miss-islingtonsetpull_requests: + pull_request14849
2019-08-04 16:25:34terry.reedysetmessages: + msg348989
2019-08-04 15:09:10ned.deilysetmessages: + msg348987
2019-08-04 14:58:27ned.deilysetmessages: + msg348986
2019-08-04 10:25:59taleinatsetmessages: + msg348979
2019-08-04 10:19:32taleinatsetstage: needs patch -> patch review
pull_requests: + pull_request14845
2019-08-03 20:44:33ned.deilysetmessages: + msg348968
2019-08-01 19:22:57taleinatsetmessages: + msg348873
2019-07-29 22:29:04terry.reedysetnosy: - miss-islington
title: IDLE test_sidebar testcases test_click_selection and test_drag_selection can fail on macOS -> IDLE test_sidebar test_(click/drag)_selection fail on macOS 8.6
messages: + msg348692

stage: patch review -> needs patch
2019-07-29 22:23:02miss-islingtonsetmessages: + msg348691
2019-07-29 22:15:54miss-islingtonsetnosy: + miss-islington
messages: + msg348690
2019-07-29 21:57:57miss-islingtonsetpull_requests: + pull_request14776
2019-07-29 21:57:49miss-islingtonsetstage: needs patch -> patch review
pull_requests: + pull_request14775
2019-07-29 21:57:40terry.reedysetmessages: + msg348687
2019-07-29 21:36:16terry.reedysetnosy: + steve.dower

messages: + msg348685
stage: patch review -> needs patch
2019-07-29 21:30:29terry.reedysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request14773
2019-07-29 15:26:46taleinatsetmessages: + msg348670
2019-07-29 14:45:37ned.deilysetmessages: + msg348667
2019-07-29 14:41:39ned.deilycreate