https://buildbot.python.org/all/#/builders/597/builds/178

test_error (idlelib.idle_test.test_run.HandleErrorTest) ... FAIL
(...)

======================================================================
FAIL: test_error (idlelib.idle_test.test_run.HandleErrorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.8.cstratak-RHEL8-aarch64.refleak/build/Lib/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/home/buildbot/buildarea/3.8.cstratak-RHEL8-aarch64.refleak/build/Lib/idlelib/idle_test/test_run.py", line 352, in test_error
    eq(self.func.called, 2)
AssertionError: 4 != 2



I can reproduce the issue on 3.8, 3.9 and master branches:

14:00:18 vstinner@apu$ ./python -m test -m test_error -v test_idle  test_idle  
== CPython 3.8.5+ (heads/3.8:61f23cb62d, Aug 10 2020, 14:00:04) [GCC 10.2.1 20200723 (Red Hat 10.2.1-1)]
== Linux-5.7.11-200.fc32.x86_64-x86_64-with-glibc2.29 little-endian
== cwd: /home/vstinner/python/3.8/build/test_python_43858
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.68 Run tests sequentially
0:00:00 load avg: 1.68 [1/2] test_idle
test_error (idlelib.idle_test.test_run.HandleErrorTest) ... ok

----------------------------------------------------------------------

Ran 1 test in 0.004s

OK
0:00:00 load avg: 1.68 [2/2] test_idle
test_error (idlelib.idle_test.test_run.HandleErrorTest) ... FAIL

======================================================================
FAIL: test_error (idlelib.idle_test.test_run.HandleErrorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/3.8/Lib/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/home/vstinner/python/3.8/Lib/idlelib/idle_test/test_run.py", line 352, in test_error
    eq(self.func.called, 2)
AssertionError: 4 != 2

----------------------------------------------------------------------

Ran 1 test in 0.007s

FAILED (failures=1)
test test_idle failed
test_idle failed

== Tests result: FAILURE ==

1 test OK.

1 test failed:
    test_idle

Total duration: 722 ms
Tests result: FAILURE


It seems like the test was added recently:

commit f2e161c27964a59bc5ab20d96f87ba5862c6222d
Author: Terry Jan Reedy <tjreedy@udel.edu>
Date:   Sun Aug 9 16:08:30 2020 -0400

    bpo-41468: Improve and test IDLE run error exit  (GH-21798)
    
    A message box pops up when an unexpected error stops the run process.  Tell users it is likely a random glitch, but report it if not.