$ ./python -m test.regrtest -uall -R 3:3 test_idle
[1/1] test_idle
beginning 6 repetitions
123456
test test_idle failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython-debug/Lib/idlelib/idle_test/test_warning.py", line 76, in test_idlever
    self.assertEqual(len(w), 1)
AssertionError: 0 != 1

1 test failed:
    test_idle

The test method was added in 3.5.

       with warnings.catch_warnings(record=True) as w:
            warnings.simplefilter("always")
            import idlelib.idlever
            self.assertEqual(len(w), 1)
            self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
            self.assertIn("version", str(w[-1].message))

It is copied replacements from https://docs.python.org/3.6/library/warnings.html#testing-warnings.

I see that test.support has a wrapper that might or might be easier or make a difference.

Replacing "import idlelib.idlever" with "test.support.import_fresh_module('idlelib.idlever')" might fix the problem.  I will test after I install the 10GB VS2015 package needed to build 3.5/6 on Windows.

import_fresh_module() emits the deprecation warning twice.

>>> from test.support import import_fresh_module
>>> import_fresh_module('idlelib.idlever')
/home/serhiy/py/cpython-debug/Lib/test/support/__init__.py:166: DeprecationWarning: 
The separate Idle version was eliminated years ago;
idlelib.idlever is no longer used by Idle
and will be removed in 3.6 or later.  Use
    from sys import version
    IDLE_VERSION = version[:version.index(' ')]

  __import__(name)
/home/serhiy/py/cpython-debug/Lib/importlib/__init__.py:126: DeprecationWarning: 
The separate Idle version was eliminated years ago;
idlelib.idlever is no longer used by Idle
and will be removed in 3.6 or later.  Use
    from sys import version
    IDLE_VERSION = version[:version.index(' ')]

  return _bootstrap._gcd_import(name[level:], package, level)
<module 'idlelib.idlever' from '/home/serhiy/py/cpython-debug/Lib/idlelib/idlever.py'>

From Mantin Panter's msg265606:

======================================================================
FAIL: test_idlever (idlelib.idle_test.test_warning.ImportWarnTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.5.bolen-tiger/build/Lib/idlelib/idle_test/test_warning.py", line 76, in test_idlever
    self.assertEqual(len(w), 1)
AssertionError: 0 != 1

Serhiy got this by running leak tests, so that sys.modules caching prevented re-execution of warning code.  I am curious: are OSX builtbots doing something similar?  Or is the warning not being issued on the first import, making this a true failure?

In msg255485 'copied replacements' should be 'copied with replacements'.  But 'import idlever' is not a reliable replacement for "# trigger a warning".

I wonder whether 'import_fresh_module' really executes the module twice, which seems foolish, or mere displays one warning twice.

It doesn't really matter here.  I have decided to remove the test.  We don't know if *anyone* imports idlever, and 3.6 will break nearly all external imports, not just that one, so this particular warning and test are not worth the time already spent on them.

New changeset 5f561804bc8e by Terry Jan Reedy in branch '3.5':
Issue #25747: remove undependable and possibly useless test.
https://hg.python.org/cpython/rev/5f561804bc8e

New changeset 9e1c859562bb by Terry Jan Reedy in branch '2.7':
Backports: #25747: remove bad test.  #27044: stop test_idle from leaking.
https://hg.python.org/cpython/rev/9e1c859562bb

idlever.py is now gone in 3.6.