This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author terry.reedy
Recipients brian.curtin, ezio.melotti, michael.foord, ocean-city, pitrou, r.david.murray, terry.reedy
Date 2012-07-19.23:21:47
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1342740111.43.0.0586923435631.issue10652@psf.upfronthosting.co.za>
In-reply-to
Content
I think this test disabling and failure issue should get some attention.
Do the failures happen on non-Windows systems too?

With installed 3.3.0b1 on Win7, test___all__, test_tcl, test_tk, test_ttk_textonly, and test_ttk_guionly all run if run first (_tk and _ttk_guionly with -ugui). All modify os.environ, resulting in
Warning -- os.environ was modified by test_xxx
and all disable at least one of the other tests run afterward. (test___all__ disables but is not disabled itself.) In standard alphabetic running order, test___all__ 'wins'.

I would like to add test/test_idle, #15392, but I presume it currently would also not run after test___all__ or any of the others.

---
I patched my installation, but without the unneeded temporary (saved_values = self.saved_values is only an optimization for the repeated access in the loop that follows).
        sys.modules.clear()
        sys.modules.update(self.saved_modules)
        del self.saved_modules

With this,
python -m test -ugui test___all__ test_tcl test_tk test_ttk_guionly test_ttk_textonly
runs and passes all five tests instead of two (test___all__ and test_ttk_textonly).

Unfortunately, it causes new failures running the entire test suite:
test_concurrent_futures (ok before)
test_decimal (ok before)
test_email (numberous failures, just one before)
I stopped at test_faulthandler, there might be more but this was enough to reject the patch as is. 

I decided to try the principle of doing the minimum necessary to get the tests to pass. I arrived at the following, which lets all five tests run and pass, at least when run in the given order. 

        try:
            del sys.modules['tkinter']
            del sys.modules['tkinter._fix']
            del sys.modules['tkinter.ttk']
            del sys.modules['tkinter.test']
            del sys.modules['tkinter.test.support']
            del sys.modules['tkinter.test.runtktests']
        except KeyError:
            pass

The only new failure with python -m test is test_multi-processing, which normally does not even give a warning.

[197/368/6] test_multiprocessing
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing
test test_multiprocessing failed -- Traceback (most recent call last):
  File "C:\Programs\Python33\lib\test\test_multiprocessing.py", line 1909, in test_mymanager_context_prestarted
    self.assertEqual(manager._process.exitcode, 0)
AssertionError: -15 != 0

This might be an unrelated, intermittant failure.

Since the exception above will trigger for nearly all tests (costly), and since the above does not guarantee to delete all tkinter.* modules, and since future tkinter tests might expand to test other subpackages and idle tests will import other subpackages, I tried this in __exit__.

        if 'tkinter' in sys.modules:
            sysmod = sys.modules
            tknames = set()
            for name in sysmod:
                if name.startswith('tkinter'):
                    tknames.add(name)
            for name in tknames:
                del sysmod[name]

So aside from the initial test, the extra time is only used when needed. All five tests run and pass and there are no new failures with python -m test. Before:

[368/368/7] test_zlib
316 tests OK.
7 tests failed:
    test_asyncore test_datetime test_distutils test_email test_ftplib
    test_import test_tcl
2 tests altered the execution environment:
    test___all__ test_builtin
43 tests skipped:
    test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
    test_codecmaps_kr test_codecmaps_tw test_crypt test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue test_largefile
    test_nis test_openpty test_ossaudiodev test_pipes test_poll
    test_posix test_pty test_pwd test_readline test_resource
    test_smtpnet test_socketserver test_syslog test_threadsignals
    test_timeout test_tk test_tools test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_zipfile64
5 skips unexpected on win32:
    test_gdb test_readline test_tk test_tools test_ttk_guionly

After:
315 tests OK.
6 tests failed:
    test_asyncore test_datetime test_distutils test_email test_ftplib
    test_import
4 tests altered the execution environment:
    test___all__ test_builtin test_tcl test_ttk_textonly
43 tests skipped:
    test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
    test_codecmaps_kr test_codecmaps_tw test_crypt test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue test_largefile
    test_nis test_openpty test_ossaudiodev test_pipes test_poll
    test_posix test_pty test_pwd test_readline test_resource
    test_smtpnet test_socketserver test_syslog test_threadsignals
    test_timeout test_tk test_tools test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_zipfile64
3 skips unexpected on win32:
    test_gdb test_readline test_tools

I am puzzled why one less failure and two fewer unexpected skips means one few test ok. Is that because of the two more warnings?

Running with -ugui will cause test_tk and test_ttk_guionly to run instead of skip. So the above patch seems to be a pure win on Windows. What about other systems? And do any of you test experts have anything better?
History
Date User Action Args
2012-07-19 23:21:52terry.reedysetrecipients: + terry.reedy, pitrou, ocean-city, ezio.melotti, r.david.murray, michael.foord, brian.curtin
2012-07-19 23:21:51terry.reedysetmessageid: <1342740111.43.0.0586923435631.issue10652@psf.upfronthosting.co.za>
2012-07-19 23:21:50terry.reedylinkissue10652 messages
2012-07-19 23:21:47terry.reedycreate