Message165888
I think this test disabling and failure issue should get some attention.
Do the failures happen on non-Windows systems too?
With installed 3.3.0b1 on Win7, test___all__, test_tcl, test_tk, test_ttk_textonly, and test_ttk_guionly all run if run first (_tk and _ttk_guionly with -ugui). All modify os.environ, resulting in
Warning -- os.environ was modified by test_xxx
and all disable at least one of the other tests run afterward. (test___all__ disables but is not disabled itself.) In standard alphabetic running order, test___all__ 'wins'.
I would like to add test/test_idle, #15392, but I presume it currently would also not run after test___all__ or any of the others.
---
I patched my installation, but without the unneeded temporary (saved_values = self.saved_values is only an optimization for the repeated access in the loop that follows).
sys.modules.clear()
sys.modules.update(self.saved_modules)
del self.saved_modules
With this,
python -m test -ugui test___all__ test_tcl test_tk test_ttk_guionly test_ttk_textonly
runs and passes all five tests instead of two (test___all__ and test_ttk_textonly).
Unfortunately, it causes new failures running the entire test suite:
test_concurrent_futures (ok before)
test_decimal (ok before)
test_email (numberous failures, just one before)
I stopped at test_faulthandler, there might be more but this was enough to reject the patch as is.
I decided to try the principle of doing the minimum necessary to get the tests to pass. I arrived at the following, which lets all five tests run and pass, at least when run in the given order.
try:
del sys.modules['tkinter']
del sys.modules['tkinter._fix']
del sys.modules['tkinter.ttk']
del sys.modules['tkinter.test']
del sys.modules['tkinter.test.support']
del sys.modules['tkinter.test.runtktests']
except KeyError:
pass
The only new failure with python -m test is test_multi-processing, which normally does not even give a warning.
[197/368/6] test_multiprocessing
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing
test test_multiprocessing failed -- Traceback (most recent call last):
File "C:\Programs\Python33\lib\test\test_multiprocessing.py", line 1909, in test_mymanager_context_prestarted
self.assertEqual(manager._process.exitcode, 0)
AssertionError: -15 != 0
This might be an unrelated, intermittant failure.
Since the exception above will trigger for nearly all tests (costly), and since the above does not guarantee to delete all tkinter.* modules, and since future tkinter tests might expand to test other subpackages and idle tests will import other subpackages, I tried this in __exit__.
if 'tkinter' in sys.modules:
sysmod = sys.modules
tknames = set()
for name in sysmod:
if name.startswith('tkinter'):
tknames.add(name)
for name in tknames:
del sysmod[name]
So aside from the initial test, the extra time is only used when needed. All five tests run and pass and there are no new failures with python -m test. Before:
[368/368/7] test_zlib
316 tests OK.
7 tests failed:
test_asyncore test_datetime test_distutils test_email test_ftplib
test_import test_tcl
2 tests altered the execution environment:
test___all__ test_builtin
43 tests skipped:
test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
test_codecmaps_kr test_codecmaps_tw test_crypt test_curses
test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
test_fork1 test_gdb test_grp test_ioctl test_kqueue test_largefile
test_nis test_openpty test_ossaudiodev test_pipes test_poll
test_posix test_pty test_pwd test_readline test_resource
test_smtpnet test_socketserver test_syslog test_threadsignals
test_timeout test_tk test_tools test_ttk_guionly test_urllib2net
test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
test_zipfile64
5 skips unexpected on win32:
test_gdb test_readline test_tk test_tools test_ttk_guionly
After:
315 tests OK.
6 tests failed:
test_asyncore test_datetime test_distutils test_email test_ftplib
test_import
4 tests altered the execution environment:
test___all__ test_builtin test_tcl test_ttk_textonly
43 tests skipped:
test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
test_codecmaps_kr test_codecmaps_tw test_crypt test_curses
test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
test_fork1 test_gdb test_grp test_ioctl test_kqueue test_largefile
test_nis test_openpty test_ossaudiodev test_pipes test_poll
test_posix test_pty test_pwd test_readline test_resource
test_smtpnet test_socketserver test_syslog test_threadsignals
test_timeout test_tk test_tools test_ttk_guionly test_urllib2net
test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
test_zipfile64
3 skips unexpected on win32:
test_gdb test_readline test_tools
I am puzzled why one less failure and two fewer unexpected skips means one few test ok. Is that because of the two more warnings?
Running with -ugui will cause test_tk and test_ttk_guionly to run instead of skip. So the above patch seems to be a pure win on Windows. What about other systems? And do any of you test experts have anything better? |
|
Date |
User |
Action |
Args |
2012-07-19 23:21:52 | terry.reedy | set | recipients:
+ terry.reedy, pitrou, ocean-city, ezio.melotti, r.david.murray, michael.foord, brian.curtin |
2012-07-19 23:21:51 | terry.reedy | set | messageid: <1342740111.43.0.0586923435631.issue10652@psf.upfronthosting.co.za> |
2012-07-19 23:21:50 | terry.reedy | link | issue10652 messages |
2012-07-19 23:21:47 | terry.reedy | create | |
|