On official Python3.2 beta1 windows binary, I noticed following
command fails. (test_tcl alone won't fail)

I couldn't reproduce this on binary built from source without
installation.

//////////////////////////////////////////////////////////////

C:\Python32>.\python.exe -m test.regrtest -v test___all__ test_tcl

(snip)

[2/2] test_tcl
testCall (test.test_tcl.TclTest) ... ERROR
testCallException (test.test_tcl.TclTest) ... ERROR
testCallException2 (test.test_tcl.TclTest) ... ERROR
testEval (test.test_tcl.TclTest) ... ERROR
testEvalException (test.test_tcl.TclTest) ... ERROR
testEvalException2 (test.test_tcl.TclTest) ... ERROR
testEvalFile (test.test_tcl.TclTest) ... ERROR
testEvalFileException (test.test_tcl.TclTest) ... ERROR
testGetVar (test.test_tcl.TclTest) ... ERROR
testGetVarArray (test.test_tcl.TclTest) ... ERROR
testGetVarArrayException (test.test_tcl.TclTest) ... ERROR
testGetVarException (test.test_tcl.TclTest) ... ERROR
testLoadWithUNC (test.test_tcl.TclTest) ... ERROR
testPackageRequireException (test.test_tcl.TclTest) ... ERROR
testSetVar (test.test_tcl.TclTest) ... ERROR
testSetVarArray (test.test_tcl.TclTest) ... ERROR
testUnsetVar (test.test_tcl.TclTest) ... ERROR
testUnsetVarArray (test.test_tcl.TclTest) ... ERROR
testUnsetVarException (test.test_tcl.TclTest) ... ERROR
testFlattenLen (test.test_tcl.TkinterTest) ... ok

======================================================================
ERROR: testCall (test.test_tcl.TclTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python32\lib\test\test_tcl.py", line 25, in setUp
    self.interp = Tcl()
  File "C:\Python32\lib\tkinter\__init__.py", line 1768, in Tcl
    return Tk(screenName, baseName, className, useTk)
  File "C:\Python32\lib\tkinter\__init__.py", line 1674, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, want
objects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories:
    C:/Python32/lib/tcl8.5 C:/lib/tcl8.5 C:/lib/tcl8.5 C:/library C:/library C:/
tcl8.5.2/library C:/tcl8.5.2/library



This probably means that Tcl wasn't installed properly.


(snip)

----------------------------------------------------------------------
Ran 20 tests in 1.752s

FAILED (errors=19)
test test_tcl failed -- multiple errors occurred
1 test failed:
    test_tcl
1 test altered the execution environment:
    test___all__

test___all__ imports a lot of modules, but I found one of
following modules can bring same error.

# just import one of these in test_main(test___all__)

#    import idlelib.AutoComplete
#    import tkinter.scrolledtext
#    import tkinter.ttk
#    import turtle

//////////////////////////////////////////////////////

And actually, "import tkinter" is enough.

I think this happens because 

1. test___all__.py imports tkinter module, and it
   imports tkinter/_fix.py
2. _fix.py sets TCL_LIBRARY etc as top level routine
3. regrtest.py resets os.environ after test___all__.py ends.
   so TCL_LIBRARY gone.
4. test_tcl.py tries to import tkinter module, but it won't
   be loaded twice, so TCL_LIBRARY won't be set.
5. tcl/tk cannot find its library

I think this can be fixed by save & restore sys.modules
before & after each tests, so that tkinter module can be
loaded again.

How about this patch? Is this kind of *fix* acceptable?

# I hope this works.

I think this test disabling and failure issue should get some attention.
Do the failures happen on non-Windows systems too?

With installed 3.3.0b1 on Win7, test___all__, test_tcl, test_tk, test_ttk_textonly, and test_ttk_guionly all run if run first (_tk and _ttk_guionly with -ugui). All modify os.environ, resulting in
Warning -- os.environ was modified by test_xxx
and all disable at least one of the other tests run afterward. (test___all__ disables but is not disabled itself.) In standard alphabetic running order, test___all__ 'wins'.

I would like to add test/test_idle, #15392, but I presume it currently would also not run after test___all__ or any of the others.

---
I patched my installation, but without the unneeded temporary (saved_values = self.saved_values is only an optimization for the repeated access in the loop that follows).
        sys.modules.clear()
        sys.modules.update(self.saved_modules)
        del self.saved_modules

With this,
python -m test -ugui test___all__ test_tcl test_tk test_ttk_guionly test_ttk_textonly
runs and passes all five tests instead of two (test___all__ and test_ttk_textonly).

Unfortunately, it causes new failures running the entire test suite:
test_concurrent_futures (ok before)
test_decimal (ok before)
test_email (numberous failures, just one before)
I stopped at test_faulthandler, there might be more but this was enough to reject the patch as is. 

I decided to try the principle of doing the minimum necessary to get the tests to pass. I arrived at the following, which lets all five tests run and pass, at least when run in the given order. 

        try:
            del sys.modules['tkinter']
            del sys.modules['tkinter._fix']
            del sys.modules['tkinter.ttk']
            del sys.modules['tkinter.test']
            del sys.modules['tkinter.test.support']
            del sys.modules['tkinter.test.runtktests']
        except KeyError:
            pass

The only new failure with python -m test is test_multi-processing, which normally does not even give a warning.

[197/368/6] test_multiprocessing
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing
test test_multiprocessing failed -- Traceback (most recent call last):
  File "C:\Programs\Python33\lib\test\test_multiprocessing.py", line 1909, in test_mymanager_context_prestarted
    self.assertEqual(manager._process.exitcode, 0)
AssertionError: -15 != 0

This might be an unrelated, intermittant failure.

Since the exception above will trigger for nearly all tests (costly), and since the above does not guarantee to delete all tkinter.* modules, and since future tkinter tests might expand to test other subpackages and idle tests will import other subpackages, I tried this in __exit__.

        if 'tkinter' in sys.modules:
            sysmod = sys.modules
            tknames = set()
            for name in sysmod:
                if name.startswith('tkinter'):
                    tknames.add(name)
            for name in tknames:
                del sysmod[name]

So aside from the initial test, the extra time is only used when needed. All five tests run and pass and there are no new failures with python -m test. Before:

[368/368/7] test_zlib
316 tests OK.
7 tests failed:
    test_asyncore test_datetime test_distutils test_email test_ftplib
    test_import test_tcl
2 tests altered the execution environment:
    test___all__ test_builtin
43 tests skipped:
    test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
    test_codecmaps_kr test_codecmaps_tw test_crypt test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue test_largefile
    test_nis test_openpty test_ossaudiodev test_pipes test_poll
    test_posix test_pty test_pwd test_readline test_resource
    test_smtpnet test_socketserver test_syslog test_threadsignals
    test_timeout test_tk test_tools test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_zipfile64
5 skips unexpected on win32:
    test_gdb test_readline test_tk test_tools test_ttk_guionly

After:
315 tests OK.
6 tests failed:
    test_asyncore test_datetime test_distutils test_email test_ftplib
    test_import
4 tests altered the execution environment:
    test___all__ test_builtin test_tcl test_ttk_textonly
43 tests skipped:
    test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
    test_codecmaps_kr test_codecmaps_tw test_crypt test_curses
    test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll test_fcntl
    test_fork1 test_gdb test_grp test_ioctl test_kqueue test_largefile
    test_nis test_openpty test_ossaudiodev test_pipes test_poll
    test_posix test_pty test_pwd test_readline test_resource
    test_smtpnet test_socketserver test_syslog test_threadsignals
    test_timeout test_tk test_tools test_ttk_guionly test_urllib2net
    test_urllibnet test_wait3 test_wait4 test_winsound test_xmlrpc_net
    test_zipfile64
3 skips unexpected on win32:
    test_gdb test_readline test_tools

I am puzzled why one less failure and two fewer unexpected skips means one few test ok. Is that because of the two more warnings?

Running with -ugui will cause test_tk and test_ttk_guionly to run instead of skip. So the above patch seems to be a pure win on Windows. What about other systems? And do any of you test experts have anything better?

I would like to see my patch (cut and paste from message above) or something like it in the next beta. It fixes the problem (since forever?) of tcl/tk/ttk tests not running properly (at least on windows) as part of the test suite. (I presume the buildbots have extra skips in order to be green.) I am reluctant to apply myself without review simply because I am ignorant enough to miss something that others might see wrong.

It would be much better to have an actual patch to review.  Then, we need someone who can test it on Windows.

I wonder if it would be sensible to have test___all__ restore the state of sys.modules after it runs.

> I wonder if it would be sensible to have test___all__ restore the
> state of sys.modules after it runs.

Probably not. I don't think we want to debug the consequences of having
duplicate modules lying around by way of old references registered here
and there.

Yeah, we probably do not want to go there, even though technically I think those would be bugs.  But bugs that it would only be "nice" to fix, not necessary to fix.

I tested this on Windows in my installed 3.3.0b1. Copying the inserted lines to my repository copy (where I cannot test it) gives the attached patch. On 3.2.3, the insertion line would be 1056 instead of 1158.

Hirokazu's original patch was to restore sys.modules after *every* test, not just sys.modules. That *did* cause problems because (at minimum) of references in importlib. Restoring only after __all__ would only let test_tcl run but not test_tk. And that is assuming alphabetic, not randomized order.

I was recently bitten by this issue.  I've reviewed Terry's patch on Rietveld, and have tested it myself some.  It seems to get the job done without getting in the way.

Would using import_fresh_module() in test_tcl (and others) work?

It does, in fact.  Here's a patch.

Once I worked around the example.bat problems so repository _tkinter will run for one python version (3.2), Zach's patch seems to solve the test problem as well as mine, and it is better self-contained. In other words,
python -m test -ugui test___all__ test_tcl test_tk test_ttk_guionly test_ttk_textonly
runs all five tests with os.environ change the only problem noted.

If Ezio or someone else who knows testing better than me agrees that it is the right patch, I am willing to apply it.

LGTM.

The changes on os.environ should be unrelated.  I've seen a few other tests (e.g. test_distutils) that change it on Windows, and started writing a patch for it, but I never finished it.

After editing 2.7 files to match Zach's patch, and also after adding the changes in my patch, and also deleting tcltk directory and rerunning external.bat to rebuild tcltk/ for 2.7 with tcl/tk 8.5.2,
the tests still do not all work right.

--------------
F:\Python\dev\py27\PCbuild> python_d -m test.regrtest -ugui test___all__ test_tcl test_tk test_ttk_guionly test_ttk_text
only

test___all__
Warning -- os.environ was modified by test___all__

test_tcl
test test_tcl failed -- multiple errors occurred; run in verbose mode for details

test_tk
test_tk skipped -- tk not available: Can't find a usable init.tcl in the following directories:
    F:/Python/dev/py27/lib/tcl8.5 F:/Python/dev/py27/lib/tcl8.5 F:/Python/dev/lib/tcl8.5 F:/Python/dev/py27/library F:/P
ython/dev/library F:/Python/dev/tcl8.5.2/library F:/Python/tcl8.5.2/library
This probably means that Tcl wasn't installed properly.

------
If test_tcl is run first, it works, test_ttk_guionly gives the same message as test_tk, test_ttk_testonly is OK. So the attached patch, 10652_tkfix_27.diff is an improvement, but not a complete fix.

I am going to concentrate on 3.x (which means manually delete tcltk and re-compile) and apply if all goes well.

New changeset b923234b60cb by Terry Jan Reedy in branch '3.2':
Issue # 10652: make tcl/tk tests run after __all__ test, patch by Zachary Ware.
http://hg.python.org/cpython/rev/b923234b60cb

New changeset 596e8855895e by Terry Jan Reedy in branch '3.3':
Issue # 10652: make tcl/tk tests run after __all__ test, patch by Zachary Ware.
http://hg.python.org/cpython/rev/596e8855895e

New changeset 7c76b70075db by Terry Jan Reedy in branch 'default':
#10652 null merge from 3.3 which had wrong issue #
http://hg.python.org/cpython/rev/7c76b70075db

Terry, I just tested your 2.7 patch, and it does work, but the workaround that (I think) we have both been using on 3.x to find the Tcl/Tk .dlls (copying them into PCbuild) doesn't work on 2.7 for some reason.  However, if you add ..\tcltk\bin to PATH, which the PCbuild\rt.bat script (called by Tools\buildbot\test(-amd64).bat) does, your patch allows all of the test modules to run properly.

I wonder if this has anything to do with issue17883's buildbot problems?

Having another chance to look at this one, my previous message was incorrect; Terry's patch does not fix the issue for an installed Python 2.7.  The only change it needs to work, though, is to replace 'Tkinter' with 'FixTk' in the import_fresh_module call.

An alternative that works and also removes repeated "Warning -- os.environ was modified by test_*" is to import FixTk at the top of test_support, allowing the environment to be set up and to persist throughout all of the tests.  I'm not sure if this is the right way to go about the problem, though.

Zach, do you have any further thoughts in light of patches pushed since? 
What do you think is the exact remaining issue?

As for what's actually wrong here, Hirokazu Yamamoto's diagnosis in msg123615 (adjusted for 2.7) is correct.

Either of the last two patches I posted should work to fix this issue, but they're both just band-aids rather than a real, once-and-for-all fix.  #20035 (which I need to rewrite again) will be a once-and-for-all fix for 3.5 by getting rid of tkinter._fix, but I'm not sure if such an invasive fix is appropriate for 2.7 and 3.4.  I prefer the second band-aid (import FixTk at the top of test_support) just because it's simpler and also prevents the 'os.environ has been changed' warnings.

A workaround that doesn't require a patch is to just set TCL_LIBRARY manually in your environment before running the tests, which is how the 3.x buildbots are currently working (see Tools/buildbot/test.bat:4).

For the record, I'm not sure why the 3.x fix we came up with earlier in this issue worked, though I suspect it has something to do with _fix being part of the tkinter package.  The same patch fails on 2.7 because Tkinter is not a package; FixTk is a standalone module and is thus completely unaffected by support.import_fresh_module('Tkinter').  Fresh-importing FixTk itself works, since it's what we actually need to run.

The root of this issue was fixed by #20035 for 3.5+.  Terry, do you still have issues with this with 2.7, and is it still worth trying to fix?

I believe the following means 'No' and that you can close this.

f:\dev\27>python -m test.regrtest test___all__ test_tcl
Running Debug|Win32 interpreter...
Run tests sequentially
0:00:00 [1/2] test___all__
0:00:24 [2/2] test_tcl
All 2 tests OK.

Excellent.  Thanks, Terry!