Ezio, you brought up the fact that the 2.7 Windows buildbots were skipping the Tkinter tests in Terry's core-mentorship thread about tkinter on Windows last month, and I've finally learned what is causing that failure.  It appears that when 689a813f4afc and df39b0bc4106 were committed, they were not backported to 2.7.  Attached is a patch to backport those two changesets, which should make things (more) right.  

This only affects running tests via PCbuild/rt.bat (or Tools/buildbot/test.bat, which calls PCbuild/rt.bat), and doesn't do anything for finding the tcl/tk .dlls any other time.

New changeset 3c58fa7dc7f1 by Ezio Melotti in branch '2.7':
#17883: Fix buildbot testing of Tkinter on Windows.  Patch by Zachary Ware.
http://hg.python.org/cpython/rev/3c58fa7dc7f1

The buildbots run the tests and seem happy.
Thanks for the report and the patch!

Actually one buildbot is failing: http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%202.7/builds/427/steps/test/logs/stdio

I wonder if there is something special about the environment on that Snakebite machine. At the top, it says 
 PATH=C:\Perl\site\bin;C:\Perl\bin;E:\apps\activestate-python-2.7.2.5-x86\;E:\apps\activestate-python-2.7.2.5-x86\Scripts;...
Could the presence of activestate-python affect the build?

Ignoring the 'no buffer in 3.0' warnings, there are a couple of warnings that suggest updates to test_tcl.py. Another issue though.

The context for the failure is
        with test_support.EnvironmentVarGuard() as env:
            env.unset("TCL_LIBRARY")
            f = os.popen('%s -c "import Tkinter; print Tkinter"' % (unc_name,))
        self.assertTrue('Tkinter.py' in f.read())

I do not know why test_tcl is testing the ability of python to be opened from a unc path (I am not familiar with them), but anyway...

The assertion is testing the read of the printed Tkinter module representation. On my 2.7.4 install, the result is "<module 'Tkinter' from 'C:\Programs\Python27\lib\lib-tk\Tkinter.pyc'>". On my machine, the test reads that and passes. (On 3.x, the test is ('tkinter' in f.read()) as the file is tkinter/__init__.py instead of Tkinter.py.)

To see what is going on with the failing machine, an update to
        self.assertIn('Tkinter.py', f.read())
will show the f.read() that does not contain 'Tkinter.py'.

New changeset b1abc5800e2b by Terry Jan Reedy in branch '2.7':
Issue17883: Update to assertIn to see why test fails on one buildbot.
http://hg.python.org/cpython/rev/b1abc5800e2b

Here's the relevant bit of the output from that buildbot after Terry's change:

======================================================================
FAIL: testLoadWithUNC (test.test_tcl.TclTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\test\test_tcl.py", line 151, in testLoadWithUNC
    self.assertIn('Tkinter.py', f.read())
AssertionError: 'Tkinter.py' not found in ''
----------------------------------------------------------------------

...which doesn't say much.  Here's a patch that updates the test to use subprocess.Popen instead of os.popen and add some useful debugging information.  The test passes on my machine with and without the patch, so I can't do much more debugging until we get some useful output from the buildbot.

Here's a better patch after learning a bit better how subprocess.Popen works...

FTR there's also test.support.assert_python_ok and test.support.assert_python_failure (and a few convenience functions in subprocess too).  I suspect that if the import fails, stdout is empty, and stderr will contain the error message.

New changeset 63f2941477d3 by Ezio Melotti in branch '2.7':
#17833: add debug output to investigate buildbot failure.
http://hg.python.org/cpython/rev/63f2941477d3

Here's the traceback:
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%202.7/builds/435/steps/test/logs/stdio
======================================================================
ERROR: testLoadWithUNC (test.test_tcl.TclTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\test\test_tcl.py", line 152, in testLoadWithUNC
    p = Popen(cmd, stdout=PIPE, stderr=PIPE)
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\subprocess.py", line 948, in _execute_child
    startupinfo)
WindowsError: [Error 5] Access is denied

----------------------------------------------------------------------
Ran 21 tests in 0.578s

The buildbot also hangs on test_ttk_guionly.

I'm not sure where to go with this from here, without knowing more about what's going wrong.  Just to make sure there wasn't anything wrong with the Tcl/Tk build or some manner of issue with things not being properly cleaned up or something, I set the 'custom' builder on that buildbot to build cpython#2.7 and got the same result: http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%20custom/builds/9

I've got a few changes lined up to try to get some more output (namely, add a 'print cmd' to the test_tcl test, and hack up Tools/buildbot/test.bat to just run the Tcl/Tk tests in verbose mode to try and get something from test_ttk_guionly), but I don't think we want to pollute the main repository with them.  Which direction should we go from here?

Amaury and Martin: there is a problem on one machine with test.test_tcl.TclTest.testLoadWithUNC that you two jointly wrote, reviewed, and applied in #6470.

This issue (17883) is above getting the tcl/tk/tkinter tests to actually be run on 2.7, which they were not before. Zack's patch succeeded in doing that, and the tests all pass on all buildbots except (if I understand correctly) on one Snakebite machine "x86 Windows Server 2003 [SB] 2.7". 

(According to http://buildbot.python.org/all/builders/, the only other Server-2.7 bot is x86 Windows Server 2008 [SB] 2.7, and that seems not to have run these tests: the most recent(?) log says "test_tk skipped -- DLL load failed: The specified module could not be found.". But if I understand, this was back in January.)

With the original
   f = os.popen('%s -c "import Tkinter; print Tkinter"' % (unc_name,))
f.read() returns '', so 'Tkinter' is not found in the null string and the assertion that it is fails.

To get more info -- from an error traceback presumed to be on stderr -- we tried using subprocess.POpen instead and the child process startup fails with "WindowsError: [Error 5] Access is denied", so the Python import is never tried. Unless there is an error in the subprocess call or if Win Servers require a different version of UNC, it would seem that the test is not at fault. 

a) Is Windows Server 2003 is really meant to be spported?
b) Are UNCs expected to behave differently on Server 2003?
c) Can UNCs be disabled on a particular machine?
In other words, should we just ignore this error, and possibly catch it with 'except WinError"?

> a) Is Windows Server 2003 is really meant to be spported?

Yes

> b) Are UNCs expected to behave differently on Server 2003?

No.

> c) Can UNCs be disabled on a particular machine?

I may misunderstand "can": yes, it is possible, but no, it is not desirable.

Zach: Temporarily committing changes to test specific buildbot issues is fine; asking the slave operator for direct access to the machine may also work.

Thanks for your response, Martin.

Martin v. Löwis wrote:
> Zach: Temporarily committing changes to test specific buildbot issues
> is fine

I suppose it would also be beneficial to get some output from the other bots which are *not* failing, to have something to compare with.  In that case, here's a (very) temporary patch we can try for a round of builds.

> asking the slave operator for direct access to the machine 
> may also work.

Trent, what's your policy on letting non-committers play around in your machines? :)  Alternately, do you know of anything about that machine that could be causing the failures?

It seems that the changes committed by Ezio are causing the AMD64 Windows buildbot to hang in test_ttk_guionly.

A note that this buildbot is running as a service so no desktop is available.  The same test however is working (skipped successfully) on the 3.3 branch, if that helps.

Jeremy, can you give me the output of a test run with issue17883-tmp-test.diff applied?  (Or just what a `python -m test -v test_ttk_guionly` gives with ..\tcltk\bin on the PATH)

(Note that the buildbot is 64-bit so the paths need to be tcltk64)

The test_ttk_guionly passed successfully *from the command-line*.  It is currently hung in that test from the buildbot scheduler *as a service*.

I do not know of a simple way to test changes via the buildbot service.

Ok, I've managed to cobble together a buildbot setup to test the hang myself and have come up with a couple possible solutions.

The first is to change the buildbot service on the slaves to allow desktop interaction.  This way, the tests can actually run.  If that's not possible, it's probably simplest to just pass -u-gui to the buildslaves in question, since there is in fact no gui available.  That should at least fix the test_ttk_guionly problem.

As for the test_tcl problem, how about this patch?  It simply wraps the Popen call in a try/except, and converts to a skip in case of an 'Access is denied' error.

Although the solution for test_ttk_guionly of disabling the gui resource is a good quick fix, I'm curious as to why there isn't the same issue (test hangs) when run using 3.3 on the same machine.

Note that by running as a service without desktop interaction several assumptions wrt permissions have been uncovered, so I'm quite hesitant to flip that switch.

> Jeremy Kloth added the comment:
>
> Although the solution for test_ttk_guionly of disabling the gui resource is a good quick fix, I'm curious as to why there isn't the same issue (test hangs) when run using 3.3 on the same machine.

As it turns out, 3.3(+)'s test.support has an _is_gui_available
function that prevents the hang, added in issue9931.  Ironically, it
was not added to 2.7 because the 2.7 buildbots weren't showing
symptoms...because the tests weren't actually being run.

I'll work on trying to backport the patch that added
_is_gui_available, but my experience with ctypes is practically nil,
so if anyone can get to it before I can post a patch, don't hold off
on my account.

It seems all it took to backport was a copy and paste.  Here's the patch.

Ping!

The buildbots still seem to be failing, are my proposed fixes acceptable?  Both patches still apply cleanly.

Buildbots are still hanging on test_ttk_guionly and failing test_tcl and both patches still apply cleanly.

Is there anything I can do to help move this forward again?

If there are no objections, I'd like to commit both of these in the next day or two (which is nice to be able to say :)).

The changes look fine to me.

New changeset 358496e67a89 by Zachary Ware in branch '2.7':
Issue #17883: Backport test.test_support._is_gui_available()
http://hg.python.org/cpython/rev/358496e67a89

New changeset 72c3ca3ed22a by Zachary Ware in branch '2.7':
Issue #17883: Tweak test_tcl testLoadWithUNC to skip the test in the
http://hg.python.org/cpython/rev/72c3ca3ed22a

I'll leave this open until the buildbots prove happy.

I did make a couple extra changes to the patch to TclTest.testLoadWithUNC to make a couple of skip conditions actually report a skip instead of just returning, and to remove a superfluous 'import sys'.

The buildbots appear to be happy, no more hanging and no more testLoadWithUNC failure.  There is a failure in test_tcl on one bot, but it is not related to this issue.