classification
Title: Fix buildbot testing of Tkinter
Type: behavior Stage: resolved
Components: Tests, Tkinter, Windows Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: zach.ware Nosy List: Todd.Rovito, amaury.forgeotdarc, brian.curtin, ezio.melotti, jkloth, loewis, pitrou, python-dev, roger.serwy, terry.reedy, trent, zach.ware
Priority: normal Keywords: patch

Created on 2013-04-30 21:05 by zach.ware, last changed 2013-11-04 18:28 by zach.ware. This issue is now closed.

Files
File name Uploaded Description Edit
issue17883-skip-on-error.diff zach.ware, 2013-06-25 16:48 Skip on error
issue17883-add_is_gui_available.diff zach.ware, 2013-06-25 21:37 Add _is_gui_available to 2.7 test.test_support
Messages (30)
msg188184 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-04-30 21:05
Ezio, you brought up the fact that the 2.7 Windows buildbots were skipping the Tkinter tests in Terry's core-mentorship thread about tkinter on Windows last month, and I've finally learned what is causing that failure.  It appears that when 689a813f4afc and df39b0bc4106 were committed, they were not backported to 2.7.  Attached is a patch to backport those two changesets, which should make things (more) right.  

This only affects running tests via PCbuild/rt.bat (or Tools/buildbot/test.bat, which calls PCbuild/rt.bat), and doesn't do anything for finding the tcl/tk .dlls any other time.
msg188456 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-05 19:36
New changeset 3c58fa7dc7f1 by Ezio Melotti in branch '2.7':
#17883: Fix buildbot testing of Tkinter on Windows.  Patch by Zachary Ware.
http://hg.python.org/cpython/rev/3c58fa7dc7f1
msg188464 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-05 20:26
The buildbots run the tests and seem happy.
Thanks for the report and the patch!
msg188473 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-05 22:06
Actually one buildbot is failing: http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%202.7/builds/427/steps/test/logs/stdio
msg188488 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-05-06 03:16
I wonder if there is something special about the environment on that Snakebite machine. At the top, it says 
 PATH=C:\Perl\site\bin;C:\Perl\bin;E:\apps\activestate-python-2.7.2.5-x86\;E:\apps\activestate-python-2.7.2.5-x86\Scripts;...
Could the presence of activestate-python affect the build?

Ignoring the 'no buffer in 3.0' warnings, there are a couple of warnings that suggest updates to test_tcl.py. Another issue though.

The context for the failure is
        with test_support.EnvironmentVarGuard() as env:
            env.unset("TCL_LIBRARY")
            f = os.popen('%s -c "import Tkinter; print Tkinter"' % (unc_name,))
        self.assertTrue('Tkinter.py' in f.read())

I do not know why test_tcl is testing the ability of python to be opened from a unc path (I am not familiar with them), but anyway...

The assertion is testing the read of the printed Tkinter module representation. On my 2.7.4 install, the result is "<module 'Tkinter' from 'C:\Programs\Python27\lib\lib-tk\Tkinter.pyc'>". On my machine, the test reads that and passes. (On 3.x, the test is ('tkinter' in f.read()) as the file is tkinter/__init__.py instead of Tkinter.py.)

To see what is going on with the failing machine, an update to
        self.assertIn('Tkinter.py', f.read())
will show the f.read() that does not contain 'Tkinter.py'.
msg188489 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-06 03:22
New changeset b1abc5800e2b by Terry Jan Reedy in branch '2.7':
Issue17883: Update to assertIn to see why test fails on one buildbot.
http://hg.python.org/cpython/rev/b1abc5800e2b
msg188562 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-05-06 16:14
Here's the relevant bit of the output from that buildbot after Terry's change:

======================================================================
FAIL: testLoadWithUNC (test.test_tcl.TclTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\test\test_tcl.py", line 151, in testLoadWithUNC
    self.assertIn('Tkinter.py', f.read())
AssertionError: 'Tkinter.py' not found in ''
----------------------------------------------------------------------

...which doesn't say much.  Here's a patch that updates the test to use subprocess.Popen instead of os.popen and add some useful debugging information.  The test passes on my machine with and without the patch, so I can't do much more debugging until we get some useful output from the buildbot.
msg188594 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-05-06 21:10
Here's a better patch after learning a bit better how subprocess.Popen works...
msg188595 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-06 21:14
FTR there's also test.support.assert_python_ok and test.support.assert_python_failure (and a few convenience functions in subprocess too).  I suspect that if the import fails, stdout is empty, and stderr will contain the error message.
msg188621 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-07 06:35
New changeset 63f2941477d3 by Ezio Melotti in branch '2.7':
#17833: add debug output to investigate buildbot failure.
http://hg.python.org/cpython/rev/63f2941477d3
msg188676 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-07 17:57
Here's the traceback:
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%202.7/builds/435/steps/test/logs/stdio
======================================================================
ERROR: testLoadWithUNC (test.test_tcl.TclTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\test\test_tcl.py", line 152, in testLoadWithUNC
    p = Popen(cmd, stdout=PIPE, stderr=PIPE)
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "E:\Data\buildslave\cpython\2.7.snakebite-win2k3r2sp2-x86\build\lib\subprocess.py", line 948, in _execute_child
    startupinfo)
WindowsError: [Error 5] Access is denied

----------------------------------------------------------------------
Ran 21 tests in 0.578s

The buildbot also hangs on test_ttk_guionly.
msg188790 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-05-09 20:28
I'm not sure where to go with this from here, without knowing more about what's going wrong.  Just to make sure there wasn't anything wrong with the Tcl/Tk build or some manner of issue with things not being properly cleaned up or something, I set the 'custom' builder on that buildbot to build cpython#2.7 and got the same result: http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%20custom/builds/9

I've got a few changes lined up to try to get some more output (namely, add a 'print cmd' to the test_tcl test, and hack up Tools/buildbot/test.bat to just run the Tcl/Tk tests in verbose mode to try and get something from test_ttk_guionly), but I don't think we want to pollute the main repository with them.  Which direction should we go from here?
msg188792 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-05-09 22:24
Amaury and Martin: there is a problem on one machine with test.test_tcl.TclTest.testLoadWithUNC that you two jointly wrote, reviewed, and applied in #6470.

This issue (17883) is above getting the tcl/tk/tkinter tests to actually be run on 2.7, which they were not before. Zack's patch succeeded in doing that, and the tests all pass on all buildbots except (if I understand correctly) on one Snakebite machine "x86 Windows Server 2003 [SB] 2.7". 

(According to http://buildbot.python.org/all/builders/, the only other Server-2.7 bot is x86 Windows Server 2008 [SB] 2.7, and that seems not to have run these tests: the most recent(?) log says "test_tk skipped -- DLL load failed: The specified module could not be found.". But if I understand, this was back in January.)

With the original
   f = os.popen('%s -c "import Tkinter; print Tkinter"' % (unc_name,))
f.read() returns '', so 'Tkinter' is not found in the null string and the assertion that it is fails.

To get more info -- from an error traceback presumed to be on stderr -- we tried using subprocess.POpen instead and the child process startup fails with "WindowsError: [Error 5] Access is denied", so the Python import is never tried. Unless there is an error in the subprocess call or if Win Servers require a different version of UNC, it would seem that the test is not at fault. 

a) Is Windows Server 2003 is really meant to be spported?
b) Are UNCs expected to behave differently on Server 2003?
c) Can UNCs be disabled on a particular machine?
In other words, should we just ignore this error, and possibly catch it with 'except WinError"?
msg188811 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-05-10 06:39
> a) Is Windows Server 2003 is really meant to be spported?

Yes

> b) Are UNCs expected to behave differently on Server 2003?

No.

> c) Can UNCs be disabled on a particular machine?

I may misunderstand "can": yes, it is possible, but no, it is not desirable.

Zach: Temporarily committing changes to test specific buildbot issues is fine; asking the slave operator for direct access to the machine may also work.
msg188845 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-05-10 16:00
Thanks for your response, Martin.

Martin v. Löwis wrote:
> Zach: Temporarily committing changes to test specific buildbot issues
> is fine

I suppose it would also be beneficial to get some output from the other bots which are *not* failing, to have something to compare with.  In that case, here's a (very) temporary patch we can try for a round of builds.

> asking the slave operator for direct access to the machine 
> may also work.

Trent, what's your policy on letting non-committers play around in your machines? :)  Alternately, do you know of anything about that machine that could be causing the failures?
msg191423 - (view) Author: Jeremy Kloth (jkloth) * Date: 2013-06-18 18:27
It seems that the changes committed by Ezio are causing the AMD64 Windows buildbot to hang in test_ttk_guionly.

A note that this buildbot is running as a service so no desktop is available.  The same test however is working (skipped successfully) on the 3.3 branch, if that helps.
msg191426 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-06-18 18:57
Jeremy, can you give me the output of a test run with issue17883-tmp-test.diff applied?  (Or just what a `python -m test -v test_ttk_guionly` gives with ..\tcltk\bin on the PATH)
msg191435 - (view) Author: Jeremy Kloth (jkloth) * Date: 2013-06-18 21:16
(Note that the buildbot is 64-bit so the paths need to be tcltk64)

The test_ttk_guionly passed successfully *from the command-line*.  It is currently hung in that test from the buildbot scheduler *as a service*.

I do not know of a simple way to test changes via the buildbot service.
msg191867 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-06-25 16:48
Ok, I've managed to cobble together a buildbot setup to test the hang myself and have come up with a couple possible solutions.

The first is to change the buildbot service on the slaves to allow desktop interaction.  This way, the tests can actually run.  If that's not possible, it's probably simplest to just pass -u-gui to the buildslaves in question, since there is in fact no gui available.  That should at least fix the test_ttk_guionly problem.

As for the test_tcl problem, how about this patch?  It simply wraps the Popen call in a try/except, and converts to a skip in case of an 'Access is denied' error.
msg191873 - (view) Author: Jeremy Kloth (jkloth) * Date: 2013-06-25 18:36
Although the solution for test_ttk_guionly of disabling the gui resource is a good quick fix, I'm curious as to why there isn't the same issue (test hangs) when run using 3.3 on the same machine.

Note that by running as a service without desktop interaction several assumptions wrt permissions have been uncovered, so I'm quite hesitant to flip that switch.
msg191885 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-06-25 21:07
> Jeremy Kloth added the comment:
>
> Although the solution for test_ttk_guionly of disabling the gui resource is a good quick fix, I'm curious as to why there isn't the same issue (test hangs) when run using 3.3 on the same machine.

As it turns out, 3.3(+)'s test.support has an _is_gui_available
function that prevents the hang, added in issue9931.  Ironically, it
was not added to 2.7 because the 2.7 buildbots weren't showing
symptoms...because the tests weren't actually being run.

I'll work on trying to backport the patch that added
_is_gui_available, but my experience with ctypes is practically nil,
so if anyone can get to it before I can post a patch, don't hold off
on my account.
msg191886 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-06-25 21:37
It seems all it took to backport was a copy and paste.  Here's the patch.
msg195972 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-08-23 14:14
Ping!

The buildbots still seem to be failing, are my proposed fixes acceptable?  Both patches still apply cleanly.
msg201848 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-10-31 20:42
Buildbots are still hanging on test_ttk_guionly and failing test_tcl and both patches still apply cleanly.

Is there anything I can do to help move this forward again?
msg201989 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-03 03:28
If there are no objections, I'd like to commit both of these in the next day or two (which is nice to be able to say :)).
msg202066 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2013-11-03 23:18
The changes look fine to me.
msg202085 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-11-04 04:27
New changeset 358496e67a89 by Zachary Ware in branch '2.7':
Issue #17883: Backport test.test_support._is_gui_available()
http://hg.python.org/cpython/rev/358496e67a89
msg202086 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-11-04 04:52
New changeset 72c3ca3ed22a by Zachary Ware in branch '2.7':
Issue #17883: Tweak test_tcl testLoadWithUNC to skip the test in the
http://hg.python.org/cpython/rev/72c3ca3ed22a
msg202087 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-04 04:56
I'll leave this open until the buildbots prove happy.

I did make a couple extra changes to the patch to TclTest.testLoadWithUNC to make a couple of skip conditions actually report a skip instead of just returning, and to remove a superfluous 'import sys'.
msg202152 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-04 18:28
The buildbots appear to be happy, no more hanging and no more testLoadWithUNC failure.  There is a failure in test_tcl on one bot, but it is not related to this issue.
History
Date User Action Args
2013-11-04 18:28:50zach.waresetstatus: open -> closed
resolution: fixed
messages: + msg202152

stage: patch review -> resolved
2013-11-04 04:56:51zach.waresetmessages: + msg202087
2013-11-04 04:52:47python-devsetmessages: + msg202086
2013-11-04 04:27:50python-devsetmessages: + msg202085
2013-11-03 23:18:43brian.curtinsetassignee: terry.reedy -> zach.ware
messages: + msg202066
2013-11-03 03:28:26zach.waresetnosy: + brian.curtin

messages: + msg201989
stage: resolved -> patch review
2013-10-31 20:42:32zach.waresetmessages: + msg201848
2013-10-31 20:38:15zach.waresetfiles: - issue17883-tmp-test.diff
2013-10-31 20:38:07zach.waresetfiles: - issue17883.diff
2013-10-31 20:38:03zach.waresetfiles: - 2.7_rt.bat_tcltk_fix.diff
2013-08-24 01:43:07ezio.melottisetassignee: ezio.melotti -> terry.reedy
2013-08-23 14:14:18zach.waresetmessages: + msg195972
2013-06-25 21:37:51zach.waresetfiles: + issue17883-add_is_gui_available.diff

messages: + msg191886
2013-06-25 21:07:19zach.waresetmessages: + msg191885
2013-06-25 18:36:38jklothsetmessages: + msg191873
2013-06-25 16:48:31zach.waresetfiles: + issue17883-skip-on-error.diff

nosy: + pitrou
messages: + msg191867

resolution: fixed -> (no value)
2013-06-18 21:16:55jklothsetmessages: + msg191435
2013-06-18 18:57:17zach.waresetmessages: + msg191426
2013-06-18 18:27:08jklothsetnosy: + jkloth
messages: + msg191423
2013-05-10 16:00:43zach.waresetfiles: + issue17883-tmp-test.diff
nosy: + trent
messages: + msg188845

2013-05-10 06:39:06loewissetmessages: + msg188811
2013-05-09 22:24:14terry.reedysetnosy: + loewis, amaury.forgeotdarc
messages: + msg188792
2013-05-09 20:28:26zach.waresetmessages: + msg188790
2013-05-07 17:57:26ezio.melottisetmessages: + msg188676
2013-05-07 06:37:30ezio.melottisetmessages: + msg188621
2013-05-07 02:10:59Todd.Rovitosetnosy: + Todd.Rovito
2013-05-06 21:14:39ezio.melottisetmessages: + msg188595
2013-05-06 21:10:55zach.waresetfiles: - issue17883.diff
2013-05-06 21:10:42zach.waresetfiles: + issue17883.diff

messages: + msg188594
2013-05-06 16:14:15zach.waresetfiles: + issue17883.diff

messages: + msg188562
2013-05-06 03:22:39python-devsetmessages: + msg188489
2013-05-06 03:16:50terry.reedysetmessages: + msg188488
2013-05-05 22:06:39ezio.melottisetstatus: closed -> open

messages: + msg188473
2013-05-05 20:26:15ezio.melottisetstatus: open -> closed
messages: + msg188464

assignee: ezio.melotti
resolution: fixed
stage: patch review -> resolved
2013-05-05 19:36:18python-devsetnosy: + python-dev
messages: + msg188456
2013-04-30 21:16:28ezio.melottisetnosy: + roger.serwy

stage: patch review
2013-04-30 21:05:15zach.warecreate