classification
Title: test_socket failures on Mac OS X
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: baikie, brett.cannon, michael.foord, ncoghlan, ned.deily, python-dev, r.david.murray, ronaldoussoren
Priority: normal Keywords:

Created on 2011-09-11 05:40 by ncoghlan, last changed 2014-02-01 10:21 by ned.deily. This issue is now closed.

Repositories containing patches
http://hg.python.org/sandbox/ncoghlan#sendmsg
Messages (14)
msg143854 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-11 05:40
Several of the new socket tests introduced in #6560 are failing on the Mac OS X buildbots. These appear to be due to platform problems rather than anything in the test suite or the new sendmsg/recvmsg code, but it would be good if a developer on Mac OS X could take a closer look at them.

To avoid polluting the buildbot results, the affected tests will be flagged as expected failures when run on Mac OS X (this isn't in the repo yet, but once it is, the tests will be marked with "anticipate_failure" in the test_socket code).
msg143860 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-11 11:12
First attempt didn't quite work - the FD passing tests somehow seem to be reporting both 'ERROR' *and* 'expected failure', which is causing the test overall to remain red.

http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%20custom/builds/41/steps/test/logs/stdio
msg143863 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-11 12:54
Ah, I believe I see why the expected failure isn't working properly - the failing testFDPass* tests are causing the subsequent tear down code to also fail.
msg144241 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-18 11:35
OK, I've now looked into *why* the socket tests are throwing errors in tearDown, and it has to do with the way the threaded client/server tests in test_socket are set up.

Specifically, ThreadableTest uses tearDown to reraise any exception raised in the client thread, and these are therefore outside the scope of the "expectedFailure" suppression in unittest.

Now that I've tracked this down, it would be fairly straightforward to fix this specifically within test_socket.ThreadableTest by appropriately adjusting the definition of ThreadableTest.clientRun to discard exceptions encountered in tests flagged as expected failures.

However, I'm wondering if that's the right thing to do. Perhaps it would make more sense to change unittest itself so that "expectedFailure" also suppresses tearDown errors. It doesn't seem all that unusual for a known failing test to also cause problems for the tearDown code.

Added Michael to the nosy list to ask for his advice/opinion. In the meantime, I'll work on a patch that adjusts ThreadableTest directly.
msg144248 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2011-09-18 18:15
See issue 10548. There is some resistance to expectedFailure masking errors in setUp/tearDown as these aren't the place where you would normally expect the expected failure...
msg144257 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-18 23:15
OK, I'll just deal with the problem directly in test_socket then.

It looks like my latest attempt (suppressing unittest._ExpectedFailure in test_socket.ThreadableTest.clientRun) did the trick, so I'll push the updated tests some time this evening:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%202%20custom/builds/44/steps/test/logs/stdio
msg144264 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-09-19 10:26
New changeset a4e4facad164 by Nick Coghlan in branch 'default':
Close issue 12958 by flagging expected failures in test_socket on Mac OS X
http://hg.python.org/cpython/rev/a4e4facad164
msg159694 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-04-30 13:53
New changeset e64bec91ac91 by Richard Oudkerk in branch 'default':
Issue #14669: Skip multiprocessing connection pickling test on MacOSX
http://hg.python.org/cpython/rev/e64bec91ac91
msg187635 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-04-23 12:45
Not sure what changed, but these tests are reported as failures when I run 'make test' and not as expected failures.
msg187637 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-23 12:55
Ronald: in 3.4 only?  If so I wonder if the subtests patch had an unintended side effect...I'm pretty sure it touched the expected failure machinery somehow.
msg187642 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-04-23 13:22
Yes, only on 3.4:


======================================================================
ERROR: testFDPassSeparate (test.test_socket.RecvmsgSCMRightsStreamTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 261, in _tearDown
    raise exc
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 273, in clientRun
    test_func()
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 2644, in _testFDPassSeparate
    array.array("i", [fd1]))]),
  File "/Users/ronald/Projects/python/rw/default/Lib/test/test_socket.py", line 1790, in sendmsgToServer
    *(args + self.sendmsg_to_server_defaults[len(args):]))
OSError: [Errno 22] Invalid argument


(And simular for the other 3 marked socket tests)
msg187643 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-04-23 13:28
It's entirely possible the hack in test_socket (where it catches unittest._ExpectedFailure to suppress the issues in the tearDown code after the test fails) were invalidated by the subtest changes.
msg187646 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-04-23 14:37
FWIW I've filed a bug report about the behavior of sendmsg with Apple, the RADAR number is 13716133.
msg209884 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-02-01 10:21
FYI, the OS X platform bugs causing the three "send timeout" test failures were fixed in OS X 10.7 causing those test cases to have previously silently skipped "unexpected successes".  See Issue20474 for more details.
History
Date User Action Args
2014-02-01 10:21:12ned.deilysetmessages: + msg209884
2013-04-23 14:37:54ronaldoussorensetmessages: + msg187646
2013-04-23 13:28:23ncoghlansetmessages: + msg187643
2013-04-23 13:22:41ronaldoussorensetmessages: + msg187642
2013-04-23 12:55:09r.david.murraysetnosy: + r.david.murray
messages: + msg187637
2013-04-23 12:45:47ronaldoussorensetnosy: + ronaldoussoren

messages: + msg187635
versions: + Python 3.4
2012-04-30 13:53:07python-devsetmessages: + msg159694
2011-09-19 10:26:46python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg144264

resolution: fixed
stage: resolved
2011-09-18 23:15:52ncoghlansetmessages: + msg144257
2011-09-18 18:15:54michael.foordsetmessages: + msg144248
2011-09-18 11:35:06ncoghlansetassignee: ncoghlan

messages: + msg144241
nosy: + michael.foord
2011-09-12 20:33:20baikiesetnosy: + baikie
2011-09-11 20:12:38brett.cannonsetnosy: + brett.cannon
2011-09-11 12:54:02ncoghlansetmessages: + msg143863
2011-09-11 11:12:31ncoghlansethgrepos: + hgrepo67
messages: + msg143860
2011-09-11 08:49:24ned.deilysetnosy: + ned.deily
2011-09-11 05:46:03ncoghlanlinkissue6560 superseder
2011-09-11 05:40:43ncoghlancreate