classification
Title: buildbot: test_cmd_line failure on Tiger: [Errno 9] Bad file descriptor
Type: behavior Stage:
Components: Library (Lib), Tests Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: db3l, gregory.p.smith, ncoghlan, ned.deily, pitrou, ronaldoussoren, vstinner
Priority: normal Keywords: buildbot

Created on 2010-04-19 11:53 by vstinner, last changed 2011-01-06 00:18 by pitrou. This issue is now closed.

Messages (14)
msg103583 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-19 11:53
The error only occured once, and it was not reproduced when test_cmd_line was rerunning in verbose mode.

http://www.python.org/dev/buildbot/builders/x86 Tiger 3.x/builds/25/steps/test/logs/stdio

test test_cmd_line failed -- Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_cmd_line.py", line 58, in test_q
    self.verify_valid_flag('-Qold')
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_cmd_line.py", line 49, in verify_valid_flag
    data = self.start_python(cmd_line)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_cmd_line.py", line 34, in start_python
    p = spawn_python(*args)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/script_helper.py", line 29, in spawn_python
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/subprocess.py", line 694, in __init__
    self.stdin = io.open(p2cwrite, 'wb', bufsize)
OSError: [Errno 9] Bad file descriptor

Re-running test test_cmd_line in verbose mode
test_directories (test.test_cmd_line.CmdLineTest) ... ok
test_large_PYTHONPATH (test.test_cmd_line.CmdLineTest) ... ok
test_optimize (test.test_cmd_line.CmdLineTest) ... ok
test_q (test.test_cmd_line.CmdLineTest) ... ok
test_run_code (test.test_cmd_line.CmdLineTest) ... ok
test_run_module (test.test_cmd_line.CmdLineTest) ... ok
test_run_module_bug1764407 (test.test_cmd_line.CmdLineTest) ... ok
test_site_flag (test.test_cmd_line.CmdLineTest) ... ok
test_unbuffered_input (test.test_cmd_line.CmdLineTest) ... ok
test_unbuffered_output (test.test_cmd_line.CmdLineTest) ... ok
test_usage (test.test_cmd_line.CmdLineTest) ... ok
test_verbose (test.test_cmd_line.CmdLineTest) ... ok
test_version (test.test_cmd_line.CmdLineTest) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.860s

OK
msg103584 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-19 11:54
See also #8457.
msg103598 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-04-19 14:15
I have also seen this test failure occasionally but only on OS X 10.4 systems, not 10.5 or 10.6.
msg121275 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-11-16 09:20
With 3.2a4, OS X 10.4 Tiger buildbot failures reported:


======================================================================
ERROR: test_run_code (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_cmd_line.py", line 95, in test_run_code
    assert_python_failure('-c')
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/script_helper.py", line 55, in assert_python_failure
    return _assert_python(False, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/script_helper.py", line 29, in _assert_python
    env=env)
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/subprocess.py", line 683, in __init__
    self.stdin = io.open(p2cwrite, 'wb', bufsize)
OSError: [Errno 9] Bad file descriptor

======================================================================
ERROR: test_run_module (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_cmd_line.py", line 72, in test_run_module
    assert_python_failure('-m')
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/script_helper.py", line 55, in assert_python_failure
    return _assert_python(False, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/script_helper.py", line 29, in _assert_python
    env=env)
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/subprocess.py", line 683, in __init__
    self.stdin = io.open(p2cwrite, 'wb', bufsize)
OSError: [Errno 9] Bad file descriptor

======================================================================
ERROR: test_version (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_cmd_line.py", line 48, in test_version
    rc, out, err = assert_python_ok('-V')
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/script_helper.py", line 48, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/script_helper.py", line 29, in _assert_python
    env=env)
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/subprocess.py", line 683, in __init__
    self.stdin = io.open(p2cwrite, 'wb', bufsize)
OSError: [Errno 9] Bad file descriptor

On my 10.4 system, it seems to be a race condition: sometimes I see all three of these failures, sometimes just one, sometimes none.  Again, only on 10.4 (Tiger), not 10.5 or 10.6.  But the 10.4 machine I'm using is by far the slowest of the three so it is possible that could be a factor.  Perhaps there is a race condition with cleaning up the p2c pipe from a previous run?
msg124320 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-12-18 21:16
See also msg124254.
msg124321 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-18 21:19
> Perhaps there is a race condition with cleaning up the p2c pipe from a 
> previous run?

This would be rather strange, since it would have a different file descriptor. Or it means there's a bug somewgere (perhaps in subprocess, or perhaps even in io.open).
It someone can reproduce even intermittently on their home machine, it would be nice to dig a bit deeper.
msg125260 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-04 00:34
The code surrounding the p2c file descriptors in subprocess has changed a bit recently since this issue was filed.  someone still needs to come up with a way to reproduce it.
msg125263 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-04 00:41
Well, it occurred today again :)
http://www.python.org/dev/buildbot/all/builders/x86%20Tiger%203.x/builds/1884/steps/test/logs/stdio

test test_cmd_line failed -- Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_cmd_line.py", line 95, in test_run_code
    assert_python_failure('-c')
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/script_helper.py", line 57, in assert_python_failure
    return _assert_python(False, *args, **env_vars)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/script_helper.py", line 30, in _assert_python
    env=env)
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/subprocess.py", line 721, in __init__
    self.stdin = io.open(p2cwrite, 'wb', bufsize)
OSError: [Errno 9] Bad file descriptor


But I would suggest an OS X bug instead. My random guess is that the child process exits very quickly (as you see, it runs "python -c" which is an immediate error), therefore the other end of the pipe gets closed and OS X then returns EBADF when we try to do something with our end of the pipe.
msg125277 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-04 01:43
that race condition, if thats what this is, is likely not OS X specific.  the child process could complete or close its fds before we reach that code on any OS.  We either need to guard these io.open(p2c*...) lines against EBADF OSError's or we should create the io wrappers before _execute_child() is called.
msg125306 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-04 11:15
> that race condition, if thats what this is, is likely not OS X
> specific.  the child process could complete or close its fds before we
> reach that code on any OS.  We either need to guard these
> io.open(p2c*...) lines against EBADF OSError's or we should create the
> io wrappers before _execute_child() is called.

Creating them before _execute_child() is the obvious and simple solution
(if my diagnosis is correct). Is there any downside to doing so?
msg125352 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-04 18:47
Off the top of my head I don't _think_ so.. at least for posix.  I'm not sure about the implications of the if mswindows: code in there though.
msg125354 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-04 18:53
Well we could try a run of it on the buildbots, then :)
msg125357 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-04 19:07
Trying it in r87744.
msg125486 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-06 00:18
I'm closing it in the hope that it's fixed. If it reappears, we'll have to reopen.
History
Date User Action Args
2011-01-06 00:18:15pitrousetstatus: open -> closed

messages: + msg125486
resolution: fixed
nosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
2011-01-04 19:07:21pitrousetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg125357
2011-01-04 18:53:30pitrousetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg125354
2011-01-04 18:47:38gregory.p.smithsetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg125352
2011-01-04 11:15:04pitrousetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg125306
2011-01-04 01:43:32gregory.p.smithsetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg125277
2011-01-04 00:41:05pitrousetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg125263
2011-01-04 00:34:44gregory.p.smithsetassignee: gregory.p.smith ->
messages: + msg125260
nosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
2010-12-18 21:19:46pitrousetnosy: gregory.p.smith, db3l, ronaldoussoren, ncoghlan, pitrou, vstinner, ned.deily
messages: + msg124321
2010-12-18 21:16:58ned.deilysetnosy: + pitrou, gregory.p.smith, ncoghlan
messages: + msg124320

assignee: gregory.p.smith
type: behavior
2010-12-18 21:16:51ned.deilylinkissue10727 superseder
2010-11-16 09:20:39ned.deilysetnosy: + ronaldoussoren
messages: + msg121275
2010-04-19 14:15:59ned.deilysetnosy: + ned.deily
messages: + msg103598
2010-04-19 11:54:08vstinnersetmessages: + msg103584
2010-04-19 11:53:45vstinnercreate