classification
Title: test_subprocess failures on awfully slow builtbots
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, barry, neologix, pitrou, r.david.murray
Priority: normal Keywords: buildbot

Created on 2012-06-23 09:07 by neologix, last changed 2014-07-31 18:06 by neologix. This issue is now closed.

Messages (6)
msg163557 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-06-23 09:07
Some test_subprocess tests are failing on really slow buildbots, such as the Ubtuntu ARM one:
"""
======================================================================
ERROR: test_wait_timeout (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/test/test_subprocess.py", line 718, in test_wait_timeout
    self.assertEqual(p.wait(timeout=3), 0)
  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/subprocess.py", line 1494, in wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/python', '-c', 'import time; time.sleep(0.1)']' timed out after 3 seconds

======================================================================
FAIL: test_check_output_timeout (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/test/test_subprocess.py", line 140, in test_check_output_timeout
    self.assertEqual(c.exception.output, b'BDFL')
AssertionError: b'' != b'BDFL'

======================================================================
FAIL: test_check_output_timeout (test.test_subprocess.ProcessTestCaseNoPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/buildarea/3.x.warsaw-ubuntu-arm/build/Lib/test/test_subprocess.py", line 140, in test_check_output_timeout
    self.assertEqual(c.exception.output, b'BDFL')
AssertionError: b'' != b'BDFL'
"""

The timeouts for those tests are already at 3 seconds.
We could double them to 6 seconds and see if things get better: that would increase the running time on all the buildbots, though. Any other idea?
msg163565 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-06-23 09:41
Barry (the buildbot owner) could take a look.
msg163793 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-24 15:13
You could make the test a loop, with the timeout increasing each time through the loop, failing only if all tries fail.  That way on faster machines the test will pass faster.  It'll take even longer on slow machines, but they are slow anyway so that may be acceptable.
msg163996 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-06-25 17:35
> You could make the test a loop, with the timeout increasing each time through the loop, failing only if all tries fail.  That way on faster machines the test will pass faster.  It'll take even longer on slow machines, but they are slow anyway so that may be acceptable.

Yes, sure, but then it'll only make the test non determinist: let's
say there's a non reproductible bug in subprocess' timeout
implementation, if we retry upon error, it won't get caught unless if
fails a couple time in a row, which may never happen.
The idea is to have a timeout large enough so that the probablility of
false positives is *low enough*.
msg221870 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-29 18:05
Does this issue need to be addressed, can it be closed as "out of date" or what?
msg224433 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-07-31 18:06
Closing, I haven't seen this in a while.
History
Date User Action Args
2014-07-31 18:06:34neologixsetstatus: open -> closed
resolution: out of date
messages: + msg224433

stage: needs patch -> resolved
2014-06-29 18:05:20BreamoreBoysetversions: + Python 3.4, Python 3.5, - Python 3.3
nosy: + BreamoreBoy
title: test_subprocess fqailures on awfully slow builtbots -> test_subprocess failures on awfully slow builtbots
messages: + msg221870

2012-06-25 17:35:09neologixsetmessages: + msg163996
2012-06-24 15:13:19r.david.murraysetnosy: + r.david.murray
messages: + msg163793
2012-06-23 09:41:11pitrousetnosy: + barry
messages: + msg163565
2012-06-23 09:07:22neologixcreate