classification
Title: test_asyncio: relax timings even more
Type: behavior Stage:
Components: asyncio, Tests Versions: Python 3.4
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, jcea, ncoghlan, skrah, vstinner, yselivanov
Priority: normal Keywords: buildbot

Created on 2014-01-21 22:29 by skrah, last changed 2014-06-06 13:38 by vstinner. This issue is now closed.

Messages (10)
msg208714 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-21 22:29
It seems that the relaxed timings in some tests are still not
sufficient:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/7417/steps/test/logs/stdio

======================================================================
FAIL: test_time_and_call_at (test.test_asyncio.test_base_events.BaseEventLoopTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_asyncio/test_base_events.py", line 127, in test_time_and_call_at
    self.assertTrue(0.09 <= t1-t0 <= 0.9, t1-t0)
AssertionError: False is not true : 2.06036564335227
msg208745 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2014-01-22 04:31
I won't have time to fix this (and bardly to even review a fix) but this seems a straightforward fix and you can go ahead without me -- maybe you can get someone else with some asyncio experience to review your patch (e.g. Antoine or Victor).
msg208906 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-23 10:33
I'm seeing other unrelated timeouts on the OpenIndiana machines.
Jesús, are the machines perhaps operating under an unusually high load?
msg208909 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-23 10:43
> I'm seeing other unrelated timeouts on the OpenIndiana machines.

I read somewhere that the host of these virtual machines is *highly* loaded (load > 6). It's more a buildbot configuration issue than a test_asyncio issue.
msg208911 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-23 10:46
I see. That's astonishing, because usually the builds are ultra-fast
on those machines.
msg208912 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-23 10:53
Tests failing because of timings is really annoying. It would be nice to add something in test.support to configure timings. A function can maybe wait until the system load is lower than a threshold, or we should add a parameter to configure a tolerance between the perfect timing and the expected timing.

The tolerance can be a simple factor, a floating point number >= 1.0.

For example, replace:
    0.09 <= t1-t0 <= 0.9, t1-t0
with
    0.1 - 0.01 * factor <= t1-t0 <= 0.1 + 0.8 * factor
or a new test function:
    check_timing(t1-t0, 0.1, 0.01, 0.8)
msg208913 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-01-23 10:59
+1 for doing something about timing failures. Perhaps also a
decorator @skip_if_load_higher_than(x).


[Nick, I'm adding you because our current test suite could lead
to "commit impossible" situations with Zuul.]
msg208939 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-01-23 14:32
I have a draft PEP for the Zuul idea, and yes, it notes the increased consequences of declaring a buildbot stable when it sometimes isn't. There's a reason OpenStack treats third party test results as advisory only rather than as gating criteria :P
msg219882 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-06-06 13:15
Let's open a new issue for system load detection. This one is not asyncio specific.
msg219883 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-06 13:38
> Let's open a new issue for system load detection. This one is not asyncio specific.

I opened issues #20910 and #20964 for example.
History
Date User Action Args
2014-06-06 13:38:03vstinnersetmessages: + msg219883
2014-06-06 13:15:10skrahsetstatus: open -> closed
resolution: not a bug
messages: + msg219882

stage: needs patch ->
2014-06-06 11:41:41vstinnersetnosy: + yselivanov, gvanrossum
components: + asyncio
2014-01-23 14:32:42ncoghlansetmessages: + msg208939
2014-01-23 10:59:32skrahsetnosy: + ncoghlan
messages: + msg208913
2014-01-23 10:53:01vstinnersetmessages: + msg208912
2014-01-23 10:46:30skrahsetnosy: - gvanrossum
messages: + msg208911
2014-01-23 10:43:20vstinnersetnosy: + vstinner
messages: + msg208909
2014-01-23 10:33:42skrahsetnosy: + jcea
messages: + msg208906
2014-01-22 04:31:32gvanrossumsetmessages: + msg208745
2014-01-21 22:29:51skrahcreate