classification
Title: enhancement: add assertDuration context manager to unittest module
Type: Stage: resolved
Components: Library (Lib) Versions:
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: matt-davis, pablogsal, serhiy.storchaka, steven.daprano
Priority: normal Keywords:

Created on 2020-09-15 02:09 by matt-davis, last changed 2020-09-15 09:33 by steven.daprano. This issue is now closed.

Messages (5)
msg376923 - (view) Author: Matthew Davis (matt-davis) Date: 2020-09-15 02:09
# Summary

I propose an additional unit test type for the unittest module.
TestCase.assertDuration(min=None, max=None), which is a context manager, similar to assertRaises. It runs the code inside it, and then fails the test if the duration of the code inside was not between min and max.


# Use case

I want to check that when I call a certain function in my application, it doesn't take far more time than I expect, or far less time.

e.g. if I'm trying to do things concurrently, I want to test that they are indeed concurrent, by measuring whether the duration equals the sum of all processes' duration, or the max.

# MWE

```
import unittest
from time import sleep, time
from multiprocessing import Pool

def slow(x):
    sleep(x)
    
# blocking sleep for 0, 1, 2, 3 seconds, concurrently
def together():
    with Pool(4) as p:
        p.map(slow, range(4))

class TestConcurrent(unittest.TestCase):
    # this is how you do it today
    def test_today(self):
        start = time()
        together()
        end = time()
        duration = end - start
        self.assertGreaterEqual(duration, 2)
        # max should be 3 seconds, plus some overhead
        # if together() called slow() in series,
        # total duration would be 0 + 1 + 2 + 3 = 6 > 4
        self.assertLessEqual(duration, 4) 
        
    # this is how I want to do it
    def test_simpler(self):
        with self.assertDuration(min=2, max=4):
            together()
```

# Solution

I just need to add a new context manager next to this one:
https://github.com/python/cpython/blob/6b34d7b51e33fcb21b8827d927474ce9ed1f605c/Lib/unittest/case.py#L207
msg376930 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-15 07:45
I think this can be very error prone because a particular operation could be interrupted by GIL switching in the middle of the context manager and be reported as taking much more than the execution of the code itself is taking. This will be very confused to users that expect it to "just work"
msg376932 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-09-15 08:46
I concur with Pablo. It is too unreliable.
msg376933 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-15 09:13
Thanks for the proposal Matthew, unfortunately, this addition is not reliable enough for the standard library and OTOH can be easily implemented in-situ for the users that know what the limitations are.
msg376935 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2020-09-15 09:33
Matt, you can add this to your own unit tests by just subclassing unittest.TestCase and adding a new assertDuration method. Copy the existing method's implementations (its open source and you should have the source code already, but if not you can find it here:

https://github.com/python/cpython/blob/3.8/Lib/unittest/__init__.py

If you are looking for some timing code to use (in case you don't already have your own) you can try this:

https://github.com/ActiveState/code/tree/master/recipes/Python/577896_Benchmark_code
History
Date User Action Args
2020-09-15 09:33:51steven.dapranosetnosy: + steven.daprano
messages: + msg376935
2020-09-15 09:13:02pablogsalsetmessages: + msg376933
2020-09-15 09:11:15pablogsalsetstatus: open -> closed
resolution: rejected
stage: resolved
2020-09-15 08:46:40serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg376932
2020-09-15 07:45:39pablogsalsetnosy: + pablogsal
messages: + msg376930
2020-09-15 02:09:28matt-daviscreate