classification
Title: Add ability to wholesale silence DeprecationWarnings while running the test suite
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: lukasz.langa, miss-islington, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2021-08-06 16:34 by lukasz.langa, last changed 2021-08-18 12:10 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27634 merged lukasz.langa, 2021-08-06 16:38
PR 27784 merged lukasz.langa, 2021-08-16 18:17
PR 27785 merged lukasz.langa, 2021-08-16 18:27
PR 27793 merged lukasz.langa, 2021-08-17 11:22
PR 27809 merged miss-islington, 2021-08-18 11:19
PR 27810 merged miss-islington, 2021-08-18 11:19
Messages (11)
msg399102 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-06 16:34
Sometimes we have the following problem:
- there are way too many deprecations raised from a given library to silence them one by one;
- the deprecations are different between maintenance branches and we don't want to make tests between the branches hopelessly conflicting.

In particular, there is such a case right now with asyncio in 3.9 vs. later branches. 3.8 deprecated the loop= argument in a bunch of functions but due to poor warning placement, most of them were silent. This is being fixed in BPO-44815 which would go out to users in 3.9.7 but that created 220 new warnings when running test_asyncion in regression tests. Fixing them one by one would be both tedious, and would make the 3.9 branch forever conflicting with newer branches in many asyncio test files. In 3.11 there's a new round of deprecations raised in test_asyncio, making the branches different. Moreover, those warnings are typically silenced by `assertWarns` context managers which should only be used when actually testing the warnings, *not* to silence irrelevant warnings.

So, what the PR does is it introduces:

- `support.ignore_deprecations_from("path.to.module", like=".*msg regex.*")`, and
- `support.clear_ignored_deprecations()`

The former adds a new filter to warnings, the message regex is mandatory. The latter removes only the filters that were added by the former, leaving all other filters alone.

Example usage is in `test_support`, and later, should this be merged, will be in asyncio tests on the 3.9 branch.
msg399104 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-06 16:57
For example, all that's needed to silence the 220 new warnings in all asyncio tests, is adding this in Lib/test/test_asyncio/__init__.py:

```
def setUpModule():
    support.ignore_deprecations_from("asyncio.base_events", like=r".*loop argument.*")
    support.ignore_deprecations_from("asyncio.unix_events", like=r".*loop argument.*")
    support.ignore_deprecations_from("asyncio.futures", like=r".*loop argument.*")
    support.ignore_deprecations_from("asyncio.runners", like=r".*loop argument.*")
    support.ignore_deprecations_from("asyncio.subprocess", like=r".*loop argument.*")
    support.ignore_deprecations_from("asyncio.tasks", like=r".*loop argument.*")
    support.ignore_deprecations_from("test.test_asyncio.test_queues", like=r".*loop argument.*")
    support.ignore_deprecations_from("test.test_asyncio.test_tasks", like=r".*loop argument.*")

def tearDownModule():
    support.clear_ignored_deprecations()
```

Since the `__init__.py` file in question isn't a module that runs tests itself but only a package gathering many sub-modules, we need some boilerplate like:

```
def load_tests(loader, _, pattern):
    pkg_dir = os.path.dirname(__file__)
    suite = AsyncioTestSuite()
    return support.load_package_tests(pkg_dir, loader, suite, pattern)


class AsyncioTestSuite(unittest.TestSuite):
    """A custom test suite that also runs setup/teardown for the whole package.

    Normally unittest only runs setUpModule() and tearDownModule() within each
    test module part of the test suite. Copying those functions to each file
    would be tedious, let's run this once and for all.
    """
    def run(self, result, debug=False):
        try:
            setUpModule()
            super().run(result, debug=debug)
        finally:
            tearDownModule()
```

With that, all of asyncio tests silence unnecessary deprecation warnings. Additionally, testing for warnings with `warnings_helper.check_warnings()` or `assertWarns` still works just fine as those facilities temporarily disable filtering.
msg399118 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-06 18:25
There are problems with clearing all ignored deprecation filters.

We can want to ignore warnings in narrower or wider scope: per-method, per-class, per-module. We should use the warnings.catch_warnings() context manager for restoring filters in the correct scope. For example, use setUp/tearDown, setUpClass/tearDownClass, setUpModule/tearDownModule and save the context manager as an instance/class attribute or module global.

   def setUp(self):
       self.w = warnings.catch_warnings()
       self.w.__enter__()
       warnings.filterwarnings(...)

    def tearDown():
        self.w.__exit__(None, None, None)

It is hard to use any helper here because the code should be added in multiple places.

Or use setUp/addCleanup, setUpClass/addClassCleanup, setUpModule/addModuleCleanup if you want to keep all code in one place:

   def setUp(self):
       w = warnings.catch_warnings()
       w.__enter__()
       self.addCleanup(w.__exit__, None, None, None)
       warnings.filterwarnings(...)

The helper can call catch_warnings(), __enter__(), set filters and return a caller which calls __exit__(None, None, None):

   def setUp(self):
       self.addCleanup(some_helper(...))

For class and method scopes it would be convenient to use class and method decorators correspondingly.

    @ignore_some_warnings(...)
    def test_foo(self):
        ...

@ignore_some_warnings(...)
class BarTests(unittest.TestCase):
    ...
msg399150 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-06 21:11
Serhiy, if you take a look at the code, `clear_ignored_deprecations()` only clears the filters that were added by `ignore_deprecations_from()`. It doesn't touch other filters. Sure, we can abuse the context manager instead but this looks pretty busy:

   def setUp(self):
       self.w = warnings.catch_warnings()
       self.w.__enter__()
       warnings.filterwarnings(...)

    def tearDown():
        self.w.__exit__(None, None, None)

Making the API symmetrical is a good idea though, I'll look into that tomorrow.
msg399156 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-06 21:49
clear_ignored_deprecations() does not know whether filters were added in setUp(), setUpClass(), setUpModule() or just in a particular method. We should use teadDown*() or add*Cleanup() to clean up at appropriate level.
msg399665 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-16 18:13
New changeset a0a6d39295a30434b088f4b66439bf5ea21a3e4e by Łukasz Langa in branch 'main':
bpo-44852: Support ignoring specific DeprecationWarnings wholesale in regrtest (GH-27634)
https://github.com/python/cpython/commit/a0a6d39295a30434b088f4b66439bf5ea21a3e4e
msg399680 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-16 20:47
New changeset c7c4cbc58e18ef5a6f4f377b1ece0a84a54335a7 by Łukasz Langa in branch '3.9':
[3.9] bpo-44852: Support ignoring specific DeprecationWarnings wholesale in regrtest (GH-27634) (GH-27785)
https://github.com/python/cpython/commit/c7c4cbc58e18ef5a6f4f377b1ece0a84a54335a7
msg399728 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-17 10:01
New changeset bc98f981326d7cb30f939dedd04b91f378255d88 by Łukasz Langa in branch '3.10':
[3.10] bpo-44852: Support ignoring specific DeprecationWarnings wholesale in regrtest (GH-27634) (GH-27784)
https://github.com/python/cpython/commit/bc98f981326d7cb30f939dedd04b91f378255d88
msg399836 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-18 11:19
New changeset 8cf07d3db3eed02b43350a5f9dbf68f1c839ea82 by Łukasz Langa in branch 'main':
bpo-44852: Support filtering over warnings without a set message (GH-27793)
https://github.com/python/cpython/commit/8cf07d3db3eed02b43350a5f9dbf68f1c839ea82
msg399840 - (view) Author: miss-islington (miss-islington) Date: 2021-08-18 12:10
New changeset d1c0e4413dd544270df1f5b8a145fd4370cb759b by Miss Islington (bot) in branch '3.10':
bpo-44852: Support filtering over warnings without a set message (GH-27793)
https://github.com/python/cpython/commit/d1c0e4413dd544270df1f5b8a145fd4370cb759b
msg399841 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-18 12:10
New changeset ebe7e6d86cf8f54d449d49866698d7f4c700cc7c by Miss Islington (bot) in branch '3.9':
bpo-44852: Support filtering over warnings without a set message (GH-27793) (GH-27810)
https://github.com/python/cpython/commit/ebe7e6d86cf8f54d449d49866698d7f4c700cc7c
History
Date User Action Args
2021-08-18 12:10:43lukasz.langasetmessages: + msg399841
2021-08-18 12:10:31miss-islingtonsetmessages: + msg399840
2021-08-18 11:19:42miss-islingtonsetpull_requests: + pull_request26277
2021-08-18 11:19:38miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request26276
2021-08-18 11:19:36lukasz.langasetmessages: + msg399836
2021-08-17 11:22:51lukasz.langasetpull_requests: + pull_request26262
2021-08-17 10:01:37lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-08-17 10:01:16lukasz.langasetmessages: + msg399728
2021-08-16 20:47:29lukasz.langasetmessages: + msg399680
2021-08-16 18:27:15lukasz.langasetpull_requests: + pull_request26254
2021-08-16 18:17:28lukasz.langasetpull_requests: + pull_request26253
2021-08-16 18:13:54lukasz.langasetmessages: + msg399665
2021-08-06 21:49:28serhiy.storchakasetmessages: + msg399156
2021-08-06 21:11:24lukasz.langasetmessages: + msg399150
2021-08-06 18:25:28serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg399118
2021-08-06 16:57:50lukasz.langasetmessages: + msg399104
2021-08-06 16:38:17lukasz.langasetkeywords: + patch
pull_requests: + pull_request26128
2021-08-06 16:34:02lukasz.langacreate