classification
Title: `catch_warnings` context manager causes all warnings to be printed every time, even after exiting
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Gerrit.Holl, Segev Finer, erik.bray, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2017-02-27 22:29 by Gerrit.Holl, last changed 2018-07-10 21:16 by Segev Finer.

Pull Requests
URL Status Linked Edit
PR 8232 open Segev Finer, 2018-07-10 21:16
Messages (9)
msg288677 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2017-02-27 22:29
Entering the `catch_warnings` context manager is causing warnings to be printed over and over again, rather than just once as it should.   Without such a context manager, the behaviour is as expected:

    $ cat ./mwe.py 
    #!/usr/bin/env python3.5
    
    import warnings
    
    for i in range(3):
        try:
            print(i, __warningregistry__)
        except NameError:
            print(i, "first warning")
        warnings.warn("I don't like this.", UserWarning)
        print(i, __warningregistry__)
    #    with warnings.catch_warnings():
    #        pass
        print(i, __warningregistry__)
        warnings.warn("I don't like this.", UserWarning)
    $ ./mwe.py 
    0 first warning
    ./mwe.py:10: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    0 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 10): True}
    0 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 10): True}
    ./mwe.py:15: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    1 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    1 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    1 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    2 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    2 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    2 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}

Uncommenting the context manager causes the warning to be printed every time:

    $ ./mwe.py 
    0 first warning
    ./mwe.py:10: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    0 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 10): True}
    0 {'version': 0, ("I don't like this.", <class 'UserWarning'>, 10): True}
    ./mwe.py:15: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    1 {'version': 2, ("I don't like this.", <class 'UserWarning'>, 15): True}
    ./mwe.py:10: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    1 {'version': 2, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    1 {'version': 2, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    ./mwe.py:15: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    2 {'version': 4, ("I don't like this.", <class 'UserWarning'>, 15): True}
    ./mwe.py:10: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)
    2 {'version': 4, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    2 {'version': 4, ("I don't like this.", <class 'UserWarning'>, 15): True, ("I don't like this.", <class 'UserWarning'>, 10): True}
    ./mwe.py:15: UserWarning: I don't like this.
      warnings.warn("I don't like this.", UserWarning)

I think this is undesirable.  There is code deep inside a module that I'm using that is using the context manager, and as a consequence, I get warnings printed every time that should be printed only once.

See also on Stack Overflow: http://stackoverflow.com/q/42496579/974555
msg288678 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2017-02-27 22:36
I suppose this is a consequence of the change in:

http://bugs.python.org/issue4180

and although I can see why the warnings filter would be reset when entering the context manager, I don't think it is desirable to have side effects for modules that are entirely unrelated.
msg288680 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2017-02-27 22:42
To clarify, this behaviour crosses module boundaries.  So even if some piece of code many layers deep buried in a module uses this context manager, it has global consequences.
msg288709 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2017-02-28 12:29
To resolve this, the `catch_warnings` context manager upon exiting should not only reset `filters`, but also `_filters_version`, and perhaps an associated dictionary?  Not sure if I'm understanding the code in `warnings.c` correctly, and whether, for this change, it would suffice to change `warnings.py` or whether `warnings.c` would need to be updated as well.
msg298588 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2017-07-18 14:18
I get bitten by this frequently, and find myself patching many libraries just to avoid them from calling .catch_warnings.

Does anyone know whether to fix this it would suffice to edit warnings.py, or would I need to dig into _warnings.c as well?
msg298822 - (view) Author: Segev Finer (Segev Finer) * Date: 2017-07-21 20:18
I think you will need to save and restore _filters_version and _onceregistry. Since _filters_version is a static variable in _warnings.c you will probably also have to modify it somehow to make it accessible to warnings.py.
msg320662 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-06-28 11:56
I just encountered this as well.  The way catch_warnings is implemented is a bit "dumb" in how it treats _filter_version (it calls _filters_mutated even if the active filters are not actually changed when entering catch_warnings).

More significantly, _filter_version is not fine-grained enough.  If some warning was already displayed, calling catch_warnings() should not later cause that same warning to be displayed again unless the filters were modified in such a way, during catch_warnings(), that that warning should be displayed (e.g. changed to 'always').

I'm not really sure what to do about that though.  Maybe the "filter version" should be per-warning?  Currently the value assigned to each warning in __warningregistry__ is not used (it is just set to True), so maybe that could actually be used for this.
msg321380 - (view) Author: Segev Finer (Segev Finer) * Date: 2018-07-10 11:00
Hmm, I originally missed the per module __warningregistry__... Need to read the code more closely...

What should the behavior be? We can add a flag that will make all warning actions be "always" when catch_warnings is in effect. But that doesn't feel right with the way catch_warnings currently behaves or what is expected of it.

Another idea would be to add a callback that is called whenever we set a module registry, allowing catch_warnings to backup the original registry of each module and restore them all on exit (Also the _onceregistry and _filters_version).
msg321393 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2018-07-10 18:49
Yes, I think `catch_warnings` should back up and restore the relevant `__warningregistry__`.  At which point it's not even clear to me what value there is in the _filter_version...
History
Date User Action Args
2018-07-10 21:16:53Segev Finersetkeywords: + patch
stage: patch review
pull_requests: + pull_request7769
2018-07-10 18:49:26erik.braysetmessages: + msg321393
2018-07-10 11:00:59Segev Finersetmessages: + msg321380
2018-06-28 11:56:17erik.braysetnosy: + erik.bray
messages: + msg320662
2017-07-22 03:47:32serhiy.storchakasetnosy: + pitrou, serhiy.storchaka
2017-07-21 20:18:07Segev Finersetversions: + Python 3.6, Python 3.7
nosy: + Segev Finer

messages: + msg298822

components: + Library (Lib)
type: behavior
2017-07-18 14:20:38Gerrit.Hollsettitle: `catch_warnings` context manager should reset warning registry to previous state upon exiting, to prevent warnings from being reprinted -> `catch_warnings` context manager causes all warnings to be printed every time, even after exiting
2017-07-18 14:18:14Gerrit.Hollsetmessages: + msg298588
2017-02-28 12:29:09Gerrit.Hollsetmessages: + msg288709
2017-02-28 12:21:31Gerrit.Hollsettitle: `catch_warnings` context manager causes warnings to be reprinted -> `catch_warnings` context manager should reset warning registry to previous state upon exiting, to prevent warnings from being reprinted
2017-02-27 22:42:24Gerrit.Hollsetmessages: + msg288680
2017-02-27 22:36:12Gerrit.Hollsetmessages: + msg288678
2017-02-27 22:29:56Gerrit.Hollcreate