Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In developer mode (-X dev), ResourceWarning is only emited once per line numbers #76270

Closed
vstinner opened this issue Nov 20, 2017 · 14 comments
Closed
Labels
3.7 (EOL) end of life

Comments

@vstinner
Copy link
Member

BPO 32089
Nosy @ncoghlan, @pitrou, @vstinner, @serhiy-storchaka
PRs
  • bpo-32089: Fix warnings filters in dev mode #4482
  • bpo-32089: Use default action for ResourceWarning #4584
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-11-27.11:14:41.427>
    created_at = <Date 2017-11-20.14:57:20.398>
    labels = ['3.7']
    title = 'In developer mode (-X dev), ResourceWarning is only emited once per line numbers'
    updated_at = <Date 2017-11-27.11:14:41.426>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-11-27.11:14:41.426>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-11-27.11:14:41.427>
    closer = 'vstinner'
    components = []
    creation = <Date 2017-11-20.14:57:20.398>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 32089
    keywords = ['patch']
    message_count = 14.0
    messages = ['306548', '306603', '306609', '306635', '306637', '306638', '306639', '306641', '306646', '306868', '306905', '306946', '307049', '307050']
    nosy_count = 4.0
    nosy_names = ['ncoghlan', 'pitrou', 'vstinner', 'serhiy.storchaka']
    pr_nums = ['4482', '4584']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue32089'
    versions = ['Python 3.7']

    @vstinner
    Copy link
    Member Author

    The -X dev mode currently *hides* some ResourceWarning warnings:

    $ cat x.py 
    def func():
        open('/etc/issue')
    func()
    func()
    $ ./python x.py 
    x.py:2: ResourceWarning: unclosed file <_io.TextIOWrapper name='/etc/issue' mode='r' encoding='UTF-8'>
      open('/etc/issue')
    x.py:2: ResourceWarning: unclosed file <_io.TextIOWrapper name='/etc/issue' mode='r' encoding='UTF-8'>
      open('/etc/issue')

    haypo@selma$ ./python -X dev x.py
    x.py:2: ResourceWarning: unclosed file <_io.TextIOWrapper name='/etc/issue' mode='r' encoding='UTF-8'>
    open('/etc/issue')

    The problem is that the "-W default" inserted by -X dev overrides the final filter on ResourceWarning:

    $ ./python -X dev -c 'import warnings, pprint; pprint.pprint(warnings.filters)'
    [('default',
      re.compile('', re.IGNORECASE),
      <class 'Warning'>,
      re.compile(''),
      0),
     ('ignore', None, <class 'BytesWarning'>, None, 0),
     ('always', None, <class 'ResourceWarning'>, None, 0)]

    @vstinner vstinner added the 3.7 (EOL) end of life label Nov 20, 2017
    @vstinner
    Copy link
    Member Author

    Attached PR 4482 fixes warnings filters:

    haypo@selma$ ./python -X dev -c 'import warnings, pprint; pprint.pprint(warnings.filters)'
    [('ignore', None, <class 'BytesWarning'>, None, 0),
    ('always', None, <class 'ResourceWarning'>, None, 0),
    ('default', None, <class 'Warning'>, None, 0)]

    @vstinner
    Copy link
    Member Author

    New changeset 09f3a8a by Victor Stinner in branch 'master':
    bpo-32089: Fix warnings filters in dev mode (bpo-4482)
    09f3a8a

    @pitrou
    Copy link
    Member

    pitrou commented Nov 21, 2017

    I don't agree with this. You're comparing "-X dev" with a pydebug build of Python, not with a normal Python.

    Your example shows the problem. If a program calls func() 1000 times in a row, they will show the same warning 1000 times. What does it bring to have the exact same warning (and line number) displayed 1000 times instead of once? Nothing. There is a reason the filter "default" isn't the same as "always", and it's that "always" can be completely overwhelming.

    @pitrou pitrou reopened this Nov 21, 2017
    @vstinner
    Copy link
    Member Author

    If a program calls func() 1000 times in a row, they will show the same warning 1000 times. What does it bring to have the exact same warning (and line number) displayed 1000 times instead of once? Nothing. There is a reason the filter "default" isn't the same as "always", and it's that "always" can be completely overwhelming.

    I don't know the rationale of the "always" action rather than "default". The choice was made 7 years ago, when Georg Brandl implemented the new ResourceWarning category: commit 08be72d.

    The problem of ResourceWarning is that the warning logs where the last reference to the resource was cleared, or where the a garbage collection was triggered. Multiple resources can be created from different places, but all "die" at the same place.

    In Python 3.6, I enhanced ResourceWarning to log also the traceback where the leaked resource has been created, if the tracemalloc is tracing memory allocations. When tracemalloc is used, it's useful to log ResourceWarning multiple times even if the warning itself is logged multipletimes. Well, that's an unusual usage of the ResourceWarning, we can also suggest to use -W always::ResourceWarning in that case.

    I don't agree with this. You're comparing "-X dev" with a pydebug build of Python, not with a normal Python.

    In general, I would like to offer the same experience in "development mode" (-X dev) and with a pydebug build.

    For ResourceWarning, your rationale only concerns pydebug build, no?

    @pitrou
    Copy link
    Member

    pitrou commented Nov 21, 2017

    Le 21/11/2017 à 14:46, STINNER Victor a écrit :

    I don't know the rationale of the "always" action rather than "default".

    Neither do I. But I don't think it matters a lot. pydebug builds are
    almost only used by Python core developers. "developer mode" is target
    at every Python developer.

    The problem of ResourceWarning is that the warning logs where the last reference to the resource was cleared, or where the a garbage collection was triggered. Multiple resources can be created from different places, but all "die" at the same place.

    If they have different names, they will be logged separately.

    >>> def f(fname):
    ...     open(fname)
    ...
    >>> f('setup.py')
    __main__:2: ResourceWarning: unclosed file <_io.TextIOWrapper
    name='setup.py' mode='r' encoding='UTF-8'>
    >>> f('setup.py')
    >>> f('LICENSE.txt')
    __main__:2: ResourceWarning: unclosed file <_io.TextIOWrapper
    name='LICENSE.txt' mode='r' encoding='UTF-8'>
    >>> f('LICENSE.txt')
    >>>

    For ResourceWarning, your rationale only concerns pydebug build, no?

    Why? I'm talking about "-X dev", not pydebug builds.

    @pitrou
    Copy link
    Member

    pitrou commented Nov 21, 2017

    I forgot: the snippet above was made with "python -Wdefault::ResourceWarning".

    @serhiy-storchaka
    Copy link
    Member

    See also bpo-27535.

    @vstinner
    Copy link
    Member Author

    Antoine:

    If they have different names, they will be logged separately.

    Oh wow, nice behaviour, I like it :-)

    Antoine:

    > For ResourceWarning, your rationale only concerns pydebug build, no?
    Why? I'm talking about "-X dev", not pydebug builds.

    Oops sorry, typo (brain error), I mean: "also concerns".

    Serhiy:

    See also bpo-27535.

    Oh, not leaking memory in warnings registies is a nice side effect of the "always" action :-)

    --

    It's much more complicated than what I expected to get the "correct behaviour" for warnings :-/

    @vstinner
    Copy link
    Member Author

    Antoine: What do you think of using the "default" action for pydebug build and -X dev?

    @pitrou
    Copy link
    Member

    pitrou commented Nov 24, 2017

    Antoine: What do you think of using the "default" action for pydebug build and -X dev?

    That would be fine with me.

    @ncoghlan
    Copy link
    Contributor

    +1 from me for using "default" instead of "always" for ResourceWarning.

    Folks can always combine "-X tracemalloc" with "-W always::ResourceWarning" if want to ensure they see absolutely every resource warning, rather than only representative ones.

    @vstinner
    Copy link
    Member Author

    New changeset 21c7730 by Victor Stinner in branch 'master':
    bpo-32089: Use default action for ResourceWarning (bpo-4584)
    21c7730

    @vstinner
    Copy link
    Member Author

    The "default warning filters" are documented, but only for Python compiled in release mode:
    https://docs.python.org/dev/library/warnings.html#default-warning-filters

    So I didn't touch this documentation.

    The initial issue is now fixed: the development mode (-X dev) now behaves as the debug mode (pydebug build) for the default warning filters.

    The second issue, Antoine's complain about the "always" action, was also fixed.

    I close the issue. At the end, we should get a better debug experience ;-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants