Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unittest.TestCase.assertWarns raises RuntimeEror if sys.modules changes size #73806

Closed
kernc mannequin opened this issue Feb 22, 2017 · 9 comments
Closed

unittest.TestCase.assertWarns raises RuntimeEror if sys.modules changes size #73806

kernc mannequin opened this issue Feb 22, 2017 · 9 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@kernc
Copy link
Mannequin

kernc mannequin commented Feb 22, 2017

BPO 29620
Nosy @akuchling, @ambv, @serhiy-storchaka, @wkschwartz, @kernc, @miss-islington
PRs
  • bpo-29620: iterate over a copy of sys.modules #4800
  • [3.8] bpo-29620: iterate over a copy of sys.modules (GH-4800) #20816
  • [3.7] bpo-29620: iterate over a copy of sys.modules (GH-4800) #20817
  • [3.9] bpo-29620: iterate over a copy of sys.modules (GH-4800) #29605
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-06-11.18:36:06.899>
    created_at = <Date 2017-02-22.16:22:51.894>
    labels = ['3.7', '3.8', 'type-bug', 'library']
    title = 'unittest.TestCase.assertWarns raises RuntimeEror if sys.modules changes size'
    updated_at = <Date 2021-11-18.16:09:52.825>
    user = 'https://github.com/kernc'

    bugs.python.org fields:

    activity = <Date 2021-11-18.16:09:52.825>
    actor = 'lukasz.langa'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-06-11.18:36:06.899>
    closer = 'akuchling'
    components = ['Library (Lib)']
    creation = <Date 2017-02-22.16:22:51.894>
    creator = 'kernc'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 29620
    keywords = ['patch']
    message_count = 9.0
    messages = ['288370', '323821', '356671', '361886', '371315', '371317', '371318', '371319', '406547']
    nosy_count = 8.0
    nosy_names = ['akuchling', 'lukasz.langa', 'serhiy.storchaka', 'William.Schwartz', 'kernc', 'miss-islington', 'Arthit Suriyawongkul', 'Maximilian Peters']
    pr_nums = ['4800', '20816', '20817', '29605']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue29620'
    versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

    @kernc
    Copy link
    Mannequin Author

    kernc mannequin commented Feb 22, 2017

    If any of the objects in sys.modules is a module-like object that performs some additional imports in its __getattribute__ (as appropriate) handler, the following simple unit test test case:

        import unittest
        import warnings
    ... # Ensure one of the imported modules is a module-like object as above
    
        class Case(unittest.TestCase):
            def test_assertWarns(self):
                with self.assertWarns(UserWarning):
                    warnings.warn('Some warning')

    fails with:

    \======================================================================
    ERROR: test_assertWarns (example.Case)
    \----------------------------------------------------------------------
    
        Traceback (most recent call last):
          File "/tmp/example.py", line 9, in test_assertWarns
            with self.assertWarns(UserWarning):
          File "/usr/lib/python3.4/unittest/case.py", line 205, in __enter__
            for v in sys.modules.values():
        RuntimeError: dictionary changed size during iteration
        

    The problem is in the iteration over sys.modules in unittest.case._AssertWarnsContext.__enter__() and accessing every module's __warningregistry__ attribute. On this access, the module-like objects may perform arbitrary actions including importing of further modules which extends sys.modules.

    for v in sys.modules.values():
    if getattr(v, '__warningregistry__', None):

    The simple proposed fix with no foreseen side-effects is to wrap sys.modules.values() call as a tuple().

    @kernc kernc mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Feb 22, 2017
    @wkschwartz
    Copy link
    Mannequin

    wkschwartz mannequin commented Aug 21, 2018

    I am also running into this problem. I'm not 100%, but I'm pretty sure that looping over sys.modules and accessing __warningregistry__ on each module triggers one of my module's __getattr__ functions (PEP-562), which in turn uses setuptools entry points to import an arbitrary set of other modules.

    Bizarrely, however, I cannot reproduce on macOS, only Linux. Presumably there's something platform dependent about how the default unittest test runner orders my tests, and hence which modules have already been loaded by the time I call assertWarnsRegex.

    @serhiy-storchaka serhiy-storchaka added the 3.8 only security fixes label Aug 21, 2018
    @ArthitSuriyawongkul
    Copy link
    Mannequin

    ArthitSuriyawongkul mannequin commented Nov 15, 2019

    Confirmed this behavior on

    • Python 3.6.8 64-bit on Windows Server 2016 (AppVeyor "Visual Studio 2017" build environment)
    • Python 3.6.7 64-bit on Linux

    @MaximilianPeters
    Copy link
    Mannequin

    MaximilianPeters mannequin commented Feb 12, 2020

    Ran into the same issue using Python 3.6.8 [GCC 5.4.0 20160609] on Linux.

    The same code worked on Windows 10 with Python 3.7 32-bit and 64-bit.

    @akuchling
    Copy link
    Member

    New changeset 46398fb by kernc in branch 'master':
    bpo-29620: iterate over a copy of sys.modules (GH-4800)
    46398fb

    @akuchling
    Copy link
    Member

    New changeset 3e499cd by Miss Islington (bot) in branch '3.8':
    bpo-29620: iterate over a copy of sys.modules (GH-4800) (GH-20816)
    3e499cd

    @akuchling
    Copy link
    Member

    New changeset f881c86 by Miss Islington (bot) in branch '3.7':
    bpo-29620: iterate over a copy of sys.modules (GH-4800) (GH-20817)
    f881c86

    @akuchling
    Copy link
    Member

    Changes applied to master, 3.8, and 3.7. Thanks!

    @ambv
    Copy link
    Contributor

    ambv commented Nov 18, 2021

    New changeset 71d842b by Miss Islington (bot) in branch '3.9':
    bpo-29620: iterate over a copy of sys.modules (GH-4800) (GH-29605)
    71d842b

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants