classification
Title: test.support.import_fresh_module fails to correctly block submodules when fresh is specified
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, ncoghlan, p-ganssle, serhiy.storchaka, shihai1991
Priority: normal Keywords: patch

Created on 2020-04-03 15:18 by p-ganssle, last changed 2021-10-01 07:08 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
test_support_repro.zip p-ganssle, 2020-04-03 15:18 Module to reproduce the issue
Pull Requests
URL Status Linked Edit
PR 19390 closed shihai1991, 2020-04-06 10:21
PR 28654 merged serhiy.storchaka, 2021-09-30 14:00
PR 28657 merged serhiy.storchaka, 2021-09-30 16:27
PR 28658 merged serhiy.storchaka, 2021-09-30 16:35
Messages (7)
msg365702 - (view) Author: Paul Ganssle (p-ganssle) * (Python committer) Date: 2020-04-03 15:18
It seems that test.support.import_fresh_module gets tripped up with its module blocking when you attempt to get a fresh copy of a submodule of a module where you are also importing the module that you are trying to block (bit of a doozy of a sentence there...). So, for example, with the following configuration in mymodule/__init__.py:

    from .other import other
    
    try:
        from ._a import attr
    except ImportError:
        from ._b import attr

(Assuming _a.attr = "A" and _b.attr = "B"), if you attempt to do:

    m = test.support.import_fresh_module("mymodule", fresh=("mymodule._other",), blocked=("mymodule._a"))

Then you'll find that m.attr is pulled from _a.attr. Here's a small script to demonstrate:

    from test.support import import_fresh_module
    import sys

    def import_ab(fresh_other):
        fresh = ("mymodule._other", ) if fresh_other else ()

        mods_out = []
        for to_block in "_b", "_a":
            blocked = (f"mymodule.{to_block}",)

            mods_out.append(import_fresh_module("mymodule",
                                                fresh=fresh, blocked=blocked))
        return mods_out


    for fresh_other in [True, False]:
        mymodule_a, mymodule_b = import_ab(fresh_other)

        qualifier = "With" if fresh_other else "Without"
        print(f"{qualifier} a fresh import of mymodule._other")

        print(f"a: {mymodule_a.attr}")
        print(f"b: {mymodule_b.attr}")
        print()

When you run it with a suitably configured module on Python 3.8:

$ python importer.py 
With a fresh import of mymodule._other
a: A
b: A

Without a fresh import of mymodule._other
a: A
b: B

It also happens if you add `mymodule._a` or `mymodule._b` to the fresh list when you are trying to block the other one.

I *think* the problem is that in the step where _save_and_remove_module is called on fresh_name (see here: https://github.com/python/cpython/blob/76db37b1d37a9daadd9e5b320f2d5a53cd1352ec/Lib/test/support/__init__.py#L328-L329), it's necessarily populating `sys.modules` with a fresh import of the top-level module we're trying to import (mymodule) *before* the blocking goes into effect, then the final call to importlib.import_module just hits that cache.

I think either of the following options will fix this issue:

1. Switching the order of how "fresh" and "blocked" are resolved or
2. Deleting `sys.modules[name]` if it exists immediately before calling `importlib.import_module(name)

That said, I'm still having some weird statefulness problems if I block a C module's import and *then* block a Python module's import, so there may be some other underlying pathology to the current approach.
msg365846 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-04-06 10:23
> I *think* the problem is that in the step where _save_and_remove_module is called on fresh_name (see here: https://github.com/python/cpython/blob/76db37b1d37a9daadd9e5b320f2d5a53cd1352ec/Lib/test/support/__init__.py#L328-L329)

Looks like deleting a module name after `__import__(name)` is not good enought in https://github.com/python/cpython/blob/master/Lib/test/support/__init__.py#L244.(some redundant submodules should be removed, no?)

paul's example can be passed in PR19390.
msg402961 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 13:58
I just encountered this bug after rewriting datetime tests in issue45229. It is also the cause of issue40058.

Not knowing about this issue I have wrote different patch than Hai Shi. It makes the code of import_fresh_module() clearer and fixes many other issues.
msg402970 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 16:20
New changeset ec4d917a6a68824f1895f75d113add9410283da7 by Serhiy Storchaka in branch 'main':
bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654)
https://github.com/python/cpython/commit/ec4d917a6a68824f1895f75d113add9410283da7
msg402972 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 16:56
New changeset 7873884d4730d7e637a968011b8958bd79fd3398 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654) (GH-28657)
https://github.com/python/cpython/commit/7873884d4730d7e637a968011b8958bd79fd3398
msg402973 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 16:57
New changeset f7e99c98130a705f88348dde60adb98d5bfd8b98 by Serhiy Storchaka in branch '3.9':
[3.9] bpo-40173: Fix test.support.import_helper.import_fresh_module() (GH-28654) (GH-28658)
https://github.com/python/cpython/commit/f7e99c98130a705f88348dde60adb98d5bfd8b98
msg402975 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 17:32
_save_and_remove_module in the old code did too much and too little. It tested that the fresh module is importable, saved the current state of the fresh module and its submodules and removed the fresh module and its submodules. Since it tested importability before saving all states, it can have a side effect of importing other fresh module, and since it did it before blocking, the imported "fresh" module could import the blocked modules.

> 1. Switching the order of how "fresh" and "blocked" are resolved or

It does not help, because the step of resolving "fresh" includes saving the sys.modules state, and the step of resolving "blocked" modifies sys.modules. Also the step of resolving "fresh" can modify sys.modules before saving its sate.

2. Deleting `sys.modules[name]` if it exists immediately before calling `importlib.import_module(name)

It would be not enough, because the problem can occur in one of additional fresh modules imported by the initial module. We need to remove all fresh modules before attempting to import them.

The new import_fresh_module() consists of the following steps:

1. Save the state of all fresh and blocked modules and their submodules.
2. Remove the fresh modules and their submodules and block the blocked modules and their submodules.
3. Import all additional fresh modules and return None if any of them is not importable.
4. Import the initial module and raise ImportError if it is not importable.
5. Restore states of all fresh and blocked modules and their submodules.

It does not save and restore the state of all modules, because some indirectly imported modules can not support repeated importing. It can be reconsidered in new versions.
History
Date User Action Args
2021-10-01 07:08:55serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-09-30 17:32:20serhiy.storchakasetmessages: + msg402975
2021-09-30 16:57:50serhiy.storchakasetmessages: + msg402973
2021-09-30 16:56:52serhiy.storchakasetmessages: + msg402972
2021-09-30 16:35:59serhiy.storchakasetpull_requests: + pull_request27024
2021-09-30 16:27:12serhiy.storchakasetpull_requests: + pull_request27023
2021-09-30 16:20:47serhiy.storchakasetmessages: + msg402970
2021-09-30 14:00:50serhiy.storchakasetpull_requests: + pull_request27020
2021-09-30 13:59:09serhiy.storchakasetversions: + Python 3.10, Python 3.11, - Python 3.7, Python 3.8
2021-09-30 13:58:57serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg402961
2021-09-30 13:18:30serhiy.storchakalinkissue45229 dependencies
2020-04-06 10:23:01shihai1991setmessages: + msg365846
2020-04-06 10:21:02shihai1991setkeywords: + patch
stage: patch review
pull_requests: + pull_request18754
2020-04-04 15:00:46shihai1991setnosy: + shihai1991
2020-04-03 15:18:43p-gansslecreate