Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

importlib.invalidate_caches() doesn't clear all caches #77350

Closed
gvanrossum opened this issue Mar 28, 2018 · 9 comments
Closed

importlib.invalidate_caches() doesn't clear all caches #77350

gvanrossum opened this issue Mar 28, 2018 · 9 comments
Assignees
@brettcannon
Labels
3.7 (EOL) end of life 3.8 only security fixes

Comments

@gvanrossum
Copy link
Member

BPO 33169
Nosy @gvanrossum, @brettcannon, @ncoghlan, @ned-deily, @ericsnowcurrently
PRs
  • bpo-33169: Remove values of None from sys.path_importer_cache when invalidating caches #6402
  • [3.7] bpo-33169: Remove values of None from sys.path_importer_cache when invalidating caches (GH-6402) #6403
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/brettcannon'
closed_at = <Date 2018-04-07.00:02:53.795>
created_at = <Date 2018-03-28.14:30:18.602>
labels = ['3.7', '3.8']
title = "importlib.invalidate_caches() doesn't clear all caches"
updated_at = <Date 2018-04-07.00:02:53.710>
user = 'https://github.com/gvanrossum'

    bugs.python.org fields:

    activity = <Date 2018-04-07.00:02:53.710>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2018-04-07.00:02:53.795>
closer = 'brett.cannon'
components = []
creation = <Date 2018-03-28.14:30:18.602>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 33169
keywords = ['patch']
message_count = 9.0
messages = ['314595', '314672', '314677', '314684', '314685', '314745', '315042', '315044', '315045']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'brett.cannon', 'ncoghlan', 'ned.deily', 'eric.snow']
pr_nums = ['6402', '6403']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33169'
versions = ['Python 3.7', 'Python 3.8']
    @gvanrossum
    Copy link
    Member Author

    See python/mypy#4811. To summarize, importlib.invalidate_caches() doesn't clear the negative cache in sys.path_importer_cache.

    Could be related to https://bugs.python.org/issue30891?

    @brettcannon
    Copy link
    Member

    Correct because as documented it calls invalidate_caches() on meta-path finders and the one that handles sys.path calls invalidate_caches() on the finders found in sys.path_importer_cache() (as documented by https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder.invalidate_caches). Since None isn't a finder it doesn't do anything there as there is no method to call. So the "full of lies" comment by the OP on the mypy repo is over-the-top/incorrect.

    Now tweaking

    cpython/Lib/importlib/_bootstrap_external.py

    Lines 1180 to 1186 in 5219952

    @classmethod
    def invalidate_caches(cls):
    """Call the invalidate_caches() method on all path entry finders
    stored in sys.path_importer_caches (where implemented)."""
    for finder in sys.path_importer_cache.values():
    if hasattr(finder, 'invalidate_caches'):
    finder.invalidate_caches()
    to delete entries in sys.path_importer_cache() that are set to None is straight-forward, but I don't think we can backport since it would be going against what the documentation says it does.

    @brettcannon brettcannon added the 3.8 only security fixes label Mar 29, 2018
    @ncoghlan
    Copy link
    Contributor

    I agree that we can't change 3.6 at this late stage, but the change is borderline enough on the "new feature"/"bug fix" scale that I think it's worth asking Ned if he's comfortable with us changing it for 3.7.0b4.

    Ned: the change we'd like to make is to update PathFinder.invalidate_caches to delete all entries in sys.path_importer_cache that have a value of None.

    @ned-deily
    Copy link
    Member

    The current behavior is causing problems for at least some use cases so it does seems to be closer to a bug than a feature. If there is agreement that it is desirable to make the change, I think it's better to do it now in 3.7 than later.

    @gvanrossum
    Copy link
    Member Author

    SGTM.

    On Fri, Mar 30, 2018, 06:41 Ned Deily <report@bugs.python.org> wrote:

    Ned Deily <nad@python.org> added the comment:

    The current behavior is causing problems for at least some use cases so it
    does seems to be closer to a bug than a feature. If there is agreement
    that it is desirable to make the change, I think it's better to do it now
    in 3.7 than later.

    ----------

    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue33169\>

    @brettcannon
    Copy link
    Member

    OK, I will update importlib.machinery.PathFinder.invalidate_caches() to also drop None from sys.path_importer_cache (objects that are not None and do not define invalidate_caches() will be left alone just like they are today). I should hopefully be able to get to this before b4.

    @brettcannon brettcannon added the 3.7 (EOL) end of life label Mar 31, 2018
    @brettcannon brettcannon self-assigned this Mar 31, 2018
    @brettcannon
    Copy link
    Member

    New changeset 9e2be60 by Brett Cannon in branch 'master':
    bpo-33169: Remove values of None from sys.path_importer_cache when invalidating caches (GH-6402)
    9e2be60

    @brettcannon
    Copy link
    Member

    New changeset a09bb87 by Brett Cannon in branch '3.7':
    [3.7] bpo-33169: Remove values of None from sys.path_importer_cache when invalidating caches (GH-6402) (GH-6403)
    a09bb87

    @brettcannon
    Copy link
    Member

    All done in 3.7 and master.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @brettcannon brettcannon

    Labels
    3.7 (EOL) end of life 3.8 only security fixes
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @brettcannon @ncoghlan @gvanrossum @ned-deily