classification
Title: importlib.invalidate_caches() doesn't clear all caches
Type: Stage: resolved
Components: Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, eric.snow, gvanrossum, ncoghlan, ned.deily
Priority: normal Keywords: patch

Created on 2018-03-28 14:30 by gvanrossum, last changed 2018-04-07 00:02 by brett.cannon. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6402 merged brett.cannon, 2018-04-06 22:02
PR 6403 merged brett.cannon, 2018-04-06 23:23
Messages (9)
msg314595 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-03-28 14:30
See https://github.com/python/mypy/pull/4811. To summarize, importlib.invalidate_caches() doesn't clear the negative cache in sys.path_importer_cache.

Could be related to https://bugs.python.org/issue30891?
msg314672 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-03-29 20:37
Correct because as documented it calls invalidate_caches() on meta-path finders and the one that handles sys.path calls invalidate_caches() on the finders found in sys.path_importer_cache() (as documented by https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder.invalidate_caches). Since None isn't a finder it doesn't do anything there as there is no method to call. So the "full of lies" comment by the OP on the mypy repo is over-the-top/incorrect.

Now tweaking https://github.com/python/cpython/blob/521995205a2cb6b504fe0e39af22a81f785350a3/Lib/importlib/_bootstrap_external.py#L1180-L1186 to delete entries in sys.path_importer_cache() that are set to None is straight-forward, but I don't think we can backport since it would be going against what the documentation says it does.
msg314677 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-03-30 05:02
I agree that we can't change 3.6 at this late stage, but the change is borderline enough on the "new feature"/"bug fix" scale that I think it's worth asking Ned if he's comfortable with us changing it for 3.7.0b4.

Ned: the change we'd like to make is to update PathFinder.invalidate_caches to delete all entries in sys.path_importer_cache that have a value of None.
msg314684 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-03-30 13:41
The current behavior is causing problems for at least some use cases so it does seems to be closer to a bug than a feature.  If there is agreement that it is desirable to make the change, I think it's better to do it now in 3.7 than later.
msg314685 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-03-30 15:01
SGTM.

On Fri, Mar 30, 2018, 06:41 Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily <nad@python.org> added the comment:
>
> The current behavior is causing problems for at least some use cases so it
> does seems to be closer to a bug than a feature.  If there is agreement
> that it is desirable to make the change, I think it's better to do it now
> in 3.7 than later.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33169>
> _______________________________________
>
msg314745 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-03-31 18:07
OK, I will update importlib.machinery.PathFinder.invalidate_caches() to also drop None from sys.path_importer_cache (objects that are not None and do not define invalidate_caches() will be left alone just like they are today). I should hopefully be able to get to this before b4.
msg315042 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-04-06 23:10
New changeset 9e2be60634914f23db2ae5624e4acc9335bf5fea by Brett Cannon in branch 'master':
bpo-33169: Remove values of `None` from sys.path_importer_cache when invalidating caches (GH-6402)
https://github.com/python/cpython/commit/9e2be60634914f23db2ae5624e4acc9335bf5fea
msg315044 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-04-07 00:02
New changeset a09bb87c1eebb07b01b8105cf536704893aec565 by Brett Cannon in branch '3.7':
[3.7] bpo-33169: Remove values of `None` from sys.path_importer_cache when invalidating caches (GH-6402) (GH-6403)
https://github.com/python/cpython/commit/a09bb87c1eebb07b01b8105cf536704893aec565
msg315045 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-04-07 00:02
All done in 3.7 and master.
History
Date User Action Args
2018-04-07 00:02:53brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg315045

stage: patch review -> resolved
2018-04-07 00:02:20brett.cannonsetmessages: + msg315044
2018-04-06 23:23:13brett.cannonsetpull_requests: + pull_request6108
2018-04-06 23:10:22brett.cannonsetmessages: + msg315042
2018-04-06 22:02:25brett.cannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request6107
2018-03-31 18:07:29brett.cannonsetassignee: brett.cannon
messages: + msg314745
versions: + Python 3.7
2018-03-30 15:01:30gvanrossumsetmessages: + msg314685
2018-03-30 13:41:51ned.deilysetmessages: + msg314684
2018-03-30 05:02:38ncoghlansetnosy: + ned.deily
messages: + msg314677
2018-03-29 20:37:25brett.cannonsetmessages: + msg314672
versions: + Python 3.8, - Python 3.6
2018-03-29 20:27:21brett.cannonsetnosy: + brett.cannon, ncoghlan, eric.snow
2018-03-28 14:30:18gvanrossumcreate