Title: Clear sys.path_importer_cache from importlib.invalidate_caches()
Type: enhancement Stage: test needed
Components: Documentation Versions: Python 3.6, Python 3.5
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: anish.shah, brett.cannon, eric.snow, ncoghlan
Priority: normal Keywords:

Created on 2016-02-03 21:11 by brett.cannon, last changed 2016-03-18 20:26 by brett.cannon. This issue is now closed.

Messages (8)
msg259518 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-03 21:11
Would it make sense to clear sys.path_importer_cache when someone calls importlib.invalidate_caches()? It is a cache after all.
msg260689 - (view) Author: Anish Shah (anish.shah) * Date: 2016-02-22 16:53
Hi Brett, I'm looking for some issues to solve. Is this issue confirmed? Can I work on this?
msg260692 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-22 17:41
Sorry, this has not been decided upon yet. Since it's a change in semantics either I just need to make a decision or convince someone else to provide an opinion as to whether this change makes sense.
msg260752 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2016-02-23 23:46
(Note that PathFinder.invalidate_caches() loops over sys.path_importer_cache and invalidates the cache of each path entry finder therein.  What you're suggesting is clearing sys.path_importer_cache after the loop finishes.)

Hmm.  The distinction is between clearing the cache of each finder and clearing the caches in the import machinery, sort of.  sys.path_importer_cache effectively belongs to PathFinder, just like sys.path_hooks.  I expect they are part of sys only as a historical artifact (i.e. pre-importlib).  So I'd say there isn't any conceptual problem in that regard.

The point of importlib.invalidate_caches() is to allow finders to pick up out-of-band events.  In the case of sys.path_importer_cache, it would allow new entries in sys.path_hooks to have a shot and old entries to have another chance.  That seems reasonable.

The only caveat I see is that sys.path_importer_cache is not an "internal" cache (as the importlib.invalidate_caches() docs refer to them).  Furthermore, it has some established semantics attached:

  * a path hook will only be called once for a path entry (entries in the cache are fixed in place once added)
  * None is a sentinel for not-found

Consequently,someone may inject None into the cache to prevent finding a spec/loader. With this chance, calls to importlib.invalidate_caches() could cause them problems.  In this regard there may be a backward-compatibility issue.

Other than that, I consider clearing sys.path_importer_cache in PathFinder.invalidate_caches() to be a reasonable change.
msg260753 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-23 23:59
Damn, good point about the backwards-compatibility issue. Then I think a doc note for importlib.invalidate_caches() saying that there is also sys.path_importer_cache would be good enough.
msg260754 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-24 00:00
Or another option is to add a keyword-only argument to importlib.invalidate_caches() to also clear the the cache of importers itself on top of the caches of the importers themselves.
msg260756 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2016-02-24 01:51
Re: the kw-only arg, it seems weird to accommodate the implementation of one meta-path finder in the signature of importlib.invalidate_caches().

Here's an alternative:

Use a different sys.path_importer_cache sentinel internally (in PathFinder) and never clear None when clearing the cache.

Here's another alternative:

Deprecate sys.path_importer_cache and move/alias it to an internal cache of PathFinder. *  Then add methods to PathFinder to cover the use cases that folks have for interacting with sys.path_importer_cache directly.  In this case it may also make sense to expose the active PathFinder instance as an importlib attribute, for easier access to this key finder.  Doing the same on ImportState will be part of my (eventual) proposal. :)

Of course, neither of those alternatives addresses the case where someone expects sys.path_importer_cache entries to remain fixed once in place.  However, that's a tenuous implicit guarantee already. :)

* The same could be done with sys.path_hooks...
msg261993 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-03-18 20:26
Eric's convinced me this isn't worth it.
Date User Action Args
2016-03-18 20:26:09brett.cannonsetstatus: open -> closed
resolution: rejected
messages: + msg261993
2016-02-24 01:51:26eric.snowsetmessages: + msg260756
2016-02-24 00:00:23brett.cannonsetmessages: + msg260754
2016-02-23 23:59:15brett.cannonsetassignee: brett.cannon
messages: + msg260753
components: + Documentation, - Library (Lib)
versions: + Python 3.5
2016-02-23 23:46:07eric.snowsetmessages: + msg260752
2016-02-22 17:41:02brett.cannonsetmessages: + msg260692
2016-02-22 16:53:07anish.shahsetmessages: + msg260689
2016-02-04 07:46:01anish.shahsetnosy: + anish.shah
2016-02-03 21:11:00brett.cannoncreate