Would it make sense to clear sys.path_importer_cache when someone calls importlib.invalidate_caches()? It is a cache after all.

Hi Brett, I'm looking for some issues to solve. Is this issue confirmed? Can I work on this?

Sorry, this has not been decided upon yet. Since it's a change in semantics either I just need to make a decision or convince someone else to provide an opinion as to whether this change makes sense.

(Note that PathFinder.invalidate_caches() loops over sys.path_importer_cache and invalidates the cache of each path entry finder therein.  What you're suggesting is clearing sys.path_importer_cache after the loop finishes.)

Hmm.  The distinction is between clearing the cache of each finder and clearing the caches in the import machinery, sort of.  sys.path_importer_cache effectively belongs to PathFinder, just like sys.path_hooks.  I expect they are part of sys only as a historical artifact (i.e. pre-importlib).  So I'd say there isn't any conceptual problem in that regard.

The point of importlib.invalidate_caches() is to allow finders to pick up out-of-band events.  In the case of sys.path_importer_cache, it would allow new entries in sys.path_hooks to have a shot and old entries to have another chance.  That seems reasonable.

The only caveat I see is that sys.path_importer_cache is not an "internal" cache (as the importlib.invalidate_caches() docs refer to them).  Furthermore, it has some established semantics attached:

  * a path hook will only be called once for a path entry (entries in the cache are fixed in place once added)
  * None is a sentinel for not-found

Consequently,someone may inject None into the cache to prevent finding a spec/loader. With this chance, calls to importlib.invalidate_caches() could cause them problems.  In this regard there may be a backward-compatibility issue.

Other than that, I consider clearing sys.path_importer_cache in PathFinder.invalidate_caches() to be a reasonable change.

Damn, good point about the backwards-compatibility issue. Then I think a doc note for importlib.invalidate_caches() saying that there is also sys.path_importer_cache would be good enough.

Or another option is to add a keyword-only argument to importlib.invalidate_caches() to also clear the the cache of importers itself on top of the caches of the importers themselves.

Re: the kw-only arg, it seems weird to accommodate the implementation of one meta-path finder in the signature of importlib.invalidate_caches().

Here's an alternative:

Use a different sys.path_importer_cache sentinel internally (in PathFinder) and never clear None when clearing the cache.

Here's another alternative:

Deprecate sys.path_importer_cache and move/alias it to an internal cache of PathFinder. *  Then add methods to PathFinder to cover the use cases that folks have for interacting with sys.path_importer_cache directly.  In this case it may also make sense to expose the active PathFinder instance as an importlib attribute, for easier access to this key finder.  Doing the same on ImportState will be part of my (eventual) proposal. :)

Of course, neither of those alternatives addresses the case where someone expects sys.path_importer_cache entries to remain fixed once in place.  However, that's a tenuous implicit guarantee already. :)


* The same could be done with sys.path_hooks...

Eric's convinced me this isn't worth it.