This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: importlib.invalidate_caches() does not invalidate _NamespacePath's _last_parent_path-based cache
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: hroncok, miss-islington, petr.viktorin, vstinner
Priority: normal Keywords: patch

Created on 2021-11-03 11:21 by hroncok, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 29384 merged hroncok, 2021-11-03 11:37
PR 29964 closed miss-islington, 2021-12-07 14:45
PR 30922 merged petr.viktorin, 2022-01-26 15:43
PR 31076 merged petr.viktorin, 2022-02-02 13:22
Messages (11)
msg405606 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-11-03 11:21
Recently, when debugging a weird problem (see for details if interested, but not important for this issue), I've realized that the _NamespacePath class (from importlib/ has a cache that (in some cases) uses tuple(sys.path) as the key. I was expecting importlib.invalidate_caches() to invalidate this cache, but it doesn't.

Consider the following directory structure:

├── PATH1
│   └── namespace
│       └── sub1
│           └──
└── PATH2
    └── namespace
        └── sub2

Here is a helper to create it (on Linux-ish):

$ mkdir -p PATH1/namespace/sub1
$ mkdir -p PATH2/namespace/sub2
$ touch PATH1/namespace/sub1/
$ touch PATH2/namespace/sub2/

Run Python with PYTHONPATH=PATH1:PATH2 (output slightly formatted for readability):

>>> import namespace
>>> namespace.__path__
>>> import namespace.sub1  # works
>>> import namespace.sub2  # works
>>> exit()

The namespace packages seem to work as expected.

Now move PATH2/namespace out of the way:

$ mv PATH2/namespace PATH2/cant-import-this

Run Python again:

>>> import namespace
>>> namespace.__path__
>>> ...

While this interpreter still runs, move the PATH2/namespace module back in:

$ mv PATH2/cant-import-this PATH2/namespace

>>> ...
>>> namespace.__path__
>>> import namespace.sub2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'

>>> import importlib
>>> importlib.invalidate_caches()  # invalidate the cache, not helpful
>>> namespace.__path__
>>> import namespace.sub2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'

>>> import sys
>>> sys.path.remove('')  # changing sys.path solves this
>>> namespace.__path__
>>> import namespace.sub2
>>> namespace.__path__

importlib.invalidate_caches() documentation says:

> This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.

That makes me think calling importlib.invalidate_caches() should also invalidate the cache of _NamespacePaths.

(This also affects older Pythons, but since it is a behavior change, I've only marked 3.11).
msg405607 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-11-03 11:28
I have a fix in mind, will submit a draft pull request without tests shortly, continue with the tests later.
msg405620 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-11-03 14:25
The PR is now ready for review.
msg406846 - (view) Author: miss-islington (miss-islington) Date: 2021-11-23 15:38
New changeset ae1965ccb4b1fad63fab40fe8805d1b8247668d3 by Miro Hrončok in branch 'main':
bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… (GH-29384)
msg406848 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-11-23 15:50
Now, is this a bugfix, or a new feature?
I lean toward calling it a bugfix and backporting to earlier versions.
msg406851 - (view) Author: Miro Hrončok (hroncok) * Date: 2021-11-23 15:59
I consider it a bugfix. I don't expect users to rely on the previous behavior and be surprised by the new, but yet I know very well.
msg407936 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-12-07 14:59
Sadly, the backport is non-trivial. I'm putting on my TODO list, but I doubt I'll get to it soon :(
msg411368 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-01-23 13:36
The automated backport to Python 3.9 failed. Miro asked me to close the PR:

Can someone backport the fix to 3.9? (create a PR)
msg411759 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2022-01-26 15:44
Embarassingly, it seems it bust needs regen-importlib, at least for 3.10
msg411877 - (view) Author: miss-islington (miss-islington) Date: 2022-01-27 14:00
New changeset 5c39e474dbd61397c2ff877fa17d742bf4646702 by Petr Viktorin in branch '3.10':
[3.10] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922)
msg412359 - (view) Author: miss-islington (miss-islington) Date: 2022-02-02 13:50
New changeset 8d239bfdcc98f4e4a592d9cda569380a9c15a5c0 by Petr Viktorin in branch '3.9':
[3.9] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922) (GH-31076)
Date User Action Args
2022-04-11 14:59:52adminsetgithub: 89866
2022-02-02 14:12:06petr.viktorinsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2022-02-02 13:50:54miss-islingtonsetmessages: + msg412359
2022-02-02 13:22:45petr.viktorinsetpull_requests: + pull_request29260
2022-01-27 14:00:30miss-islingtonsetmessages: + msg411877
2022-01-26 15:44:05petr.viktorinsetmessages: + msg411759
2022-01-26 15:43:15petr.viktorinsetpull_requests: + pull_request29101
2022-01-23 13:36:46vstinnersetmessages: + msg411368
2021-12-07 14:59:00petr.viktorinsetmessages: + msg407936
2021-12-07 14:45:21miss-islingtonsetpull_requests: + pull_request28189
2021-11-23 15:59:56hroncoksetmessages: + msg406851
2021-11-23 15:50:13petr.viktorinsetmessages: + msg406848
2021-11-23 15:38:11miss-islingtonsetnosy: + miss-islington
messages: + msg406846
2021-11-09 10:53:40vstinnersetnosy: + vstinner
2021-11-03 14:25:43hroncoksetcomponents: + Library (Lib)
2021-11-03 14:25:24hroncoksetmessages: + msg405620
2021-11-03 11:37:16hroncoksetkeywords: + patch
stage: patch review
pull_requests: + pull_request27642
2021-11-03 11:28:17hroncoksetmessages: + msg405607
2021-11-03 11:21:56hroncokcreate