Recently, when debugging a weird problem (see https://bugzilla.redhat.com/show_bug.cgi?id=2018551 for details if interested, but not important for this issue), I've realized that the _NamespacePath class (from importlib/_bootstrap_external.py) has a cache that (in some cases) uses tuple(sys.path) as the key. I was expecting importlib.invalidate_caches() to invalidate this cache, but it doesn't.

Consider the following directory structure:

.
├── PATH1
│   └── namespace
│   └── sub1
│   └── __init__.py
└── PATH2
└── namespace
└── sub2
└── __init__.py

Here is a helper to create it (on Linux-ish):

$ mkdir -p PATH1/namespace/sub1
$ mkdir -p PATH2/namespace/sub2
$ touch PATH1/namespace/sub1/__init__.py
$ touch PATH2/namespace/sub2/__init__.py

Run Python with PYTHONPATH=PATH1:PATH2 (output slightly formatted for readability):

$ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace',
                '.../namespace_path_cache/PATH2/namespace'])
>>> import namespace.sub1  # works
>>> import namespace.sub2  # works
>>> exit()

The namespace packages seem to work as expected.

Now move PATH2/namespace out of the way:

$ mv PATH2/namespace PATH2/cant-import-this

Run Python again:

$ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> ...

While this interpreter still runs, move the PATH2/namespace module back in:

$ mv PATH2/cant-import-this PATH2/namespace

>>> ...
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'

>>> import importlib
>>> importlib.invalidate_caches()  # invalidate the cache, not helpful
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'


>>> import sys
>>> sys.path.remove('')  # changing sys.path solves this
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace',
                '.../namespace_path_cache/PATH2/namespace'])

importlib.invalidate_caches() documentation says:

This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.

That makes me think calling importlib.invalidate_caches() should also invalidate the cache of _NamespacePaths.

(This also affects older Pythons, but since it is a behavior change, I've only marked 3.11).

I have a fix in mind, will submit a draft pull request without tests shortly, continue with the tests later.

The PR is now ready for review.

New changeset ae1965c by Miro Hrončok in branch 'main':
bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… (GH-29384)
ae1965c

Now, is this a bugfix, or a new feature?
I lean toward calling it a bugfix and backporting to earlier versions.

I consider it a bugfix. I don't expect users to rely on the previous behavior and be surprised by the new, but yet I know https://xkcd.com/1172/ very well.

Sadly, the backport is non-trivial. I'm putting on my TODO list, but I doubt I'll get to it soon :(

The automated backport to Python 3.9 failed. Miro asked me to close the PR:
#29964

Can someone backport the fix to 3.9? (create a PR)

Embarassingly, it seems it bust needs regen-importlib, at least for 3.10

New changeset 5c39e47 by Petr Viktorin in branch '3.10':
[3.10] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922)
5c39e47

New changeset 8d239bf by Petr Viktorin in branch '3.9':
[3.9] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922) (GH-31076)
8d239bf