classification
Title: Documentation on the change of __path__ in Python 3
Type: Stage:
Components: Documentation Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: brett.cannon, docs@python, eric.smith, jimli
Priority: normal Keywords:

Created on 2019-06-26 23:46 by jimli, last changed 2019-06-28 21:17 by terry.reedy.

Messages (6)
msg346701 - (view) Author: Jim Li (jimli) Date: 2019-06-26 23:46
In Python 2, `__path__` used to be a list, so all of the operations available to list are available, e.g., `insert`; you can also do indexing; e.g., `__path__[0]`.

However, I believe that starting from Python 3, it seems to be a <class '_frozen_importlib_external._NamespacePath'>, and a lot of operations that worked previously stopped working.It seems so abruptive and I can't find any deprecation notice on this.

Previously, one is able to insert an additional path to the front of the list by doing

module.__path__.insert(0, 'src/mypath')

Now the only procedure allowed seems to be append. Is there any way to mimic the old behaviour? Sorry, this is my first question so maybe this issue doesn't deserve any more attention. If there is any documentation that talks about the change of type of the NamespacePath, please kindly lemme know.

Sincerely,

Related issues include https://bugs.python.org/issue35843
msg346704 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-06-27 00:10
Can you provide a short runnable example that used to work and now does not? And please show any error messages you're seeing.
msg346755 - (view) Author: Jim Li (jimli) Date: 2019-06-27 16:32
Hi Eric,

Sorry for the late reply. I think I did not accurately describe the issue at all. As a minimal example, set up two virtual environments, one from 2.7.x, one from 3.7.2.

When you are in the virtual environment, do
- pip install protobuf==3.3.0
- python (to go into compiler)
- import google
- print(type(google.__path__))

The 2.7.x would tell you it's a <type 'list'>
The 3.7.2 would tell you it's a <class'_frozen_importlib_external._NamespacePath'>

Since it's not a list, some behaviours are gone, e.g., insert. I'm not trying to manipulate protobuf package's __path__, but trying to upgrade a legacy codebase, the legacy codebase does something like:

module.insert(0, 'some/path'), which breaks under python3, because it's not a list anymore, but a <class'_frozen_importlib_external._NamespacePath'>.


I believe the problem comes from the 'top level entry point'. For example, when you have 2 separate packages but they were organized under the same namespace.

E.g., in their setup.py

Package A:

    packages=[
        "orange",
        "orange.schemas",
    ],
    namespace_packages=["orange"]

Package B:
    packages=[
        "orange",
        "orange.server",
    ],
    namespace_packages=["orange"]

The problem comes from when you try to do `import orange`, and do operations with orange.__path__ here. In Python 2.7, it's a list; in Python 3.7, it's a <class '_frozen_importlib_external._NamespacePath'>
msg346790 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-06-28 01:20
It's a _NamespacePath in 3.7 because there's no __init__.py in the top-level "google" directory, and that makes it a namespace package.

I'm not exactly sure why it works in 2.7, frankly. <time passes> Looking some more: it's because they're playing tricks in a .pth file. I don't recall what the interactions between .pth and namespace packages.

I don't know if Google really means for "google" to be a namespace package, or if it's just a side effect of how they're installing protobuf.

In any event, I don't think we want a _NamespacePath to be modifiable.
msg346791 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-06-28 01:21
And I see you're not asking for changed behavior, just documentation. But I think it is documented that this is how namespace packages work.
msg346793 - (view) Author: Jim Li (jimli) Date: 2019-06-28 02:37
Ah, that started to make sense. To be honest I didn't really pay much attention to the _namespace's documentation, I only looked at that of __path__ and somehow expected it to mention that certain namespace won't be a type<list> anymore.

I will re-read your comments and the docs for _namespace; I was also reading this https://www.python.org/dev/peps/pep-0420/#migrating-from-legacy-namespace-packages

Thanks for the help!
History
Date User Action Args
2019-06-28 21:17:23terry.reedysetversions: + Python 3.8, Python 3.9, - Python 3.5, Python 3.6
2019-06-28 02:37:03jimlisetmessages: + msg346793
2019-06-28 01:21:56eric.smithsetmessages: + msg346791
2019-06-28 01:20:52eric.smithsetmessages: + msg346790
2019-06-27 17:04:31xtreaksetnosy: + brett.cannon
2019-06-27 16:32:09jimlisetmessages: + msg346755
2019-06-27 00:10:00eric.smithsetnosy: + eric.smith
messages: + msg346704
2019-06-26 23:46:04jimlicreate