classification
Title: __init__.py required for pkgutil.walk_packages in python3
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Anthony Sottile, Eric Wieser, docs@python, eric.snow, methane, ncoghlan, wolma
Priority: normal Keywords:

Created on 2017-01-13 04:47 by Anthony Sottile, last changed 2020-01-20 12:16 by methane.

Messages (6)
msg285361 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2017-01-13 04:47
PEP420 makes __init__.py files optional: https://docs.python.org/3/whatsnew/3.3.html#pep-420-implicit-namespace-packages

Though it seems without them, pkgutil.walk_packages does not function as desired: https://docs.python.org/3/library/pkgutil.html#pkgutil.walk_packages

Consider the following example:

```
$ tree foo
foo
├── bar
│   ├── baz.py
│   └── __init__.py
├── __init__.py
└── womp.py
```

And a test script

# test.py
```
import pkgutil

import foo


for _, mod, _ in pkgutil.walk_packages(foo.__path__, foo.__name__ + '.'):
    print(mod)
```

In both python2 and python3 I get the following output:

```
$ python2.7 test.py
foo.bar
foo.bar.baz
foo.womp
$ python3.5 test.py
foo.bar
foo.bar.baz
foo.womp
```

Removing the __init__.py files and only using python3, I get this:

```
$ find -name '__init__.*' -delete
$ python3.5 test.py
foo.bar
```

The modules are definitely importable:

```
$ python3.5 -c 'import foo.bar.baz'
$
```

Originally asked as a question on stackoverflow: http://stackoverflow.com/questions/41203765/init-py-required-for-pkgutil-walk-packages-in-python3
msg289345 - (view) Author: Wolfgang Maier (wolma) * Date: 2017-03-10 09:54
While it is rather trivial to implement the proposed functionality - all that's required here is to eliminate the check for __init__.py from pkgutil._iter_file_finder_modules - this would have undesired impacts on, e.g., pydoc.apropos:
This function would then recursively report *any* directory/subdirectory on sys.path, which is quite surely not what people want.

I think this is a fundamental problem with namespace packages: they are nice and flexible for specific imports, but they make it impossible to know whether a directory found on the filesystem is *intended* as a Python package or not.
msg289347 - (view) Author: Wolfgang Maier (wolma) * Date: 2017-03-10 10:05
> all that's required here is to eliminate the check for __init__.py from pkgutil._iter_file_finder_modules

Ok, I was exaggerating here. To do it right would require a more complex change, but that's all that's needed to get an estimate of the effect the real thing would have.
msg360298 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-01-20 08:23
> PEP420 makes __init__.py files optional

This is almost wrong.  PEP 420 added a new way for "namespace pacakge."
PEP 420 doesn't make empty __init__.py file in regular package.

(See https://dev.to/methane/don-t-omit-init-py-3hga)

Then, should pkgutil.walk_packages walk into all directories (e.g.  node_modules) ?  I don't think so.
msg360307 - (view) Author: Eric Wieser (Eric Wieser) Date: 2020-01-20 11:02
If the resolution here is that this is behaving as intended (which personally I disagree with), I think this issue should remain open as a documentation task - the docs should clearly state that this does not apply to PEP420 namespace packages.
msg360310 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-01-20 12:16
I am totally agree with Wolfgang:

> they make it impossible to know whether a directory found on the filesystem is *intended* as a Python package or not.

I think we shouldn't treat normal directory as namespace package until some portion in the directory is imported, or it is specified explicitly.

So walk_packages() should be used like:

  walk_packages("/path/to/namespace", "namespace")

I already rejected similar issue: #29642.

If you can not agree with me, please make an thread in python-dev ML or discuss.python.org.
History
Date User Action Args
2020-01-20 12:16:18methanesetmessages: + msg360310
2020-01-20 11:58:59methanesetstatus: closed -> open
assignee: docs@python
components: + Documentation, - Library (Lib)
versions: + Python 3.8, Python 3.9, - Python 3.5, Python 3.6
nosy: + docs@python

resolution: rejected ->
stage: resolved -> needs patch
2020-01-20 11:02:40Eric Wiesersetmessages: + msg360307
2020-01-20 08:23:34methanesetstatus: open -> closed

nosy: + methane
messages: + msg360298

resolution: rejected
stage: resolved
2019-09-19 00:46:01Eric Wiesersetnosy: + Eric Wieser
2017-03-10 18:20:03brett.cannonsetnosy: - brett.cannon
2017-03-10 10:05:46wolmasetmessages: + msg289347
2017-03-10 09:54:01wolmasetmessages: + msg289345
2017-03-10 07:18:52wolmasetnosy: + wolma
2017-01-13 08:28:24serhiy.storchakasetnosy: + brett.cannon, ncoghlan, eric.snow

versions: - Python 3.3, Python 3.4
2017-01-13 04:47:54Anthony Sottilecreate