Issue:
Running ModuleFinder.run_script() on numpy versions 1.16.1 to 1.18.3 (maybe more) fails with backtrace. See steps to reproduce below.
I do not see this problem on earlier versions of python than 3.8 (tested 3.4, 3.5, 3.6 on ubuntu LTSs), but the code has changed around 3.8.
The failure comes to this line of modulefinder.py
https://github.com/python/cpython/blame/master/Lib/modulefinder.py#L79
if spec.loader.is_package(name):
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)
I can work around it by changing that to check for None
if spec.loader is not None and spec.loader.is_package(name):
return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)
Environment:
Ubuntu 20.04 with default python3, python3-pip
Steps to reproduce:
# note any nump version I've tried 1.16.1 and greater fails - I included 1.18.3 to be precise for reproduciton
$ pip3 install "numpy==1.18.3"
$ cat test.py
import numpy
$ python3
>>> from modulefinder import ModuleFinder
>>> finder = ModuleFinder()
>>> finder.run_script("test.py")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/modulefinder.py", line 165, in run_script
self.load_module('__main__', fp, pathname, stuff)
...
300 lines of stack elided - see attached fulllog.txt
...
File "/usr/lib/python3.8/modulefinder.py", line 433, in scan_code
self._safe_import_hook(name, m, fromlist, level=0)
File "/usr/lib/python3.8/modulefinder.py", line 378, in _safe_import_hook
self.import_hook(name, caller, level=level)
File "/usr/lib/python3.8/modulefinder.py", line 177, in import_hook
q, tail = self.find_head_package(parent, name)
File "/usr/lib/python3.8/modulefinder.py", line 233, in find_head_package
q = self.import_module(head, qname, parent)
File "/usr/lib/python3.8/modulefinder.py", line 320, in import_module
fp, pathname, stuff = self.find_module(partname,
File "/usr/lib/python3.8/modulefinder.py", line 511, in find_module
return _find_module(name, path)
File "/usr/lib/python3.8/modulefinder.py", line 78, in _find_module
if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'
>>>
Obviously I can't tell if numpy or modulefinder is the real culprit.
Let me know if I can give any more information.
|
I opened up a new PR that should fix this properly. The root issue was that PathFinder was not setting the loader attribute for namespace packages in the spec, which it should. After fixing that, _find_module just needed to be updated to deal with NamespaceLoader.
```
$ tree namespace
namespace/
└── a.py
0 directories, 1 file
$ cat test-bpo-40350.py
import namespace.a
```
```
$ ./python
Python 3.11.0a1+ (heads/main:9e05da6224, Oct 23 2021, 20:36:14) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modulefinder
>>> modulefinder.ModuleFinder('test-bpo-40350.py')
<modulefinder.ModuleFinder object at 0x7f66f7647480>
>>> f = modulefinder.ModuleFinder('test-bpo-40350.py')
>>> f.
KeyboardInterrupt
>>> m = modulefinder.ModuleFinder()
>>> f = modulefinder.ModuleFinder()
>>> f.run_script('test-bpo-40350.py')
>>> f.modules.items()
dict_items([('__main__', Module('__main__', 'test-bpo-40350.py')), ('namespace', Module('namespace', _NamespacePath(['/home/anubis/git/cpython/namespace'])))])
```
Previously:
```
$ python
Python 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import modulefinder
>>> f = modulefinder.ModuleFinder()
>>> f.run_script('test-bpo-40350.py')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/modulefinder.py", line 161, in run_script
self.load_module('__main__', fp, pathname, stuff)
File "/usr/lib/python3.9/modulefinder.py", line 357, in load_module
self.scan_code(co, m)
File "/usr/lib/python3.9/modulefinder.py", line 430, in scan_code
self._safe_import_hook(name, m, fromlist, level=0)
File "/usr/lib/python3.9/modulefinder.py", line 375, in _safe_import_hook
self.import_hook(name, caller, level=level)
File "/usr/lib/python3.9/modulefinder.py", line 173, in import_hook
q, tail = self.find_head_package(parent, name)
File "/usr/lib/python3.9/modulefinder.py", line 229, in find_head_package
q = self.import_module(head, qname, parent)
File "/usr/lib/python3.9/modulefinder.py", line 316, in import_module
fp, pathname, stuff = self.find_module(partname,
File "/usr/lib/python3.9/modulefinder.py", line 508, in find_module
return _find_module(name, path)
File "/usr/lib/python3.9/modulefinder.py", line 77, in _find_module
if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'
```
|