classification
Title: pkgutil.walk_packages seems to not work properly on Python 3.3a
Type: Stage:
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: Arfrever, Marc.Abramowitz, Ronan.Lamy, brett.cannon, chris.jerdonek, eric.smith, eric.snow, pnasrat, python-dev
Priority: normal Keywords:

Created on 2012-06-01 23:00 by Marc.Abramowitz, last changed 2012-07-08 22:38 by chris.jerdonek. This issue is now closed.

Messages (18)
msg162110 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2012-06-01 23:00
I noticed that pip wasn't working properly on Python 3.3a - notably, it wasn't able to load any of its own VCS modules -- it does this by using pkgutil.walk_packages

I think the problem is that the behavior of pkgutil.walk_packages changed in some incompatible way in 3.3 -- take a look at the following:

[last: 0] marca@scml-marca:~/dev/git-repos/lexicon$ ../lexicon/.tox/py32/bin/python
Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgutil import walk_packages
>>> list(walk_packages('/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/pip-1.1-py3.3.egg/pip/vcs'))
[(<pkgutil.ImpImporter object at 0x1005bc710>, 'lexicon', True), (<pkgutil.ImpImporter object at 0x100649410>, 'lexicon.alias_dict', False), (<pkgutil.ImpImporter object at 0x100649410>, 'lexicon.attribute_dict', False), (<pkgutil.ImpImporter object at 0x1005bc710>, 'reg_settings', False), (<pkgutil.ImpImporter object at 0x1005bc710>, 'setup', False)]
>>> ^D

[last: 10] marca@scml-marca:~/dev/git-repos/lexicon$ ../lexicon/.tox/py33/bin/python
Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May  1 2012, 11:39:35) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgutil import walk_packages
>>> list(walk_packages('/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/pip-1.1-py3.3.egg/pip/vcs'))
[]
msg162111 - (view) Author: Paul Nasrat (pnasrat) Date: 2012-06-01 23:05
I stepped through in pdb

(Pdb) p i
<_frozen_importlib.FileFinder object at 0x101066090>
(Pdb) iter_importer_modules(i, prefix)
[]

(Pdb) p hasattr(importer, 'iter_modules')
False

2.7 uses

[<pkgutil.ImpImporter instance at 0x107a4d830>]
(Pdb) pp dir(myimp)
['__doc__', '__init__', '__module__', 'find_module', 'iter_modules', 'path']

This seems related to http://www.python.org/dev/peps/pep-0420/
msg162116 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-06-02 01:11
I don't think this is related to PEP 420. Adding Brett.
msg162121 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2012-06-02 04:34
Here's the pip issue: https://github.com/pypa/pip/issues/556
msg162122 - (view) Author: Paul Nasrat (pnasrat) Date: 2012-06-02 04:56
I was pretty tired when debugging last night and just quickly looked at hg logs, so that may be misattributed.

I'll try come up with a clearer reproducer.
msg162124 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2012-06-02 06:35
[last: 0] marca@scml-marca:~/dev/git-repos/pip$ python3.3
Python 3.3.0a4 (v3.3.0a4:7c51388a3aa7, May 30 2012, 16:58:42) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pip import vcs
>>> from pkgutil import walk_packages, iter_modules
>>> print(list(iter_modules(path=vcs.__path__, prefix=vcs.__name__+'.')))
[]
>>> print(list(iter_modules(path=vcs.__path__)))
[]
>>> import pip.vcs.git
>>> pip.vcs.git
<module 'pip.vcs.git' from './pip/vcs/git.py'>
>>> import pip.vcs.mercurial
>>> pip.vcs.mercurial
<module 'pip.vcs.mercurial' from './pip/vcs/mercurial.py'>
>>> print(list(iter_modules(path=vcs.__path__, prefix=vcs.__name__+'.')))
[]
>>> print(list(iter_modules(path=vcs.__path__)))
[]
msg162153 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-02 17:40
Basically pkgutil kind of handles importers properly, kind of doesn't. So if a module defined a __loader__ it will use it, but all the rest of its code assumes it uses only the loaders defined in pkgutil.

The problem here is that pkgutil.walk_packages() ends up calling iter_importer_modules() which only returns anything of consequence if the loader has iter_modules() defined which is a non-standard API requirement that only pkgutil loaders has implemented. Basically the docs for pkgutil were incorrect in not specifying that the walk only works for loaders that define iter_modules().
msg162182 - (view) Author: Paul Nasrat (pnasrat) Date: 2012-06-02 21:57
Ok so it seems I can't just use sys.meta_path in pip to work with ImpImporter due to according to the pydoc:

Note that ImpImporter does not currently support being used by placement on sys.meta_path.

I guess I can write a custom importer in pip for handling the vcs plugins or possibly change how we do that. Another option is to grossly hack around ala:

i = ImpImporter(path=vcs.__path__[0])
sys.path_importer_cache.setdefault(vcs.__path__[0], i)

Found another bug doing this though which I'll follow up with a patch.
msg162197 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-06-03 06:23
See also issue 14787 re: walk_packages(), which also affects versions before 3.3.
msg162238 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-03 22:35
I should mention that Guido and others on python-dev mentioned coming up with an API for finders/loaders that allowed for file-like API and possibly being able to iterate over available modules when importlib's bootstrapping landed (sorry, don't have a link for it). If pip needs some specific introspection support from finders or loaders I would try asking on python-dev about exactly what you need to see if it can get into Python 3.3.
msg162404 - (view) Author: Paul Nasrat (pnasrat) Date: 2012-06-06 12:24
We've taken a simpler approach avoiding walk_packages in pip which we'll release for 3.3. I'd say if pkgutil doesn't work correctly with importers & loaders outside of it we probably should make that very explicit in the docs, and potentially consider deprecating walk_packages.
msg162416 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-06 17:36
Yes, the docs should get updated and I will do that before Python 3.3 goes out the door.
msg162938 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-15 23:21
New changeset b92fa1c5a96b by Brett Cannon in branch 'default':
Closes issue #14982: Document that pkgutil's walk_packages() and
http://hg.python.org/cpython/rev/b92fa1c5a96b
msg164884 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-07 15:45
As someone who isn't an expert on Python's import mechanisms, I'm confused by this comment at the end of pkgutil.walk_packages():

"Note: Only works for importers which define a iter_modules() method, which is non-standard but implemented by classes defined in this module."

I'm confused because importers aren't mentioned anywhere in the context of walk_packages() (in particular, `importer` is not a parameter to the function), so it's not clear to me how the note comes into play.

For example, is "importers" in reference to the import statements inside the modules in the package that one is trying to walk, or is it a reference to whether or not the user has overridden Python's standard import with a custom import function?

In particular, does this mean that the normal case of trying to walk a directory using `pkgutil.walk_packages(['my_dir'])` won't work without doing something special, and if so, can that special thing be added?

I think it would help if this were made clearer.  I came to this after trying to start working on issue 14787.
msg164913 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-07-07 17:10
Importers are a PEP 302 API.  In the context of pkgutil, they are path hooks (see http://www.python.org/dev/peps/pep-0302/#id24), aka path importers.  A path hook is a callable that takes a path and returns a finder.  Path hooks are stored in sys.path_hooks and the finder resulting from a matching path importer is stored in sys.path_importer_cache.

pkgutil also refers to PEP 302 finders as "importers".  To be honest, the import-related nomenclature is a mess.

While you can use pkgutil.ImpImporter as a path hook, instances can also be used as metapath hooks; if you don't pass anything to ImpImporter, the resulting object acts more like a PEP 302 metapath hook that wraps the "default" import process.

(See Lib/pkgutil.py)
msg164915 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-07 18:00
Thanks a lot, Eric.  I'm going to create a new issue to clarify the note a bit, if that's okay.  At minimum, I feel like enough information should be given to let one run the examples given.

By the way, your comment, "While you can use pkgutil.ImpImporter as a path hook, instances can also be used as metapath hooks" seems to contradict the pkgutil documentation, which says, "Note that ImpImporter does not currently support being used by placement on sys.meta_path."  (According to PEP 302, "To register a meta hook, simply add the finder object to sys.meta_path.")
msg164917 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-07 18:18
I created issue 15288 to improve the documentation around this.
msg165041 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-08 22:38
FYI, I created issue 15299 and issue 15297 which also relate to pkgutil.walk_packages() not working correctly in Python 3.3 -- even with the caveat added by Brett.  These two issues are in pkgutil.walk_packages()'s code path when passed path=None.
History
Date User Action Args
2012-07-08 22:38:08chris.jerdoneksetmessages: + msg165041
2012-07-07 18:18:23chris.jerdoneksetmessages: + msg164917
2012-07-07 18:00:47chris.jerdoneksetmessages: + msg164915
2012-07-07 17:10:53eric.snowsetnosy: + eric.snow
messages: + msg164913
2012-07-07 15:45:05chris.jerdoneksetmessages: + msg164884
2012-06-15 23:22:05brett.cannonsetstatus: open -> closed
resolution: fixed
2012-06-15 23:21:14python-devsetnosy: + python-dev
messages: + msg162938
2012-06-06 17:36:05brett.cannonsetassignee: brett.cannon
messages: + msg162416
2012-06-06 12:24:32pnasratsetmessages: + msg162404
2012-06-03 22:35:01brett.cannonsetmessages: + msg162238
2012-06-03 06:23:48chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg162197
2012-06-02 21:57:34pnasratsetmessages: + msg162182
2012-06-02 17:40:12brett.cannonsetmessages: + msg162153
2012-06-02 16:33:15Ronan.Lamysetnosy: + Ronan.Lamy
2012-06-02 06:35:48Marc.Abramowitzsetmessages: + msg162124
2012-06-02 04:56:40pnasratsetmessages: + msg162122
2012-06-02 04:34:19Marc.Abramowitzsetmessages: + msg162121
2012-06-02 01:26:01Arfreversetnosy: + Arfrever
2012-06-02 01:11:48eric.smithsetnosy: + brett.cannon, eric.smith
messages: + msg162116
2012-06-01 23:05:42pnasratsetnosy: + pnasrat
messages: + msg162111
2012-06-01 23:00:34Marc.Abramowitzcreate