I noticed that pip wasn't working properly on Python 3.3a - notably, it wasn't able to load any of its own VCS modules -- it does this by using pkgutil.walk_packages

I think the problem is that the behavior of pkgutil.walk_packages changed in some incompatible way in 3.3 -- take a look at the following:

[last: 0] marca@scml-marca:~/dev/git-repos/lexicon$ ../lexicon/.tox/py32/bin/python
Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgutil import walk_packages
>>> list(walk_packages('/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/pip-1.1-py3.3.egg/pip/vcs'))
[(<pkgutil.ImpImporter object at 0x1005bc710>, 'lexicon', True), (<pkgutil.ImpImporter object at 0x100649410>, 'lexicon.alias_dict', False), (<pkgutil.ImpImporter object at 0x100649410>, 'lexicon.attribute_dict', False), (<pkgutil.ImpImporter object at 0x1005bc710>, 'reg_settings', False), (<pkgutil.ImpImporter object at 0x1005bc710>, 'setup', False)]
>>> ^D

[last: 10] marca@scml-marca:~/dev/git-repos/lexicon$ ../lexicon/.tox/py33/bin/python
Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May  1 2012, 11:39:35) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pkgutil import walk_packages
>>> list(walk_packages('/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/pip-1.1-py3.3.egg/pip/vcs'))
[]

I stepped through in pdb

(Pdb) p i
<_frozen_importlib.FileFinder object at 0x101066090>
(Pdb) iter_importer_modules(i, prefix)
[]

(Pdb) p hasattr(importer, 'iter_modules')
False

2.7 uses

[<pkgutil.ImpImporter instance at 0x107a4d830>]
(Pdb) pp dir(myimp)
['__doc__', '__init__', '__module__', 'find_module', 'iter_modules', 'path']

This seems related to http://www.python.org/dev/peps/pep-0420/

I don't think this is related to PEP 420. Adding Brett.

Here's the pip issue: https://github.com/pypa/pip/issues/556

I was pretty tired when debugging last night and just quickly looked at hg logs, so that may be misattributed.

I'll try come up with a clearer reproducer.

[last: 0] marca@scml-marca:~/dev/git-repos/pip$ python3.3
Python 3.3.0a4 (v3.3.0a4:7c51388a3aa7, May 30 2012, 16:58:42) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pip import vcs
>>> from pkgutil import walk_packages, iter_modules
>>> print(list(iter_modules(path=vcs.__path__, prefix=vcs.__name__+'.')))
[]
>>> print(list(iter_modules(path=vcs.__path__)))
[]
>>> import pip.vcs.git
>>> pip.vcs.git
<module 'pip.vcs.git' from './pip/vcs/git.py'>
>>> import pip.vcs.mercurial
>>> pip.vcs.mercurial
<module 'pip.vcs.mercurial' from './pip/vcs/mercurial.py'>
>>> print(list(iter_modules(path=vcs.__path__, prefix=vcs.__name__+'.')))
[]
>>> print(list(iter_modules(path=vcs.__path__)))
[]

Basically pkgutil kind of handles importers properly, kind of doesn't. So if a module defined a __loader__ it will use it, but all the rest of its code assumes it uses only the loaders defined in pkgutil.

The problem here is that pkgutil.walk_packages() ends up calling iter_importer_modules() which only returns anything of consequence if the loader has iter_modules() defined which is a non-standard API requirement that only pkgutil loaders has implemented. Basically the docs for pkgutil were incorrect in not specifying that the walk only works for loaders that define iter_modules().

Ok so it seems I can't just use sys.meta_path in pip to work with ImpImporter due to according to the pydoc:

Note that ImpImporter does not currently support being used by placement on sys.meta_path.

I guess I can write a custom importer in pip for handling the vcs plugins or possibly change how we do that. Another option is to grossly hack around ala:

i = ImpImporter(path=vcs.__path__[0])
sys.path_importer_cache.setdefault(vcs.__path__[0], i)

Found another bug doing this though which I'll follow up with a patch.

See also issue 14787 re: walk_packages(), which also affects versions before 3.3.

I should mention that Guido and others on python-dev mentioned coming up with an API for finders/loaders that allowed for file-like API and possibly being able to iterate over available modules when importlib's bootstrapping landed (sorry, don't have a link for it). If pip needs some specific introspection support from finders or loaders I would try asking on python-dev about exactly what you need to see if it can get into Python 3.3.

We've taken a simpler approach avoiding walk_packages in pip which we'll release for 3.3. I'd say if pkgutil doesn't work correctly with importers & loaders outside of it we probably should make that very explicit in the docs, and potentially consider deprecating walk_packages.

Yes, the docs should get updated and I will do that before Python 3.3 goes out the door.

New changeset b92fa1c5a96b by Brett Cannon in branch 'default':
Closes issue #14982: Document that pkgutil's walk_packages() and
http://hg.python.org/cpython/rev/b92fa1c5a96b

As someone who isn't an expert on Python's import mechanisms, I'm confused by this comment at the end of pkgutil.walk_packages():

"Note: Only works for importers which define a iter_modules() method, which is non-standard but implemented by classes defined in this module."

I'm confused because importers aren't mentioned anywhere in the context of walk_packages() (in particular, `importer` is not a parameter to the function), so it's not clear to me how the note comes into play.

For example, is "importers" in reference to the import statements inside the modules in the package that one is trying to walk, or is it a reference to whether or not the user has overridden Python's standard import with a custom import function?

In particular, does this mean that the normal case of trying to walk a directory using `pkgutil.walk_packages(['my_dir'])` won't work without doing something special, and if so, can that special thing be added?

I think it would help if this were made clearer.  I came to this after trying to start working on issue 14787.

Importers are a PEP 302 API.  In the context of pkgutil, they are path hooks (see http://www.python.org/dev/peps/pep-0302/#id24), aka path importers.  A path hook is a callable that takes a path and returns a finder.  Path hooks are stored in sys.path_hooks and the finder resulting from a matching path importer is stored in sys.path_importer_cache.

pkgutil also refers to PEP 302 finders as "importers".  To be honest, the import-related nomenclature is a mess.

While you can use pkgutil.ImpImporter as a path hook, instances can also be used as metapath hooks; if you don't pass anything to ImpImporter, the resulting object acts more like a PEP 302 metapath hook that wraps the "default" import process.

(See Lib/pkgutil.py)

Thanks a lot, Eric.  I'm going to create a new issue to clarify the note a bit, if that's okay.  At minimum, I feel like enough information should be given to let one run the examples given.

By the way, your comment, "While you can use pkgutil.ImpImporter as a path hook, instances can also be used as metapath hooks" seems to contradict the pkgutil documentation, which says, "Note that ImpImporter does not currently support being used by placement on sys.meta_path."  (According to PEP 302, "To register a meta hook, simply add the finder object to sys.meta_path.")

I created issue 15288 to improve the documentation around this.

FYI, I created issue 15299 and issue 15297 which also relate to pkgutil.walk_packages() not working correctly in Python 3.3 -- even with the caveat added by Brett.  These two issues are in pkgutil.walk_packages()'s code path when passed path=None.