classification
Title: An os.walk inspired replacement for pkgutil.walk_packages
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: isoschiz, ncoghlan, wolma
Priority: normal Keywords: patch

Created on 2013-01-28 11:17 by ncoghlan, last changed 2017-03-10 17:00 by BreamoreBoy.

Files
File name Uploaded Description Edit
issue17062.diff isoschiz, 2013-04-18 23:47 Implementation of suggested walk_path
Messages (5)
msg180847 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-01-28 11:17
I recently had occasion to use pkgutil.walk_packages, and my immediate thought was that it would have been a lot easier for me to use if it worked more like os.walk with topdown=True, producing tuples of (pkg, subpackages, modules)

"pkg" would be the package object at the current level (None for the top level)

"packages" would be a dictionary mapping fully qualified module names to loader objects for the subpackages (i.e. subdirectories)

"modules" would be a dictionary mapping fully qualified module names to loader objects for every submodule that wasn't a subpackage

As with editing the "subdirs" list with os.walk, editing the "packages" dictionary with this new API would keep the iterator from loading that subpackage and avoid recursing into it (this is the part I wanted in my current use case).

(This may even be PEP material, guiding some additions to the importer/finder API)
msg180848 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-01-28 11:30
Oops, forgot the proposed call signature:

    def walk_path(path=None, *, pkg=None):
        """Walk a package hierarchy, starting with the given path

        Iterator producing (package, subpackages, submodules) triples.
        The first entry is the package currently being walked, or None
        for the top level path. The subpackages and submodules entries
        are dictionaries mapping from fully qualified module names to
        the appropriate module loaders.

        Entries may be removed from the subpackages dictionary to avoid
        loading those packages and recursing into them.
        
        If both pkg and path are None, walks sys.path

        If path is not None, walks the specified path.

        If pkg is not None, walks pkg.__path__

        Providing both path and pkg results in ValueError
        """
msg180849 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-01-28 11:34
Regarding the PEP comment - the piece that would be missing is the "iter_modules" functionality. Currently pkgutil provides the support for standard filesystem imports and zipimports directly - the generic function based extension mechanism is undocumented.
msg187311 - (view) Author: Martin Morrison (isoschiz) * Date: 2013-04-18 23:47
I threw together a function that implements this. The only variation from the proposed signature was adding the onerror argument supported by the other similar functions in the module.
msg221036 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-19 21:59
Could somebody review the attached patch please.
History
Date User Action Args
2017-03-10 17:00:13BreamoreBoysetnosy: - BreamoreBoy
2017-03-10 07:18:24wolmasetnosy: + wolma
2014-06-19 21:59:06BreamoreBoysetnosy: + BreamoreBoy
messages: + msg221036
2013-04-18 23:47:51isoschizsetfiles: + issue17062.diff

nosy: + isoschiz
messages: + msg187311

keywords: + patch
2013-01-28 11:42:17serhiy.storchakasettype: enhancement
2013-01-28 11:34:18ncoghlansetmessages: + msg180849
2013-01-28 11:30:00ncoghlansetmessages: + msg180848
2013-01-28 11:17:08ncoghlancreate