classification
Title: pkgutil.iter_zipimport_modules ignores the prefix parameter for packages
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: James.Pickering, brett.cannon, eric.snow, lukasz.langa, ncoghlan, pje, python-dev
Priority: normal Keywords:

Created on 2012-03-06 15:02 by James.Pickering, last changed 2016-06-12 01:09 by lukasz.langa. This issue is now closed.

Messages (7)
msg155018 - (view) Author: James Pickering (James.Pickering) Date: 2012-03-06 15:02
If you run pkgutil.iter_zipimport_modules with a prefix parameter, and the module in question is a package, then the prefix parameter is ignored.

The most visible symptom of this is when running pkgutil.walk_packages for a zipfile. Imagine we have a module structure like this (or create one):

a/
a/__init__.py
a/b/
a/b/__init.__py

If we put this structure in a directory, add the directory to sys.path, and run pkgutil.walk_packages(), it will find modules "a" and "a.b".

If we put this structure in a zipfile, however, we add this file to sys.path, and run pkgutil.walk_packages(), it will find modules "a" and "b". This is because pkgutil.iter_zipimport_modules ignores the prefix parameter "a.".

This is incorrect.

This can be fixed by changing line ~344 of Lib/pkgutil.py from:

    yield fn[0], True

to

    yield prefix + fn[0], True

Thanks, James

P.s, This is my first Python bug report. I apologise in advance for any poor etiquette.
msg155137 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-08 00:15
Adding Brett, since the plan is to clear out a lot of the redundant code in pkgutil once importlib is fully bootstrapped as the standard import implementation.

(although this will still affect the older versions directly)
msg265221 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2016-05-10 03:06
Brett, Facebook is using the proposed patch in prod since last year. It works fine. Maybe we should just include it for the time being?
msg265223 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-05-10 03:09
I don't deal with pkgutil so I have no problem if you want to go ahead and apply the patch if you think it's reasonable to accept, Łukasz.
msg265241 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-05-10 11:49
+1 for just including the fix in the next round of maintenance releases, although a test case would be desirable.
msg268306 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-12 01:07
New changeset 9649acf7d472 by Łukasz Langa in branch '3.5':
Issue #14209: pkgutil.iter_zipimport_modules ignores the prefix for packages
https://hg.python.org/cpython/rev/9649acf7d472

New changeset 389b7456a053 by Łukasz Langa in branch 'default':
Merge 3.5, issue #14209
https://hg.python.org/cpython/rev/389b7456a053
msg268307 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2016-06-12 01:09
Done. Fix is going to be present for 3.5.2 and 3.6. Thank you, James.
History
Date User Action Args
2016-06-12 01:09:36lukasz.langasetstatus: open -> closed
resolution: fixed
messages: + msg268307

versions: + Python 3.5, Python 3.6, - Python 2.7, Python 3.2, Python 3.3
2016-06-12 01:07:40python-devsetnosy: + python-dev
messages: + msg268306
2016-05-10 17:44:09brett.cannonsetassignee: lukasz.langa
2016-05-10 11:49:16ncoghlansetmessages: + msg265241
2016-05-10 03:09:45brett.cannonsetmessages: + msg265223
2016-05-10 03:06:15lukasz.langasetnosy: + lukasz.langa
messages: + msg265221
2012-03-09 08:54:28eric.snowsetnosy: + eric.snow
2012-03-08 00:15:20ncoghlansetnosy: + brett.cannon
messages: + msg155137
2012-03-07 22:06:45eric.araujosetnosy: + pje, ncoghlan

stage: test needed
2012-03-06 15:02:24James.Pickeringcreate