Title: Implement PEP 420: Implicit Namespace Packages
Type: enhancement Stage: resolved
Components: Versions: Python 3.3
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: Yury.Selivanov, barry, brett.cannon, eric.araujo, eric.smith, eric.snow, jaraco, python-dev
Priority: normal Keywords:

Created on 2012-04-24 11:08 by eric.smith, last changed 2012-05-28 05:35 by python-dev. This issue is now closed.

Messages (13)
msg159131 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-04-24 11:08
I have created a branch features/pep-420 where I'll be developing a proof of concept implementation of PEP 420.

I've checked in a basic version, but it has these issues:

- We need to decide how finders communicate that they've found part of
a namespace package. Per Brett's suggestion, I'm currently returning a
string that means "the returned path is part of a namespace package".
I think that's an okay design.

- I guess we need to create a "namespace loader", so we can initialize __loader__. It's a little odd because it would be a loader for which there's no need for a find_module method. I'm currently setting __loader__ to the finder, which is completely wrong, but keeps things working.

- test_import and test_importlib both fail because they're checking
that imports with no fail. Those tests need to be removed.

- There are no tests yet.
msg159147 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-24 14:42
Loaders are not meant to have a find_module method; that is purely for finders which can be distinct objects. So having a namespace loader is expected and there is no expectation that it have a find_module method (actually almost all of the loaders in importlib lack a find_module method).
msg159150 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-04-24 15:00
Right, that's a typo. I meant load_module(). I'm currently working on implementing the loader for namespace modules, so my comment about the loader is premature.
msg159162 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-04-24 15:37
I created the NamespaceLoader in the feature branch. It has a load_module, but it's only ever called by the code in PathFinder.load_module:

                loader = NamespaceLoader(namespace_path)
                return loader.load_module(fullname)

namespace_path is what will become module.__path__. In order to keep the load_module API (single fullname argument), I pass it in to the constructor. There's no particular need for it to follow that API, but it does.
msg159165 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-24 15:57
Yeah, having to pass in the name to load_module is silly. I'm seriously considering making it optional for some loaders when the name was passed in to the constructor.

One thing I would like to see is that PathFinder take a new, keyword-only argument of 'namespace_loader' or something that takes an object which is called with the module name and the list of paths to use for __path__ (and thus __path__[0] becomes __file__). That way it can be controlled by users if desired. I'm also fine with it having a default value of NamespaceLoader to start since I don't see people needing to comprehend what they are allowing in this case as I do for general file loaders.
msg159276 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-04-25 12:47
I'd really prefer something like:

  return load_ns_module(fullname, namespace_path)

The point being that load_module() as defined in PEP 302 will never be called on NamespaceLoader. The loader only needs to exist to set module.__loader__, and load_module() would never be called from there.

So we could pass in a callable named load_ns_module to PathFinder.

Then NamespaceLoader's load_module function could be named load_ns_module, made a class or static method.

I think I'll experiment with this.
msg159290 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-25 14:51
What do you mean the loader is only needed to set __loader__? You need the loader to create the module (or find it in sys.modules to reload), and set all the attributes properly. If you do this then reloading namespace modules will become a special case compared to other loaders as imp.reload() calls module.__loader__.load_module().

This also prevents the creation of an importlib.find_module() which would return the loader to replace imp.find_module() since you now split the API.

I realize the finder/loader dichotomy seems superfluous (and most of the time it is), but it has already been heavily exposed and relied on and deviating from it for namespace modules runs the risk of hurting introspection.
msg159291 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-04-25 14:56
Ah. I didn't realize that reload called load_module. I'll back out the change I just made, then.

My point was that the original call to load_module isn't made through the normal "a finder returned me a loader, so I'll call it" code path. It's always made through the "I know I have a NamespaceLoader" path.
msg159293 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-25 15:00
The joys of trying to shoehorn into an existing API. I mean short of adding a new sys.namespace_loader instead of an explicit keyword argument to FileFinder I can't think of a better solution.
msg159735 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-04-30 22:54
I've modified zipimport to support namespace packages, and checked it in to the feature branch. This completes all of the functionality I think needs to be added. Next up is adding tests.
msg161541 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-05-25 00:22
New changeset 702009f3c0b1 by Eric V. Smith in branch 'default':
issue 14660: Implement PEP 420, namespace packages.
msg161542 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-05-25 00:23
I'm closing this issue. I'll open new issues as needed.
msg161758 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-05-28 05:35
New changeset f26721ab3476 by Ned Deily in branch 'default':
Issue #14660: Install namespace_pkgs test directories and files.
Date User Action Args
2012-05-28 05:35:46python-devsetmessages: + msg161758
2012-05-25 00:23:19eric.smithsetstatus: open -> closed
resolution: fixed
messages: + msg161542

stage: test needed -> resolved
2012-05-25 00:22:23python-devsetnosy: + python-dev
messages: + msg161541
2012-04-30 22:54:12eric.smithsetmessages: + msg159735
stage: test needed
2012-04-25 16:03:20eric.araujosetnosy: + eric.araujo
2012-04-25 15:00:45brett.cannonsetmessages: + msg159293
2012-04-25 14:56:40eric.smithsetmessages: + msg159291
2012-04-25 14:51:54brett.cannonsetmessages: + msg159290
2012-04-25 12:50:46Yury.Selivanovsetnosy: + Yury.Selivanov
2012-04-25 12:47:37eric.smithsetmessages: + msg159276
2012-04-24 21:54:12eric.snowsetnosy: + eric.snow
2012-04-24 15:57:16brett.cannonsetmessages: + msg159165
2012-04-24 15:37:58eric.smithsetmessages: + msg159162
2012-04-24 15:00:32eric.smithsetmessages: + msg159150
2012-04-24 14:42:32brett.cannonsetmessages: + msg159147
2012-04-24 11:08:39eric.smithcreate