classification
Title: `sys.meta_path` Skipped for Packages with Non-Standard Suffixed `__init__` Files
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Wolfgang Richter, brett.cannon, eric.smith, eric.snow, ncoghlan
Priority: normal Keywords:

Created on 2016-02-11 22:39 by Wolfgang Richter, last changed 2016-02-12 14:09 by Wolfgang Richter. This issue is now closed.

Files
File name Uploaded Description Edit
PyMetaPath.tar.gz Wolfgang Richter, 2016-02-11 22:39 Minimal code example triggering behavior
main.py Wolfgang Richter, 2016-02-12 00:51
Messages (8)
msg260138 - (view) Author: Wolfgang Richter (Wolfgang Richter) * Date: 2016-02-11 22:39
My understanding of `sys.meta_path` is that it is supposed to allow customized loading of Python modules and packages.

In fact the `importlib` machinery appears to have support for identifying packages with `__init__` files with non-standard suffixes:

https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py#L645

https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py#L1233

However, I find that when I `import wolftest` inside a folder with structure:

./
    /wolftest
        /__init__.wolf
        /something.wolf

None of my sys.meta_path finders are called at all, and instead a namespace is returned.

I was wondering why the `import` statement appears to short-circuit and not check with `sys.meta_path` handlers in this case?
msg260142 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-11 23:01
I'm at work and so I don't have access to tar on this machine ATM, so I can't look at your code example. But sys.meta_path might be one level above what you want; you might be looking for sys.path_hooks since https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder is what searches entries on sys.path. This also means that if you don't put your finder on sys.meta_path before importlib.machinery.PathFinder then it will never pick up a directory since any directory will always be viewed as a namespace package by that meta path finder.
msg260145 - (view) Author: Wolfgang Richter (Wolfgang Richter) * Date: 2016-02-11 23:17
Brett thanks for the very quick response.

I've inserted my finder as the first element in sys.meta_path.

I also overrode the @classmethod find_loader, but it doesn't appear that it's being called.
msg260146 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-11 23:26
As I said, I can't look at your code unless you upload the file separately so I don't know how much help I can be. Did you check sys.modules to make sure the directory had not already been imported or sys.path_importer_cache to make sure the directory didn't already have a finder associated with it? I know sys.meta_path isn't bypassed as that's how the builtin and frozen importer work.

And have you tried this in a newer version of Python? 3.4 is no longer receiving bugfixes so even if there is a problem there it won't be changed. And you should try and use https://docs.python.org/3/library/importlib.html#importlib.util.find_spec instead of find_loader().
msg260147 - (view) Author: Wolfgang Richter (Wolfgang Richter) * Date: 2016-02-11 23:43
Sure Brett I'll upload the code separately in an hour or two.

I've overridden all the publicly exposed functions (including find_spec).

I wanted to initially see if anything was passing through my finder in the first position of the meta_path.
msg260152 - (view) Author: Wolfgang Richter (Wolfgang Richter) * Date: 2016-02-12 00:51
Here's the file, before I experimented with find_loader.

If you take out all my print's, it's quite short.
msg260156 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-12 03:23
I found the problem: you have a bug in your code at line 45 (the first line of WolfPathFinder.find_spec()). When you try and import a top-level package the path will always be None since __path__ doesn't exist for a top-level package. You only end up with a `path` value when there is a __path__ in the parent package. If you put your print() call before the None check it will print out that you actually did have the method called.
msg260182 - (view) Author: Wolfgang Richter (Wolfgang Richter) * Date: 2016-02-12 14:09
Ah, sorry Brett!

Thanks for spotting this.

The import machinery is *really* awesome.
History
Date User Action Args
2016-02-12 14:09:33Wolfgang Richtersetmessages: + msg260182
2016-02-12 03:23:14brett.cannonsetstatus: open -> closed
resolution: not a bug
messages: + msg260156
2016-02-12 00:52:54Wolfgang Richtersetversions: + Python 3.5
2016-02-12 00:51:08Wolfgang Richtersetfiles: + main.py

messages: + msg260152
2016-02-11 23:43:40Wolfgang Richtersetmessages: + msg260147
2016-02-11 23:26:07brett.cannonsetmessages: + msg260146
2016-02-11 23:17:41Wolfgang Richtersetmessages: + msg260145
2016-02-11 23:01:22brett.cannonsetmessages: + msg260142
2016-02-11 22:41:25eric.smithsetnosy: + brett.cannon, ncoghlan, eric.smith, eric.snow
2016-02-11 22:39:43Wolfgang Richtercreate