classification
Title: LazyLoader rejecting use of SourceFileLoader
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, eric.snow, ethan.furman, kdart, ncoghlan, pitrou, python-dev
Priority: normal Keywords:

Created on 2016-01-23 19:28 by brett.cannon, last changed 2016-06-25 17:59 by brett.cannon. This issue is now closed.

Messages (13)
msg258873 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-23 19:28
It was privately reported to me that importlib.util.LazyLoader rejects using importlib.machinery.SourceFileLoader (or at least _frozen_importlib.SourceFileLoader). At least a test should be added for LazyLoader to make sure it will happily accept importlib.machinery.SourceFileLoader, and if it rejects the one from _frozen_importlib, tweak the test to accept loaders from there as well.
msg258875 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-23 19:38
One way to possibly improve this is to remove the create_module() check and replace it with a warning if create_module() doesn't return None. Another option is to use an assert statement. The final option is to just drop the check entirely, although that can make debugging difficult.
msg259157 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-28 20:30
I think I'm liking the following approach:

  if __debug__:
    hopefully_None = loader.create_module(spec)
    if hopefully_None is not None:
        warnings.warn("watch out!", ImportWarning)
msg259584 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-02-04 17:52
Just stumbled on this very issue while trying to use LazyLoader.
msg259585 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-04 17:53
My current plan is to simply remove the check in 3.5 -- the docs say it's ignored so I don't think it will hurt anything -- and add the warning I proposed in 3.6. What do you think, Antoine?
msg259586 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-02-04 17:55
By the way does this mean the LazyLoader won't work with ExtensionFileLoader? That would reduce its usefulness quite a bit.
msg259587 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-02-04 17:56
I don't know what the impact of the error is, but it seems like at least the default loader classes should be able to work with LazyLoader...
msg259588 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-04 18:02
You're right, it won't work with extension modules based on how ExtensionFileLoader is structured.
msg260592 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-21 02:37
New changeset 9f1e680896ef by Brett Cannon in branch '3.5':
Issue #26186: Remove an invalid type check in
https://hg.python.org/cpython/rev/9f1e680896ef

New changeset 86fc6cdd65de by Brett Cannon in branch 'default':
Merge for issue #26186
https://hg.python.org/cpython/rev/86fc6cdd65de
msg260593 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-21 02:44
This has been resolved by removing the check (the docs have always said the method was ignored, so that will just continue). I also did separate commit to list that BuiltinImporter and ExtensionFileLoader won't work (they would need to be updated to return a module subclass to work).

I'm leaving this issue open to decide if I want to add an explicit check in importlib.util.LazyLoader.create_module() to verify that None is returned in Python 3.5 (I probably will, but I want to think about on it for a little bit).
msg263524 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-04-15 20:09
I actually think this can be generalized thanks to some tweaks made in Python 3.6 which will lead to it working with extension modules.

I need to double-check, but I think I can:
- Eliminate the _Module class (and thus have _LazyModule inherit types.ModuleType directly)
- Call create_module() and see if it returns something
- Use spec.loader_state to store the original value of module.__class__ (probably be simply storing __class__ in the dict of values)
- Just blindly try to set module.__class__ to _LazyModule and let the exception propagate if it fails
- In _LazyModule.__getattribute__, temporarily assign the type to types.ModuleType to shut off __getattribute__() before assigning the proper value
msg269244 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-25 17:58
New changeset 7af4b3ad75b7 by Brett Cannon in branch 'default':
Issue #26186: Remove the restriction that built-in and extension
https://hg.python.org/cpython/rev/7af4b3ad75b7
msg269245 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-06-25 17:59
The restriction of lazily loading built-ins and extensions is now lifted!
History
Date User Action Args
2016-06-25 17:59:11brett.cannonsetstatus: open -> closed

messages: + msg269245
2016-06-25 17:58:24python-devsetmessages: + msg269244
2016-04-15 20:09:11brett.cannonsetmessages: + msg263524
2016-02-21 02:44:23brett.cannonsetresolution: fixed
messages: + msg260593
stage: test needed -> resolved
2016-02-21 02:37:13python-devsetnosy: + python-dev
messages: + msg260592
2016-02-04 18:02:51brett.cannonsetmessages: + msg259588
2016-02-04 17:56:32pitrousetmessages: + msg259587
2016-02-04 17:55:41pitrousetmessages: + msg259586
2016-02-04 17:53:45brett.cannonsetmessages: + msg259585
2016-02-04 17:52:18pitrousetnosy: + pitrou
messages: + msg259584
2016-01-31 20:38:02ethan.furmansetnosy: + ethan.furman
2016-01-28 20:30:36brett.cannonsetmessages: + msg259157
2016-01-23 22:41:09kdartsetnosy: + kdart
2016-01-23 19:38:58brett.cannonsetnosy: + ncoghlan, eric.snow
2016-01-23 19:38:47brett.cannonsetmessages: + msg258875
2016-01-23 19:28:00brett.cannoncreate