classification
Title: importlib examples have their exec_module()/sys.modules assignment lines reversed
Type: Stage: needs patch
Components: Documentation Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, eric.snow, iomintz, miss-islington, ncoghlan
Priority: normal Keywords: patch

Created on 2019-07-08 00:52 by iomintz, last changed 2019-07-17 22:20 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
importlib-util-module-from-spec-stale-reference.zip iomintz, 2019-07-09 22:32
Pull Requests
URL Status Linked Edit
PR 14723 merged brett.cannon, 2019-07-12 22:11
PR 14724 merged miss-islington, 2019-07-12 22:35
Messages (16)
msg347482 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-08 00:52
unzip the attached zip file and run main.py

expected output:

True
<module 'testext.sub' (namespace)>

actual output:
False
<module 'testext.sub' (namespace)>

So what?
If you follow these directions, https://docs.python.org/3.7/library/importlib.html#checking-if-a-module-can-be-imported , you will put a stale reference in sys.modules even though a fresh reference is already there. If you use that code on a package with a subpackage, the subpackage will not be set as an attribute on the package.
msg347483 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-08 00:54
Updated main.py.

New expected output:
====================
True
<module 'testext.sub' (namespace)>
<module 'testext.sub' (namespace)>

New actual output:
==================
False
<module 'testext.sub' (namespace)>
Traceback (most recent call last):
  File "main.py", line 14, in <module>
    print(module.sub)
AttributeError: module 'testext' has no attribute 'sub'
msg347508 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-08 19:16
Mind re-uploading the examples files with a more appropriate file name? (Saying "import is f-'ed" is not motivating to those of us who wrote it to try and fix this.)
msg347582 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-09 22:32
Updated. Also removed some old code which had irrelevant imports that I thought I removed already. The expected and actual output remain the same.
msg347639 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-10 21:01
Thanks! I'll try to have a look when I can.
msg347759 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-12 21:44
So the issue is the lines assigning to sys.modules and loader.exec_module() are reversed. Had you not done an import in your testext.__init__ you never would have noticed, but since you are then the normal import system is noticing there's nothing in sys.modules and thus inserting a module (importlib.uilt.module_from_spec() always creates a new module so your module that you create never ends up in sys.modules).
msg347760 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-12 22:04
I will have a PR up shortly to fix the docs.
msg347761 - (view) Author: miss-islington (miss-islington) Date: 2019-07-12 22:35
New changeset 0827064c955f88df8ba5621d8e3d81be3cfc26a9 by Miss Islington (bot) (Brett Cannon) in branch 'master':
bpo-37521: No longer treat insertion into sys.modules as optional in importlib examples (GH-14723)
https://github.com/python/cpython/commit/0827064c955f88df8ba5621d8e3d81be3cfc26a9
msg347762 - (view) Author: miss-islington (miss-islington) Date: 2019-07-12 22:51
New changeset bfb709b771230e7e466961b5ddf4852962602450 by Miss Islington (bot) in branch '3.8':
[3.8] bpo-37521: No longer treat insertion into sys.modules as optional in importlib examples (GH-14723) (GH-14724)
https://github.com/python/cpython/commit/bfb709b771230e7e466961b5ddf4852962602450
msg347763 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-12 22:52
Thanks for the report, Benjamin!
msg347844 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-13 16:42
Hmm, why is the assignment to sys.modules necessary at all, if module_from_spec() always does so?
msg347987 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-15 18:55
I'm not sure why you think importlib.util.module_from_spec() adds a module to sys.modules? https://docs.python.org/3/library/importlib.html#importlib.util.module_from_spec doesn't say that nor does https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L549 actually do that.
msg348028 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-16 16:21
I dunno, one of those three does. I suppose it's spec.loader.exec_module().
msg348047 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-17 00:11
Nope, loader.exec_module() doesn't either: https://docs.python.org/3/library/importlib.html#importlib.abc.Loader.exec_module, https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py#L776.

So unfortunately I'm still not sure what you're talking about.
msg348076 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-17 17:41
```
name = 'testext'

spec = importlib.util.find_spec(name)
print(name in sys.modules)
module = importlib.util.module_from_spec(spec)
print(name in sys.modules)
spec.loader.exec_module(module)
print(name in sys.modules)
```

This prints False, False, True though.
msg348084 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-17 22:20
If you're using your example code for 'testext' that you uploaded then that's expected because the import in your package causes a normal import which means the import machinery is adding things to sys.modules. Try it with a plain module that's empty so you don't have any other imports as a side-effect of imports and you will see the module isn't added to sys.modules.
History
Date User Action Args
2019-07-17 22:20:59brett.cannonsetmessages: + msg348084
2019-07-17 17:41:27iomintzsetmessages: + msg348076
2019-07-17 00:11:44brett.cannonsetmessages: + msg348047
2019-07-16 16:21:54iomintzsetmessages: + msg348028
2019-07-15 18:55:56brett.cannonsetmessages: + msg347987
2019-07-13 16:42:51iomintzsetmessages: + msg347844
2019-07-12 22:52:23brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg347763

stage: patch review -> needs patch
2019-07-12 22:51:51miss-islingtonsetmessages: + msg347762
2019-07-12 22:35:58miss-islingtonsetpull_requests: + pull_request14520
2019-07-12 22:35:42miss-islingtonsetnosy: + miss-islington
messages: + msg347761
2019-07-12 22:11:28brett.cannonsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request14519
2019-07-12 22:04:49brett.cannonsetmessages: + msg347760
2019-07-12 21:44:00brett.cannonsettitle: importlib.util.module_from_spec return value is not the same as in sys.modules -> importlib examples have their exec_module()/sys.modules assignment lines reversed
messages: + msg347759
components: + Documentation, - Library (Lib)
stage: needs patch
2019-07-10 21:01:07brett.cannonsetassignee: brett.cannon
messages: + msg347639
2019-07-09 22:32:30iomintzsetfiles: + importlib-util-module-from-spec-stale-reference.zip

messages: + msg347582
2019-07-08 20:00:18brett.cannonsetfiles: - importsfucked.zip
2019-07-08 20:00:12brett.cannonsetfiles: - importsfucked.zip
2019-07-08 19:16:04brett.cannonsetmessages: + msg347508
2019-07-08 19:14:19brett.cannonsetnosy: + ncoghlan, eric.snow
2019-07-08 02:37:56xtreaksetnosy: + brett.cannon
2019-07-08 00:54:43iomintzsetfiles: + importsfucked.zip

messages: + msg347483
2019-07-08 00:52:26iomintzcreate