Title: importlib examples have their exec_module()/sys.modules assignment lines reversed
Type: Stage: needs patch
Components: Documentation Versions: Python 3.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, eric.snow, iomintz, miss-islington, ncoghlan
Priority: normal Keywords: patch

Created on 2019-07-08 00:52 by iomintz, last changed 2019-07-17 22:20 by brett.cannon. This issue is now closed.

File name Uploaded Description Edit iomintz, 2019-07-09 22:32
Pull Requests
URL Status Linked Edit
PR 14723 merged brett.cannon, 2019-07-12 22:11
PR 14724 merged miss-islington, 2019-07-12 22:35
Messages (16)
msg347482 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-08 00:52
unzip the attached zip file and run

expected output:

<module 'testext.sub' (namespace)>

actual output:
<module 'testext.sub' (namespace)>

So what?
If you follow these directions, , you will put a stale reference in sys.modules even though a fresh reference is already there. If you use that code on a package with a subpackage, the subpackage will not be set as an attribute on the package.
msg347483 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-08 00:54

New expected output:
<module 'testext.sub' (namespace)>
<module 'testext.sub' (namespace)>

New actual output:
<module 'testext.sub' (namespace)>
Traceback (most recent call last):
  File "", line 14, in <module>
AttributeError: module 'testext' has no attribute 'sub'
msg347508 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-08 19:16
Mind re-uploading the examples files with a more appropriate file name? (Saying "import is f-'ed" is not motivating to those of us who wrote it to try and fix this.)
msg347582 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-09 22:32
Updated. Also removed some old code which had irrelevant imports that I thought I removed already. The expected and actual output remain the same.
msg347639 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-10 21:01
Thanks! I'll try to have a look when I can.
msg347759 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-12 21:44
So the issue is the lines assigning to sys.modules and loader.exec_module() are reversed. Had you not done an import in your testext.__init__ you never would have noticed, but since you are then the normal import system is noticing there's nothing in sys.modules and thus inserting a module (importlib.uilt.module_from_spec() always creates a new module so your module that you create never ends up in sys.modules).
msg347760 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-12 22:04
I will have a PR up shortly to fix the docs.
msg347761 - (view) Author: miss-islington (miss-islington) Date: 2019-07-12 22:35
New changeset 0827064c955f88df8ba5621d8e3d81be3cfc26a9 by Miss Islington (bot) (Brett Cannon) in branch 'master':
bpo-37521: No longer treat insertion into sys.modules as optional in importlib examples (GH-14723)
msg347762 - (view) Author: miss-islington (miss-islington) Date: 2019-07-12 22:51
New changeset bfb709b771230e7e466961b5ddf4852962602450 by Miss Islington (bot) in branch '3.8':
[3.8] bpo-37521: No longer treat insertion into sys.modules as optional in importlib examples (GH-14723) (GH-14724)
msg347763 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-12 22:52
Thanks for the report, Benjamin!
msg347844 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-13 16:42
Hmm, why is the assignment to sys.modules necessary at all, if module_from_spec() always does so?
msg347987 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-15 18:55
I'm not sure why you think importlib.util.module_from_spec() adds a module to sys.modules? doesn't say that nor does actually do that.
msg348028 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-16 16:21
I dunno, one of those three does. I suppose it's spec.loader.exec_module().
msg348047 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-17 00:11
Nope, loader.exec_module() doesn't either:,

So unfortunately I'm still not sure what you're talking about.
msg348076 - (view) Author: Io Mintz (iomintz) * Date: 2019-07-17 17:41
name = 'testext'

spec = importlib.util.find_spec(name)
print(name in sys.modules)
module = importlib.util.module_from_spec(spec)
print(name in sys.modules)
print(name in sys.modules)

This prints False, False, True though.
msg348084 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-07-17 22:20
If you're using your example code for 'testext' that you uploaded then that's expected because the import in your package causes a normal import which means the import machinery is adding things to sys.modules. Try it with a plain module that's empty so you don't have any other imports as a side-effect of imports and you will see the module isn't added to sys.modules.
Date User Action Args
2019-07-17 22:20:59brett.cannonsetmessages: + msg348084
2019-07-17 17:41:27iomintzsetmessages: + msg348076
2019-07-17 00:11:44brett.cannonsetmessages: + msg348047
2019-07-16 16:21:54iomintzsetmessages: + msg348028
2019-07-15 18:55:56brett.cannonsetmessages: + msg347987
2019-07-13 16:42:51iomintzsetmessages: + msg347844
2019-07-12 22:52:23brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg347763

stage: patch review -> needs patch
2019-07-12 22:51:51miss-islingtonsetmessages: + msg347762
2019-07-12 22:35:58miss-islingtonsetpull_requests: + pull_request14520
2019-07-12 22:35:42miss-islingtonsetnosy: + miss-islington
messages: + msg347761
2019-07-12 22:11:28brett.cannonsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request14519
2019-07-12 22:04:49brett.cannonsetmessages: + msg347760
2019-07-12 21:44:00brett.cannonsettitle: importlib.util.module_from_spec return value is not the same as in sys.modules -> importlib examples have their exec_module()/sys.modules assignment lines reversed
messages: + msg347759
components: + Documentation, - Library (Lib)
stage: needs patch
2019-07-10 21:01:07brett.cannonsetassignee: brett.cannon
messages: + msg347639
2019-07-09 22:32:30iomintzsetfiles: +

messages: + msg347582
2019-07-08 20:00:18brett.cannonsetfiles: -
2019-07-08 20:00:12brett.cannonsetfiles: -
2019-07-08 19:16:04brett.cannonsetmessages: + msg347508
2019-07-08 19:14:19brett.cannonsetnosy: + ncoghlan, eric.snow
2019-07-08 02:37:56xtreaksetnosy: + brett.cannon
2019-07-08 00:54:43iomintzsetfiles: +

messages: + msg347483
2019-07-08 00:52:26iomintzcreate