classification
Title: can't modify sys.modules during import with importlib
Type: Stage:
Components: Interpreter Core Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: benjamin.peterson, brett.cannon, eric.snow, pjenvey, python-dev
Priority: high Keywords:

Created on 2012-04-18 03:58 by benjamin.peterson, last changed 2012-04-18 14:56 by benjamin.peterson. This issue is now closed.

Messages (11)
msg158587 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-04-18 03:58
$ cat > x.py
import sys
sys.modules["x"] = 42

benjamin@localhost ~/dev/python/py3k $ python3
Python 3.2.2 (default, Feb 18 2012, 09:16:28) 
[GCC 4.5.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import x
>>> x
42

$ ./python 
Python 3.3.0a2+ (default:6762b943ee59, Apr 17 2012, 23:57:13) 
[GCC 4.5.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import x
>>> x
<module 'x' from './x.py'>

It's not clear to me whether it's the loader's responsibilty to handle this or __import__.
msg158589 - (view) Author: Philip Jenvey (pjenvey) * (Python committer) Date: 2012-04-18 04:50
__import__ needs the actual module on hand so it can e.g. attach it to its parent module
msg158590 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 05:06
Loaders are in charge of adding the module to sys.modules (per PEP 302).  importlib codifies this in the module_for_loader() decorator, which the default loaders use.
msg158591 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 05:12
3.3.0a2+:

>>> import x
>>> x
<module 'x' from './x.py'>
>>> import sys
>>> sys.modules['x']
5
>>> x
5
msg158592 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 05:30
_find_and_load() in importlib._bootstrap returns whatever the loader returns, which is the new module object.  The old code in import.c pulled it from sys.modules rather than using what the loader returned.  In both cases the respective object is what eventually gets bound to the name in the eval loop.

FWIW, the language reference says, "The first form of import statement binds the module name in the local namespace to the module object".    This looks like a corner case where backwards-compatibility breaks (when we finally start enforcing the rules).
msg158614 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-04-18 12:57
This is a really important usecase for many packages including py.test and twisted, though.
msg158623 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 14:27
that's a pretty sneaky hack, but I can see the (weak) point of it.  So, to keep backward compatibility, importlib._bootstrap._find_and_load() would have to return sys.modules[fullname] instead of the module returned by loader.load_module(fullname).

My inclination is to break the backward compatibility and work with py.test/twisted/etc. to set things right.  If we don't, then we should consider changing the spec of the import statement in the language reference.

The hash randomization case for breaking backward compatibility relied on "everyone know better" and "there aren't any big use cases" (for dependence on dict key order).  Here it's not so cut and dry.  Still, it seems like a candidate for breaking "backward compatibility", as long as the (legitimate) alternative is easy and a faithful substitute.

I was considering bringing this up on python-dev, but I'd rather hear Brett's point of view first.
msg158624 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-18 14:29
I honestly don't care enough to argue over this one (I was trying to save a dict lookup but unfortunately too many people have codified the behaviour already). Just revert http://hg.python.org/cpython/rev/005fd1fe31ab to get back the original behaviour.
msg158625 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-04-18 14:32
I would advocate breaking any compatability. In fact, I think it can be documented. This is a useful feature, and not hard to maintain.
msg158626 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-04-18 14:42
> I would advocate breaking any compatability.

Did you mean "against breaking any compatability"?  The problem is that it's just one more sticky little detail that adds to the complexity of understanding the import system.  It's things like this that turn people off to diving into hacking imports (for better or worse <wink>).  I'm fine with maintaining the status quo, as Nick would say, and documenting the behavior.  But I don't have to like it! ;)

-eric
msg158630 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-04-18 14:55
New changeset db5e3431ee4c by Benjamin Peterson in branch 'default':
rollback 005fd1fe31ab (see #14609 and #14582)
http://hg.python.org/cpython/rev/db5e3431ee4c
History
Date User Action Args
2012-04-18 14:56:12benjamin.petersonsetstatus: open -> closed
resolution: fixed
2012-04-18 14:55:56python-devsetnosy: + python-dev
messages: + msg158630
2012-04-18 14:42:58eric.snowsetmessages: + msg158626
2012-04-18 14:32:25benjamin.petersonsetmessages: + msg158625
2012-04-18 14:29:19brett.cannonsetmessages: + msg158624
2012-04-18 14:27:31eric.snowsetmessages: + msg158623
2012-04-18 12:57:17benjamin.petersonsetmessages: + msg158614
2012-04-18 05:30:05eric.snowsetmessages: + msg158592
2012-04-18 05:12:40eric.snowsetmessages: + msg158591
2012-04-18 05:06:02eric.snowsetnosy: + eric.snow
messages: + msg158590
2012-04-18 04:50:25pjenveysetnosy: + pjenvey
messages: + msg158589
2012-04-18 03:58:36benjamin.petersoncreate