classification
Title: Relative imports do not replace local variables
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Rolf Campbell, brett.cannon, eric.snow, ncoghlan, r.david.murray
Priority: normal Keywords:

Created on 2018-05-16 20:46 by Rolf Campbell, last changed 2018-05-25 13:55 by ncoghlan. This issue is now closed.

Messages (13)
msg316864 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-16 20:46
Relative imports do not replace local variables, but also don't fail.  This can cause some very strange outcomes like this simple example:

touch a.py; python3.6 -c 'a=7; b=5; from . import a as b; print(a,b)'

I would expect this to produce "7 <module ...>", but instead, it produces "7 7".

Tested in v3.6.2 and v3.6.5:
Python 3.6.5 (default, Mar 29 2018, 18:20:46) 
[GCC 8.0.1 20180317 (Red Hat 8.0.1-0.19)] on linux
msg316867 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-16 20:57
Under simple circumstances, this is only reproducible when either directly in an interactive Python session (or as -c), but I encountered this type of problem in a much more complicated project which was NOT running as part of an interactive Python session (or as part of a -c argument).
msg316872 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-05-16 21:40
It's importing 'a' from '.', which I guess in this context means from the current namespace (__main__), and a is 7.  You'll note that 'b' did get repointed, but it got repointed to what 'a' points to, instead of to 5.  If it really wasn't replacing the local, you'd get '7 5', not '7 7'.

So, this is weird but expected, I think.  You'll have to actually produce the example that doesn't involve __main__ if you want us to look in to that, but most likely it will be some similar phenomenon.

I'm not 100% sure this is the *desired* behavior of 'from . import', though, so I'm leaving this open for the import experts to look at.

Also, in 3.8 at least I get a warning from the import system, which is a clue something weird is going on :)
msg316916 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-05-17 12:01
As David notes, the issue in the example is the fact that you're setting "__main__.a", so "a.py" never gets imported as a module - it gets a hit on the parent module attribute, and hence stops there.
msg317000 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-18 01:56
Thanks David, I agree that my assumption that the local valiables were not being replaced is not really what was going on there.

I also agree that, while this might not strictly classify as a bug, it's probably not the most helpful/useful way that "from ." could be implemented for __main__. Why does it act different than in modules?

I have reproduced my original ploblem in a simplified scenario, for which I will raise a new bug.

Thanks
msg317050 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-18 19:30
Re-opening because I've found a simple example that does not involve __main__.

./func/__init__.py:func = 1
./func/__init__.py:from . import func
./func/__init__.py:print(f"Namespace value of func after func module import:{func}")
./func/func.py:print("Module imported")
./main.py:import func
./main.py:def func(x):
./main.py:	return x

If you create files that look like that, and then run: "python3.6 main.py" I get this output:
Namespace value of func after func module import:1

If I comment out the "func = 1" line, then func ends up being imported (and printing as a <module...>).
msg317060 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-05-18 21:43
It's the same answer.  __init__ *is* the package namespace, so you are setting the value of 'func' in the package (.) namespace, and what import is doing is correct.

I know this is confusing.  I banged my head against it while debugging a weird import problem in the anydb module, but it is working as designed.  I think the way __init__ works may fall under rule 14 of the Zen :)
msg317062 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-18 22:11
OK, OK, I think I finally understand what you mean here.  Let me try to repeat it just to make sure I really understand:

When requesting a member of a multi-file module (like "func" in my example), python only tries to load that member as a module from disk if there isn't something already created as part of __init__.py.

In my case, I'm trying to load "func.func" which I specifically created in line#1 of func/__init__.py, so Python sees no need to even try to load the func/func.py file.

If I comment-out the first line of func/__init__.py, then Python fails to find an item called "func.func" and so it tries to load one from disk which causes it to load "func/func.py".

My real problem here was that I shouldn't be creating entries in the "func" namespace that clash with on-disk sub-modules that I want loaded.

Thanks for your time and effort in explaining this.
msg317063 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-05-18 22:29
Yes, you are substantially correct.  A subtlety that may enhance your understanding (if it doesn't instead totally confuse you :) is that __init__ is simply the most straightforward way to affect the module namespace.  You would see the same phenomenon if in your package func had a submodule bar that did:

   import func
   func.func = 1
   from . import func

func would now be 1, because it already exists in the module namespace at the point the relative import is done, so import just returns it, it doesn't go looking for a module to import.
msg317390 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-23 12:06
OK, while I understand what you are saying, that is NOT how absolute imports work.  I'll give an example:

./main.py:import func
./main.py:print(f"Value of func.func after import func:{func.func}")
./main.py:import func.func
./main.py:print(f"Value of func.func after import func.func:{func.func}")
./func/__init__.py:func = 1
./func/__init__.py:from . import func
./func/__init__.py:print(f"Value of func after from . import func:{func}")
./func/func.py:print("Module imported")

Here, the relative import inside __init__.py does NOT load the "func.py" module because there is already an object called "func".

But, the absolute "import func.func" does load "func.py" even though there is already a "func.func" object.

Are these supposed to work differently?  That seems strange to me.
msg317405 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-05-23 13:26
Yes, while weird, that's expected behaviour.

Rather than being due to absolute vs relative imports, the difference arises from the fact that in "import pkg.module", the request is explicitly for a submodule, so the submodule import always happens, whereas if you write "from func import attr", the child module import is only attempted if "func.attr" fails to resolve after "func" is imported.

$ echo "print(__name__)" > pkg/__init__.py
$ echo "print(__name__)" > pkg/submodule.py

$ python3 -c "import pkg; pkg.submodule = 1; import pkg.submodule; print(pkg.submodule)"
pkg
pkg.submodule
<module 'pkg.submodule' from '/home/ncoghlan/devel/misc/_play/pkg/submodule.py'>

$ python3 -c "import pkg; pkg.submodule = 1; from pkg import submodule; print(pkg.submodule)"
pkg
1
msg317565 - (view) Author: Rolf Campbell (Rolf Campbell) Date: 2018-05-24 12:00
Is there any way to use relative imports and explicitly request a sub-module?
From PEP 328: "import <> is always absolute"

So it sounds like there is no way to duplicate the explicit request for a sub-module when using relative imports.
msg317673 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-05-25 13:55
Not as a statement, but you can force it with importlib.import_module:

$ python3 -c "import pkg; pkg.submodule = 1; import importlib; importlib.import_module('.submodule', 'pkg'); print(pkg.submodule)"
pkg
pkg.submodule
<module 'pkg.submodule' from '/home/ncoghlan/devel/misc/_play/pkg/submodule.py'>

We're getting off-topic for the issue tracker now, though - it's more a Stack Overflow type usage question (and the preferred answer would be to eliminate whatever's causing the shadowing problem in the originating module).
History
Date User Action Args
2018-05-25 13:55:24ncoghlansetmessages: + msg317673
2018-05-24 12:00:50Rolf Campbellsetmessages: + msg317565
2018-05-23 13:26:56ncoghlansetmessages: + msg317405
2018-05-23 12:06:30Rolf Campbellsetmessages: + msg317390
2018-05-18 22:29:23r.david.murraysetmessages: + msg317063
2018-05-18 22:11:53Rolf Campbellsetmessages: + msg317062
2018-05-18 21:43:28r.david.murraysetstatus: open -> closed
resolution: not a bug
messages: + msg317060
2018-05-18 19:31:34Rolf Campbellsetresolution: not a bug -> (no value)
2018-05-18 19:30:32Rolf Campbellsetstatus: closed -> open

messages: + msg317050
2018-05-18 01:56:19Rolf Campbellsetmessages: + msg317000
2018-05-17 12:01:54ncoghlansetstatus: open -> closed
resolution: not a bug
messages: + msg316916

stage: resolved
2018-05-16 21:40:02r.david.murraysetnosy: + eric.snow, r.david.murray, brett.cannon, ncoghlan
messages: + msg316872
2018-05-16 20:57:59Rolf Campbellsetmessages: + msg316867
2018-05-16 20:46:59Rolf Campbellcreate