classification
Title: relative import headaches
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: duplicate
Dependencies: Superseder: attribute error due to circular import
View: 992389
Assigned To: Nosy List: Jimbofbx, ncoghlan
Priority: normal Keywords:

Created on 2012-09-25 01:07 by Jimbofbx, last changed 2012-09-25 05:47 by ncoghlan. This issue is now closed.

Messages (2)
msg171208 - (view) Author: James Hutchison (Jimbofbx) Date: 2012-09-25 01:07
This might even be a bug I've stumbled upon but I'm listing it as an enhancement for now.

I really feel that relative imports in Python should just work. Regardless of the __name__, I should be able to import below me. Likewise, it should work even if I've already done an import into the symbol table. It adds additional work to us as a developer to have to do some pythonpath or code gymnastics to get something rather trivial working. Additionally, the import errors from circular imports add another challenge to work around. In C/C++ you can force it to import a file once and only once, why can't Python work the same way?

Take the following example set-up:
startPoint.py
subModule1
   /__init__.py
   /a.py
   /b.py
   /tests
      /__init__.py
      /test_a.py

a's code:
print("in a");
from subModule1 import b

b's code:
print("in b");
from subModule1 import a

test_a.py's code:
print("myname:",__name__);
from .. import a

startPoint.py is empty, and the __init__.py files are also empty.

If I run a PyDev unit test on test_a.py this is what I get:

Finding files... done.
Importing test modules ... myname: subModule1.tests.test_a
in a
in b
myname: test_a
Traceback (most recent call last):
  File "C:\eclipse\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
    mod = __import__(modname)
  File "C:\myfolder/relativeImportTest/subModule1/tests\test_a.py", line 6, in <module>
    from .. import a
ValueError: Attempted relative import in non-package

Clearly in this case, the exception given doesn't make any sense. I have __init__.py, the error says the relative import line is failing, and the error says it's because I'm in a non-package. Except, I'm in a package. It seems to go through the a_test.py file twice, even though I never explicitly import it. The first time through, I'm clearly in a package. The second time through, my name is NOT __main__ but yet I'm apparently no longer a package, which is where it fails.

Now if I change:
"from subModule1 import b" to "import subModule1.b"
and
"from subModule1 import a" to "import subModule1.a"

then everything works. But then that means I have to reference everything by the full name in my submodules. In this example, there's clearly a circular reference between a and b that wouldn't work anyways.

So lets change some things.

Now:
a.py:
import subModule1.b

b.py:
from subModule1 import a

Now the circular reference is gone between a and b. I really don't like having to do this as a means to work around a circular reference because it forces me to vary the import style of one file to another.

If we try the test code again however, it gets the same problem. If I swap which file does the relative import, then it works.

So lets make one last change:

test_a.py:
import subModule1.b # added
from .. import a

This will work, seemingly magically. It only runs the code in test_a.py once. Recall that the code in a.py is "import subModule1.b"

So basically this brings up several issues:
1. "import a.b" isn't the same as "from a import b" by more than how you reference it in the code
2. submodules are re-imported as non-module without ever importing them if you import their parent module relatively. If this is documented I don't know where.
3. import order can matter drastically to if a code runs or not for seemingly magical reasons.

And back when I was a beginner Python user, the naming convention of the full path really threw a monkey wrench in my code when I would try to move a select number of files from one project to another, or would try relative imports. If relative imports cause such headaches with circular references then I should generally stick to the full module path when referencing things. But if the full module path isn't portable then I should use relative imports.

Likewise, if I run as a PyDev unitTest, my module name is NOT __main__, so special path checks for __main__ won't work

I think the bottom line is that the import system gave me headaches as a beginner user and as an advanced user it still does every now and then so it really should be changed to something more intuitive or forgiving. I really shouldn't have to sit and think "how do I reference a function in the file just one directory level below mine?"

If there is already some magic bullet for this then it should probably be more visible.
msg171220 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-25 05:47
There is no magic bullet, but there are a whole mess of backwards compatibility constraints.

Closing this as a duplicate, because there's no clear resolvable RFE here beyond the known issues with circular imports and the differences between lazy and eager imports (already well documented in #992389, although potentially easier to resolve in 3.4 now that we have migrated to importlib).

Much of the rest of the complaint looks like a bug in pydev, as it appears to be running from a string *without creating an appropriate entry in sys.modules* first. Thus it *is* running through test_a twice due to the self import.

Many of these apparent import problems stem from poor reimplementations of import mechanics in tools like pydev and nose violating import system invariants. With importlib being used as the reference import implementation and the language reference finally including formally documented import semantics in 3.3+, this problem should hopefully reduce over time, as these ad hoc reimplementations are replaced by appropriate usage of importlib and runpy.

(Of course, the import system initialisation process is not without problems of its own, as PEP 395 describes)
History
Date User Action Args
2012-09-25 05:47:33ncoghlansetstatus: open -> closed

superseder: attribute error due to circular import
versions: + Python 3.4, - Python 3.2
nosy: + ncoghlan

messages: + msg171220
resolution: duplicate
2012-09-25 01:07:47Jimbofbxcreate