classification
Title: attribute error due to circular import
Type: enhancement Stage: needs patch
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: rejected
Dependencies: Superseder: Modify IMPORT_FROM to fallback on sys.modules
View: 17636
Assigned To: Nosy List: Rhamphoryncus, brett.cannon, bronger, dcjim, eric.araujo, eric.snow, flox, jhylton, larry, loewis, ncoghlan, pje, sbt, tim.peters
Priority: normal Keywords: patch

Created on 2004-07-16 15:09 by dcjim, last changed 2014-11-14 01:56 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
eek.zip dcjim, 2004-07-16 15:09 zip of demonstration package
issue992389_set_parent_module_attribute.diff ncoghlan, 2013-12-17 13:47 Sketch of setting parent attribute while importing submodule review
Messages (25)
msg21663 - (view) Author: Jim Fulton (dcjim) Date: 2004-07-16 15:09
This bug applied to 2.3 and 2.4. It probably applies to
earlier versions, but who cares? :)

Under some circumstances, code like:

  import eek.foo.baz
  y = eek.foo.baz.y

fails with an attribute error for "foo" if foo is still
being imported.

I've attached a zip file of a demo package "eek" that
demonstrates the problem.  If you unzip the package and:

  import eek.foo

you'll get the attribute error described above.

I think the problem is that eek's foo attribute isn't
set until the import of foo is finished.  This is too late.
msg21664 - (view) Author: Jeremy Hylton (jhylton) Date: 2004-11-07 15:30
Logged In: YES 
user_id=31392

Are the semantics of import clear enough to confirm that
this is a bug?  Specifically, this seems to revolve around
circular imports and code in __init__.  I agree that this
behavior is confusing, but I don't know if that has more to
do with the nature of circular imports than any bug.

msg21665 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2004-11-15 11:08
Logged In: YES 
user_id=1038590

If the semantics aren't clear, then isn't that a bug in
itself? If the behaviour of Jim's code is officially
undefined, then the docs need to say so.

A simple rule would seem to be "a package should not try to
import itself or any subpackages with an absolute import in
its __init__.py file".

Given the nature of __all__ and __path__ for packages, I
can't see a way to safely set eek's foo attribute before
foo's __init__.py has been processed.
msg21666 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2004-11-16 02:03
Logged In: YES 
user_id=31435

Nick, the semantics of circular imports aren't clear even if no 
packages are involved.  Note that the Reference manual is 
silent about such cases.  In effect, it's defined by the 
implementation, and some people think they know how that 
works -- although nobody has credibly claimed to fully 
understand the implementation consequences since Gordon 
McMillan vanished <0.5 wink>.

While I expect this bug report to sit here for years (it's hard 
to imagine anyone caring enough to devote the time needed 
to untangle this subsystem), I'll note in passing that this 
case "works" if bar.py uses relative import instead; i.e., if it 
begins with

import baz
y = baz.y

instead of with

import eek.foo.baz
y = eek.foo.baz.y

Then it stops referencing attributes that don't exist before 
the whole chain of imports completes.
msg21667 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-07-03 12:38
Logged In: YES 
user_id=21627

Lowering the priority, as this apparently is not a
high-priority issue.
msg65198 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-04-08 17:09
I think the lowered priority got lost somewhere along the line.
msg65337 - (view) Author: Torsten Bronger (bronger) Date: 2008-04-11 06:32
I have a very similar issue (maybe the same?) at the moment.

Assume the follwing package structure:

main.py
package/
    __init__.py   [empty]
    moduleX.py
    moduleY.py

main.py says:

    from package import moduleX

moduleX.py says:

    from . import moduleY

and moduleY.py says:

    from . import moduleX

However, this doesn't work:

    bronger@wilson:~/temp/packages-test$ python main.py
    Traceback (most recent call last):
      File "main.py", line 1, in <module>
        from package import moduleX
      File "/home/bronger/temp/packages-test/package/moduleX.py", line
1, in <module>
        from . import moduleY
      File "/home/bronger/temp/packages-test/package/moduleY.py", line
1, in <module>
        from . import moduleX
    ImportError: cannot import name moduleX

If I turn the relative imports to absolutes ones, it works.  But I'd
prefer the relative notation for intra-package imports.  That's their
purpose after all.

If you split a large module into chunks, cyclic imports are hardly
avoidable (and there's nothing bad about it; it worked fine before PEP 328).

Note that "import absolute.path.to.module as short" doesn't work either.
 So currently, in presence of cyclic imports in a package, the only
remedy is to use the full absolute paths everywhere in the source code,
which is really awkward in my opinion.
msg65436 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-04-13 05:57
This is actually a pretty tough problem - fixing it would involve some
fairly subtle changes to the way imports from packages are handled.
Given that I'm of the opinion that permitting circular imports in a code
base is an extraordinarily bad coding practice (if I'm ever tempted to
create a circular import, I consider it a sign that I need to separate
out some of the common code into a utility module), I'm not personally
going to be putting any effort into solving it (although I wouldn't
necessarily oppose anyone else trying to fix it).

However, I'll give a detailed description of the problem and a possible
solution in case anyone else wants to tackle it (since I believe there's
already a similar trick done for the sys.modules cache).

At the moment, when resolving an import chain __import__ only sets the
parent package's attribute for the submodule after the submodule import
is complete. This is what Jim describes in his original post, and is the
cause of the failure to resolve the name. Deferring the lookups solves
the problem because it means the package attributes are checked only
after the whole import chain is complete, instead of trying to get
access to a half-executed module during the import itself.

The most likely solution to the problem would be to change the attribute
on the parent package to be handled in a fashion closer to the way the
sys.modules cache is handled: set the attribute on the parent package
*before* executing the module's code, and delete that attribute if a
problem is encountered with the import.

The consequence of this would be that circular imports would be
permitted, although the module's imported in this fashion would be seen
in a half constructed state. So instead of the import failing with an
exception (which is what happens now), you would instead get a module
which you can't actually use (since most its attributes won't actually
be filled in yet, as the circular imports will normally occur near the
top of the file). Attempts to use methods, attributes and functions from
the module may or may not work depending on the order in which the
original module does things.

A clean failure indicating "You have a circular import, get rid of it"
seems better to me than possible hard to diagnose bugs due to being able
to retrieve things from a half-constructed module, but opinions
obviously differ on that. However, I really don't see this changing
without a PEP.
msg65439 - (view) Author: Torsten Bronger (bronger) Date: 2008-04-13 08:15
I dare to make a follow-up although I have no idea at all about the
internal processes in the Python interpreter.  But I've experimented
with circular imports a lot recently.  Just two points:

First, I think that circular imports don't necessarily exhibit a
sub-opimal programming style.  I had a large parser module which I just
wanted to split in order to get handy file sizes.  However, the parser
parses human documents, and layout element A defined in module A may
contain element B from module B and vice versa.  In a language with
declarations, you just include a big header file but in Python, you end
up with circular imports.  Or, you must stay with large files.

So, while I think that this clean error message Nick suggests is a good
provisional solution, it should not make the impression that the
circular import is a flaw by itself.

And secondly, the problem with modules that are not yet populated with
objects is how circular imports have worked in Python anyway.  You can
easily cope with it by not referencing the imported module's objects in
the top-level code of the importing module (but only in functions and
methods).
msg84767 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-03-31 12:25
This came up on python-dev again recently:
http://mail.python.org/pipermail/python-dev/2009-March/087955.html
msg84799 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-03-31 15:25
Good sleuthing Nick!  It's clearly the same bug that Fredrik found.

I tried to test if using Brett' importlib has the same problem, but it
can import neither p.a nor p.b, so that's not helpful as to sorting out
the import semantics.

I believe that at some point many of the details of importlib should be
seen as the reference documentation for the darkest corners of import
semantics.  But it seems we aren't there yet.
msg84803 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-03-31 15:35
Sorry, never mind about the importlib bug, that was my mistake.
importlib actually behaves exactly the same way as the built-in import.

I conclude that this is probably the best semantics of import that we
can hope for in this corner case.

I propose to close this as "works as intended" -- and perhaps document
it somewhere.
msg84844 - (view) Author: Torsten Bronger (bronger) Date: 2009-03-31 17:22
Maybe it's better to leave it open, waiting for someone to pick it up,
even if this is some time in the future?

In my opinion, this is suprising behaviour without an actual rationale,
and a current implementation feature.  I'd be a pitty for me to see it
becoming an official part of the language.

What bothers me most is that

    from . import moduleX

doesn't work but

    import package.moduleX

does work.  So the circular import itself works without problems,
however, not with a handy identifier.  This is would be an odd
asymmetry, I think.
msg84922 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-03-31 21:42
I just had a thought: we may be able to eliminate this behaviour without
mucking about in the package globals.

What if the import semantics were adjusted so that, as a last gasp
effort before bailing out with an ImportError, the import process
checked sys.modules again with the full module name?

Not a fully fleshed out idea at this point (and possibly symptomatic of
not being fully awake yet), but I'll bring it up in the current
python-dev thread anyway.
msg84966 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-04-01 02:49
I'm sorely tempted to apply the Van Lindberg clause to the last two
responses by Torsten and Nick.  If there was an easy solution it
wouldn't have been open for five years.  If you don't believe me, post a
fix.  I'll even accept a fix for the importlib package, which should
lower the bar quite a bit compared to a fix for import.c.
msg84993 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-04-01 10:38
No argument from me that my suggestion is a mere glimmering of an idea,
rather than a fully worked out definitely viable solution.

It was just an angle of attack I hadn't seen suggested before, so I
figured it was worth mentioning - the fact that a module is allowed to
exist in sys.modules while only half constructed is the reason "import
a.b.c" can work while "from a.b import c" or an explicit relative import
will fail - the first approach gets a hit in sys.modules and succeeds,
while the latter two approaches fail because the a.b package doesn't
have a 'c' attribute yet.

Figuring out a way to set the attribute in the parent package and then
roll it back later if the import fails is still likely to be the more
robust approach.
msg92114 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2009-08-31 21:44
The key distinction between this and a "bad" circular import is that
this is lazy.  You may list the import at the top of your module, but
you never touch it until after you've finished importing yourself (and
they feel the same about you.)

An ugly fix could be done today for module imports by creating a proxy
that triggers the import upon the first attribute access.  A more
general solution could be done with a lazyimport statement, triggered
when the target module finishes importing; only problem there is the
confusing error messages and other oddities if you reassign that name.
msg92115 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-08-31 21:51
I have done a lazy importer like you describe, Adam, and it does help 
solve this issue. And it does have the problem of import errors being 
triggered rather late and in an odd spot.
msg92116 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2009-08-31 21:57
It'd probably be sufficient if we raised "NameError: lazy import 'foo'
not yet complete".  That should require a set of what names this module
is lazy importing, which is checked in the failure paths of module
attribute lookup and global/builtin lookup.
msg145614 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-10-16 06:48
Changed the issue title to state clearly that the core issue is with circular imports that attempt to reference module contents at import time, regardless of the syntactic form used. All of the following module level code can fail due to this problem:

    from . import a

    from package import submodule

    from module import a

    import module
    module.a
msg175592 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-11-14 21:21
In Torsten's example

    from . import moduleX

can be replaced with

    moduleX = importlib.import_module('.moduleX', __package__)         (*)

or

    moduleX = importlib.import_module('package.moduleX')

If that is not pretty enough then perhaps the new syntax

    import .moduleX

could be introduced and made equivalent to (*).
msg186889 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-04-14 07:59
The implementation of issue #17636 (making IMPORT_FROM fall back to sys.modules when appropriate) will make "import x.y" and "from x import y" equivalent for resolution purposes during import.

That covers off the subset of circular references that we want to allow, so I'm closing this one in favour of the more precisely defined proposal.
msg206433 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-17 13:47
I'm reopening this, since PEP 451 opens up new options for dealing with it (at least for loaders that export the PEP 451 APIs rather than only the legacy loader API, which now includes all the standard loaders other than the ones for builtins and extension modules)

The attached patch doesn't have an automated test and is quite rough around the edges (as the additional check for the parent being in sys.modules confuses a couple of the importlib tests), but it proves the concept by making the following work:

[ncoghlan@lancre py3k]$ cd ../play
[ncoghlan@lancre play]$ mkdir issue992389
[ncoghlan@lancre play]$ cat > issue992389/mod.py
from . import mod
print("Success!")
[ncoghlan@lancre play]$ python3 -c "import issue992389.mod"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./issue992389/mod.py", line 1, in <module>
    from . import mod
ImportError: cannot import name mod
[ncoghlan@lancre play]$ ../py3k/python -c "import issue992389.mod"
Success!
msg206475 - (view) Author: PJ Eby (pje) * (Python committer) Date: 2013-12-17 17:30
The new patch will have weird results in the case of a parent module that defines an attribute that's later replaced by an import, e.g. if foo/__init__.py defines a variable 'bar' that's a proxy for the foo.bar module.  This is especially problematic if this proxy is used during the process of importing foo.bar

At the very least, this code should NOT be deleting the original foo.bar attribute, but rather restoring its previous value.

All in all, I don't think this is a productive route to take.  It was discussed on Python-dev previously and IIRC I outlined all the other reasons why back then.  The approach in issue17636 is the only one that doesn't change the semantics of any existing, not-currently-broken code.

In contrast, the proposed change here introduces new side-effects and *more* volatile state and temporal coupling.  I don't think this should go in, since the other approach *only* affects execution paths that would currently raise ImportError.
msg231147 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-11-14 01:55
Belatedly agreeing with PJE on this one.

If concerns around circular imports continue to be raised in a post Python 3.5 world (with the issue 17636 change), then we can look at revisiting this again, but in the meantime, lets just go with the sys.modules fallback.
History
Date User Action Args
2014-11-14 01:56:17ncoghlansetsuperseder: Modify IMPORT_FROM to fallback on sys.modules
2014-11-14 01:55:49ncoghlansetstatus: open -> closed
resolution: rejected
messages: + msg231147
2013-12-17 17:30:59pjesetnosy: + pje
messages: + msg206475
2013-12-17 13:47:35ncoghlansetstatus: closed -> open
files: + issue992389_set_parent_module_attribute.diff
superseder: Modify IMPORT_FROM to fallback on sys.modules -> (no value)

versions: + Python 3.4, - Python 3.3
keywords: + patch
nosy: + larry

messages: + msg206433
resolution: duplicate -> (no value)
2013-04-14 07:59:21ncoghlansetstatus: open -> closed
superseder: Modify IMPORT_FROM to fallback on sys.modules
resolution: duplicate
messages: + msg186889
2012-11-14 21:21:14sbtsetnosy: + sbt
messages: + msg175592
2012-09-25 05:47:33ncoghlanlinkissue16031 superseder
2012-03-07 08:49:58eric.snowsetnosy: + eric.snow
2011-10-19 17:44:27floxsetstage: needs patch
versions: + Python 3.3, - Python 3.2
2011-10-16 07:20:16floxsetnosy: + flox
2011-10-16 06:48:14ncoghlansetmessages: + msg145614
title: attribute error after non-from import -> attribute error due to circular import
2011-10-16 06:44:55ncoghlanlinkissue13187 superseder
2010-11-19 13:58:26eric.araujosetnosy: + eric.araujo
2010-08-19 16:02:54gvanrossumsetnosy: - gvanrossum
2010-08-19 15:25:21BreamoreBoysetversions: + Python 3.2, - Python 3.1, Python 2.7
2009-08-31 21:57:15Rhamphoryncussetmessages: + msg92116
2009-08-31 21:51:20brett.cannonsetmessages: + msg92115
2009-08-31 21:44:37Rhamphoryncussetnosy: + Rhamphoryncus
messages: + msg92114
2009-04-02 03:56:02brett.cannonlinkissue966431 superseder
2009-04-01 18:41:02brett.cannonsetassignee: brett.cannon ->
2009-04-01 10:38:24ncoghlansetmessages: + msg84993
2009-04-01 02:49:57gvanrossumsetmessages: + msg84966
2009-03-31 21:42:44ncoghlansetmessages: + msg84922
2009-03-31 17:22:33brongersetmessages: + msg84844
2009-03-31 15:35:26gvanrossumsetmessages: + msg84803
2009-03-31 15:25:49gvanrossumsetnosy: + gvanrossum
messages: + msg84799
2009-03-31 12:25:11ncoghlansetmessages: + msg84767
2009-02-11 02:55:04ajaksu2setassignee: brett.cannon
versions: + Python 3.1, Python 2.7, - Python 2.3
nosy: + brett.cannon
2008-04-13 08:15:36brongersetmessages: + msg65439
2008-04-13 05:57:41ncoghlansetassignee: ncoghlan -> (no value)
type: enhancement
messages: + msg65436
2008-04-12 21:55:50georg.brandlsetpriority: low -> normal
assignee: ncoghlan
2008-04-11 06:32:42brongersetnosy: + bronger
messages: + msg65337
2008-04-08 17:09:24ncoghlansetpriority: normal -> low
messages: + msg65198
2004-07-16 15:09:33dcjimcreate