classification
Title: import_module potentially imports a module twice
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: meador.inge Nosy List: Ryan.Twitchell, brett.cannon, meador.inge, ncoghlan, python-dev
Priority: normal Keywords: patch

Created on 2011-12-12 21:43 by Ryan.Twitchell, last changed 2011-12-15 04:41 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue13591.patch meador.inge, 2011-12-13 15:17 review
issue13591-2.patch meador.inge, 2011-12-14 05:45 review
Messages (9)
msg149358 - (view) Author: Ryan Twitchell (Ryan.Twitchell) Date: 2011-12-12 21:43
Use of importlib's import_module function with modules belonging to a library
can cause some modules to be imported twice, if such a module is referenced
from sibling modules, and from __init__ in the package.  I suspect this is a
bug, or at best a nuance of packages that is quite subtle.

Easier to show with an example.  Below, the module my_lib.bar will be imported
twice.  Given the following file structure:

./scratch.py
./my_lib/__init__.py
./my_lib/foo.py
./my_lib/bar.py

And the following file contents:

./scratch.py:

	import importlib
	importlib.import_module('my_lib.bar')
	importlib.import_module('my_lib.foo')


./my_lib/__init__.py:

	import my_lib.bar

	# Or alternately
	#from . import bar


./my_lib/foo.py:

	from . import bar

	print('In foo, id(bar.baz): %s' % id(bar.baz))


./my_lib/bar.py:

	def baz():
		pass

	print('In bar, id(bar.baz): %s' % id(baz))


Running scratch.py results in my_lib.bar being imported twice:
$ echo $PYTHONPATH
.
$ python --version
Python 3.2.2
$ python ./scratch.py
In bar, id(bar.baz): 21328632
In bar, id(bar.baz): 21352240
In foo, id(bar.baz): 21352240


Replacing the calls to import_module with use of the import statement, or
__import__, or simply rearranging the order of the two calls all result in the
module my_lib.bar to be imported only once.  As does eliminating the import
statement in my_lib.__init__.

This may be a misunderstanding on my part regarding the intended use of
packages, but this behavior was quite unexpected, and rather difficult to track
down in real code.
msg149374 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-12-13 02:25
At first glance, I thought this might be just the circular import problem (#992389) appearing in a different guise. However, if that was the case, switching to an import statement or __import__ shouldn't have made any difference.

What do you see if you put an "import my_lib.bar" *between* the two importlib calls?
msg149392 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-13 15:17
I can reproduce this on tip.  What happens is that 
'importlib.import_module("my_lib.bar")' is effectively computed as:

   import my_lib
   import bar

by '_bootstrap._gcd_import'.  When '_gcd_import' goes to do the import
of 'bar' it does *not* check to see if 'bar' has already been imported
by the parent import.

Here is a patch *without* tests that fixes this.  I will add the tests
next.
msg149421 - (view) Author: Ryan Twitchell (Ryan.Twitchell) Date: 2011-12-14 04:18
Confirmed that this patch fixes the behavior shown in my original example, with 3.2.
msg149425 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-14 05:45
Updated patch with tests.
msg149480 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-12-14 21:29
Patch looks good to me.
msg149498 - (view) Author: Roundup Robot (python-dev) Date: 2011-12-15 04:28
New changeset d2504d30f259 by Meador Inge in branch '3.2':
Issue #13591: import_module potentially imports a module twice.
http://hg.python.org/cpython/rev/d2504d30f259

New changeset e8fb61a0a2d7 by Meador Inge in branch 'default':
Issue #13591: import_module potentially imports a module twice.
http://hg.python.org/cpython/rev/e8fb61a0a2d7
msg149499 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-12-15 04:30
Thanks for the review Brett.  Fix committed.
msg149500 - (view) Author: Roundup Robot (python-dev) Date: 2011-12-15 04:41
New changeset 541f215a31f7 by Meador Inge in branch '3.2':
Issue #13591: Moving the NEWS line to the right release.
http://hg.python.org/cpython/rev/541f215a31f7

New changeset 92e94fd303d4 by Meador Inge in branch 'default':
Issue #13591: Moving the NEWS line to the right release.
http://hg.python.org/cpython/rev/92e94fd303d4
History
Date User Action Args
2011-12-15 04:41:12python-devsetmessages: + msg149500
2011-12-15 04:30:14meador.ingesetstatus: open -> closed
resolution: fixed
messages: + msg149499

stage: commit review -> resolved
2011-12-15 04:28:05python-devsetnosy: + python-dev
messages: + msg149498
2011-12-14 21:29:11brett.cannonsetassignee: meador.inge
messages: + msg149480
stage: patch review -> commit review
2011-12-14 05:45:04meador.ingesetfiles: + issue13591-2.patch

messages: + msg149425
2011-12-14 04:18:08Ryan.Twitchellsetmessages: + msg149421
2011-12-13 15:46:37pitrousetversions: + Python 3.3
2011-12-13 15:17:19meador.ingesetfiles: + issue13591.patch
keywords: + patch
messages: + msg149392

stage: patch review
2011-12-13 02:25:15ncoghlansetmessages: + msg149374
2011-12-13 01:53:46meador.ingesetnosy: + meador.inge
2011-12-12 21:44:50ezio.melottisetnosy: + brett.cannon, ncoghlan
2011-12-12 21:43:23Ryan.Twitchellcreate