classification
Title: Still getting two copies of importlib._bootstrap
Type: behavior Stage: resolved
Components: Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Arfrever, brett.cannon, georg.brandl, meador.inge, ncoghlan, python-dev, vstinner
Priority: release blocker Keywords:

Created on 2012-07-18 11:31 by ncoghlan, last changed 2019-07-14 17:31 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 14661 merged vstinner, 2019-07-09 09:55
Messages (8)
msg165757 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-18 11:31
I haven't worked out how yet, but importlib.machinery is managing to bypass the replacement of importlib._bootstrap with _frozen_importlib:

Python 3.3.0b1 (default:8bf691d0b004+, Jul 15 2012, 23:20:06) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib._bootstrap
[74500 refs]
>>> import importlib.machinery
[74500 refs]
>>> importlib.machinery.FileFinder
<class 'importlib._bootstrap.FileFinder'>
[74505 refs]
>>> importlib._bootstrap.FileFinder
<class '_frozen_importlib.FileFinder'>
[74505 refs]
>>> importlib.machinery.FileFinder.path_hook()("")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ncoghlan/devel/py3k/Lib/importlib/_bootstrap.py", line 1350, in path_hook_for_FileFinder
    if not _path_isdir(path):
  File "/home/ncoghlan/devel/py3k/Lib/importlib/_bootstrap.py", line 117, in _path_isdir
    path = _os.getcwd()
NameError: global name '_os' is not defined
[74566 refs]
>>> importlib._bootstrap.FileFinder.path_hook()("")
FileFinder('.')
msg165768 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-18 13:04
I would have said that it wasn't because it was just the class type not picking up on the __name__ re-assignment, but that global name failure states otherwise.
msg165771 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-18 13:27
Yup. A simpler demonstration:

Python 3.3.0b1 (default:8bf691d0b004+, Jul 15 2012, 23:20:06) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from importlib import _bootstrap, machinery
>>> _bootstrap.FileFinder is machinery.FileFinder
False

My initial theory (a missing sys.modules entry) failed miserably, as import.__init__ already has the following line:

    sys.modules['importlib._bootstrap'] = _bootstrap

I'm guessing it has to be some weird interaction with the explicit relative import, but I would have expected that to pick up on either the parent module attribute or the sys.modules entry.
msg165774 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-18 13:43
Checking with importlib.abc and importlib.util (which turned out not to have the problem) put me on the right path: the problem is the import of "imp" in importlib.__init__

What appears to be happening is that the interpreter sees the partially initialised importlib package, considers it good enough, and continues on with importing _bootstrap and machinery from source. Control eventually returns to __init__ and it overwrites the source version with the frozen version.

Changing the __init__ file to do "import _imp" instead (the same as what import_init does in the C code) was enough to fix it.

No patch yet - the actual fix is trivial, but it needs a test.
msg165777 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-18 13:52
Nice catch! Obviously that needs a big comment to avoid that problem in the future.
msg165923 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-07-20 13:40
New changeset 4431dc4bb770 by Nick Coghlan in branch 'default':
Close #15386: There was a loophole that meant importlib.machinery and imp would sometimes reference an uninitialised copy of importlib._bootstrap
http://hg.python.org/cpython/rev/4431dc4bb770
msg165924 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-20 13:49
That was a *lot* harder than I expected to create a test for - I had to import importlib right at the start of regrtest, as well as tweak the import order in runpy, and doing so actually caused test_import to *crash* completely without the bug fixed.

The trick was realising that *anything* importing imp before importlib would mask the bug.
msg347928 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-14 17:31
New changeset 8b7db5a1114e2113a756bdf8877fbe366055c69a by Victor Stinner in branch 'master':
bpo-37473: Don't import importlib ASAP in tests (GH-14661)
https://github.com/python/cpython/commit/8b7db5a1114e2113a756bdf8877fbe366055c69a
History
Date User Action Args
2019-07-14 17:31:24vstinnersetnosy: + vstinner
messages: + msg347928
2019-07-09 09:55:31vstinnersetpull_requests: + pull_request14469
2012-07-20 13:49:59ncoghlansetmessages: + msg165924
2012-07-20 13:40:21python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg165923

resolution: fixed
stage: test needed -> resolved
2012-07-20 01:54:53meador.ingesetnosy: + meador.inge
2012-07-20 01:17:57ncoghlansetassignee: ncoghlan
2012-07-18 16:59:09Arfreversetnosy: + Arfrever
2012-07-18 13:52:59brett.cannonsetmessages: + msg165777
versions: + Python 3.3
2012-07-18 13:43:51ncoghlansetpriority: normal -> release blocker

nosy: + georg.brandl
messages: + msg165774

type: behavior
stage: test needed
2012-07-18 13:27:00ncoghlansetmessages: + msg165771
2012-07-18 13:04:05brett.cannonsetnosy: + brett.cannon
messages: + msg165768
2012-07-18 11:31:42ncoghlansettitle: Still getting two copies of importlib._boostrap -> Still getting two copies of importlib._bootstrap
2012-07-18 11:31:34ncoghlancreate