classification
Title: relative import without parent succeeds with builtins.__import__
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: Ben Lewis2, brett.cannon, eric.smith, miss-islington, serhiy.storchaka, xtreak
Priority: normal Keywords: patch

Created on 2019-06-26 09:11 by Ben Lewis2, last changed 2019-09-12 09:31 by brett.cannon. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 14465 closed Ben Lewis2, 2019-06-29 15:49
PR 14956 merged Ben Lewis2, 2019-07-26 08:29
PR 15913 merged brett.cannon, 2019-09-11 10:40
PR 15925 merged miss-islington, 2019-09-11 11:38
PR 16003 merged xtreak, 2019-09-11 18:28
Messages (14)
msg346593 - (view) Author: Ben Lewis (Ben Lewis2) * Date: 2019-06-26 09:11
>>> from curses import ascii
>>> from . import ascii

The second line should raise an ImportError but instead succeeds (tested cpython 3.6.7, 3.7.0 and 3.7.3, and from interactive interpreter and scripts). Specifically, builtins.__import__ does not reproduce the behaviour of importlib._bootstrap.__import__; maybe ceval.c:import_from is neglecting to check that there are parent packages when attempting a relative import?

More details here: https://stackoverflow.com/a/56768129/5104777
msg346654 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-26 18:05
I can't reproduce:

>>> from importlib import abc
>>> from . import util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'util' from '__main__' (unknown location)
>>> import importlib.util
>>>

Did you happen to do an `import *` prior to this in your REPL? If so that might explain it as you could have pulled in __package__ and such and that's used to resolve relative imports.
msg346710 - (view) Author: Ben Lewis (Ben Lewis2) * Date: 2019-06-27 04:45
>>> foo = 'oops'
>>> from . import foo as fubar   # should raise ImportError
>>> fubar
'oops'

After further investigation, the problem is that builtins.__import__ (the c port version) does not replicate the behaviour of importlib.__import__ (the python reference version):

>>> import builtins, importlib
>>> __package__ is None
True
>>> importlib.__import__('', globals(), locals(), ('foo',), 1)
ImportError
>>> builtins.__import__('', globals(), locals(), ('foo',), 1)
<module '__main__' (built-in)>

A further discrepancy is that for deeper relative imports, builtins.__import__ raises a ValueError instead of ImportError (contrary to expectation/spec):

>>> from ...... import foo
ValueError

A simple work around uses the python implementation to restore expected behaviour:

>>> builtins.__import__ = importlib.__import__
>>> from ...... import foo
ImportError
>>> from curses import ascii
>>> from . import ascii
ImportError

PS: Brett Cannon, to replicate please copy and paste lines in correct order :-)
msg346781 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-27 22:01
Please open a separate issue for the relative import issue.
msg346856 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-28 19:22
I opened bpo-37444 for the relative import beyond top-level package issue.
msg346857 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-28 19:25
The code doing the sanity check for importlib can be found at https://github.com/python/cpython/blob/f9f8e3ce709ceb15c8db8c8dde940daf1febf13d/Lib/importlib/_bootstrap.py#L943-L948 .
msg346861 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-06-28 19:34
If you run with `-Xdev`/warnings turned on you get an idea of what's happening:

>>> builtins.__import__('', globals(), locals(), ('foo',), 1)
<stdin>:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
<module '__main__' (built-in)>

The check is being done in resolve_name() in import.c (https://github.com/python/cpython/blob/f9f8e3ce709ceb15c8db8c8dde940daf1febf13d/Python/import.c#L1543). My guess is there's an off-by-one error in the sanity check logic for attempting a relative import beyond the top-level package.
msg346866 - (view) Author: Ben Lewis (Ben Lewis2) * Date: 2019-06-28 21:12
I'll look into this further and open a PR initially for the regression tests.
msg351794 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-09-11 10:09
New changeset 92420b3e679959a7d0ce875875601a4cee45231e by Brett Cannon (Ben Lewis) in branch 'master':
bpo-37409: fix relative import with no parent (#14956)
https://github.com/python/cpython/commit/92420b3e679959a7d0ce875875601a4cee45231e
msg351832 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-09-11 11:38
New changeset 0a6693a469cfb1dd5c8048d8cb4231a7b5883251 by Brett Cannon in branch '3.8':
[3.8] bpo-37409: fix relative import with no parent (GH-14956) (GH-15913)
https://github.com/python/cpython/commit/0a6693a469cfb1dd5c8048d8cb4231a7b5883251
msg351848 - (view) Author: miss-islington (miss-islington) Date: 2019-09-11 12:50
New changeset f3480ad08823a9bc7df490bb5b54593d9483be70 by Miss Islington (bot) in branch '3.7':
[3.8] bpo-37409: fix relative import with no parent (GH-14956) (GH-15913)
https://github.com/python/cpython/commit/f3480ad08823a9bc7df490bb5b54593d9483be70
msg351948 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-09-11 15:35
The test added seems to have created an ImportWarning in test_builtin.BuiltinTest.test_import .

./python.exe -Wall -m unittest -v test.test_builtin.BuiltinTest.test_import
test_import (test.test_builtin.BuiltinTest) ... /Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/case.py:202: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  callable_obj(*args, **kwargs)
ok

----------------------------------------------------------------------
Ran 1 test in 0.002s

OK
msg352004 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-09-11 17:53
The test is same as below and given that __spec__ an __name__ are passed as None where ImportWarning is raised in Lib/importlib/_bootstrap.py 1074 . can we just use self.assertWarns(ImportWarning) in the test?


>>> __import__('', {'__package__': None, '__spec__': None, '__name__': '__main__'}, locals={}, fromlist=('foo',), level=1)
<stdin>:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: attempted relative import with no known parent package
msg352075 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-09-12 09:31
Thanks for catching the warning and the fix, Karthikeyan!
History
Date User Action Args
2019-09-12 09:31:18brett.cannonsetmessages: + msg352075
2019-09-11 18:28:35xtreaksetpull_requests: + pull_request15628
2019-09-11 17:53:18xtreaksetmessages: + msg352004
2019-09-11 15:35:04xtreaksetnosy: + xtreak
messages: + msg351948
2019-09-11 13:04:52brett.cannonsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-11 12:50:18miss-islingtonsetnosy: + miss-islington
messages: + msg351848
2019-09-11 11:38:32miss-islingtonsetpull_requests: + pull_request15565
2019-09-11 11:38:25brett.cannonsetmessages: + msg351832
2019-09-11 10:40:54brett.cannonsetpull_requests: + pull_request15554
2019-09-11 10:09:52brett.cannonsetmessages: + msg351794
2019-08-02 22:11:28brett.cannonsetassignee: brett.cannon
2019-07-26 08:29:52Ben Lewis2setpull_requests: + pull_request14724
2019-06-29 15:49:49Ben Lewis2setkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request14282
2019-06-28 21:12:20Ben Lewis2setmessages: + msg346866
2019-06-28 19:34:59brett.cannonsetmessages: + msg346861
2019-06-28 19:33:51serhiy.storchakasetnosy: + serhiy.storchaka
2019-06-28 19:28:23brett.cannonsettitle: relative import without parent -> relative import without parent succeeds with builtins.__import__
2019-06-28 19:25:37brett.cannonsetmessages: + msg346857
2019-06-28 19:22:37brett.cannonsetmessages: + msg346856
2019-06-28 19:19:00brett.cannonsetversions: + Python 3.8, Python 3.9, - Python 3.7
2019-06-27 22:01:16brett.cannonsetmessages: + msg346781
stage: resolved -> test needed
2019-06-27 04:45:43Ben Lewis2setstatus: closed -> open
resolution: rejected -> (no value)
messages: + msg346710

title: relative import_from without parent -> relative import without parent
2019-06-26 18:05:28brett.cannonsetstatus: open -> closed
resolution: rejected
messages: + msg346654

stage: resolved
2019-06-26 09:31:21eric.smithsetnosy: + brett.cannon, eric.smith
2019-06-26 09:11:26Ben Lewis2create