Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

relative import without parent succeeds with builtins.__import__ #81590

Closed
benjimin mannequin opened this issue Jun 26, 2019 · 14 comments
Closed

relative import without parent succeeds with builtins.__import__ #81590

benjimin mannequin opened this issue Jun 26, 2019 · 14 comments
Assignees
Labels
3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@benjimin
Copy link
Mannequin

benjimin mannequin commented Jun 26, 2019

BPO 37409
Nosy @brettcannon, @ericvsmith, @serhiy-storchaka, @miss-islington, @tirkarthi, @benjimin
PRs
  • [WIP] bpo-37409: Regression tests for builtins/importlib __import__ divergence #14465
  • bpo-37409: fix relative import with no parent #14956
  • [3.8] bpo-37409: fix relative import with no parent (GH-14956) #15913
  • [3.7] bpo-37409: fix relative import with no parent (GH-14956) (GH-15913) #15925
  • bpo-37409: Fix ImportWarning regarding __spec__ and __package__ being None #16003
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/brettcannon'
    closed_at = <Date 2019-09-11.13:04:52.828>
    created_at = <Date 2019-06-26.09:11:26.702>
    labels = ['interpreter-core', 'type-bug', '3.8', '3.9']
    title = 'relative import without parent succeeds with builtins.__import__'
    updated_at = <Date 2019-09-12.09:31:18.058>
    user = 'https://github.com/benjimin'

    bugs.python.org fields:

    activity = <Date 2019-09-12.09:31:18.058>
    actor = 'brett.cannon'
    assignee = 'brett.cannon'
    closed = True
    closed_date = <Date 2019-09-11.13:04:52.828>
    closer = 'brett.cannon'
    components = ['Interpreter Core']
    creation = <Date 2019-06-26.09:11:26.702>
    creator = 'Ben Lewis2'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 37409
    keywords = ['patch']
    message_count = 14.0
    messages = ['346593', '346654', '346710', '346781', '346856', '346857', '346861', '346866', '351794', '351832', '351848', '351948', '352004', '352075']
    nosy_count = 6.0
    nosy_names = ['brett.cannon', 'eric.smith', 'serhiy.storchaka', 'miss-islington', 'xtreak', 'Ben Lewis2']
    pr_nums = ['14465', '14956', '15913', '15925', '16003']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue37409'
    versions = ['Python 3.8', 'Python 3.9']

    @benjimin
    Copy link
    Mannequin Author

    benjimin mannequin commented Jun 26, 2019

    >> from curses import ascii
    >> from . import ascii

    The second line should raise an ImportError but instead succeeds (tested cpython 3.6.7, 3.7.0 and 3.7.3, and from interactive interpreter and scripts). Specifically, builtins.__import__ does not reproduce the behaviour of importlib._bootstrap.__import__; maybe ceval.c:import_from is neglecting to check that there are parent packages when attempting a relative import?

    More details here: https://stackoverflow.com/a/56768129/5104777

    @benjimin benjimin mannequin added 3.7 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Jun 26, 2019
    @brettcannon
    Copy link
    Member

    I can't reproduce:

    >>> from importlib import abc
    >>> from . import util
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: cannot import name 'util' from '__main__' (unknown location)
    >>> import importlib.util
    >>>

    Did you happen to do an import * prior to this in your REPL? If so that might explain it as you could have pulled in package and such and that's used to resolve relative imports.

    @benjimin
    Copy link
    Mannequin Author

    benjimin mannequin commented Jun 27, 2019

    >>> foo = 'oops'
    >>> from . import foo as fubar   # should raise ImportError
    >>> fubar
    'oops'

    After further investigation, the problem is that builtins.__import__ (the c port version) does not replicate the behaviour of importlib.__import__ (the python reference version):

    >>> import builtins, importlib
    >>> __package__ is None
    True
    >>> importlib.__import__('', globals(), locals(), ('foo',), 1)
    ImportError
    >>> builtins.__import__('', globals(), locals(), ('foo',), 1)
    <module '__main__' (built-in)>

    A further discrepancy is that for deeper relative imports, builtins.__import__ raises a ValueError instead of ImportError (contrary to expectation/spec):

    >>> from ...... import foo
    ValueError

    A simple work around uses the python implementation to restore expected behaviour:

    >>> builtins.__import__ = importlib.__import__
    >>> from ...... import foo
    ImportError
    >>> from curses import ascii
    >>> from . import ascii
    ImportError

    PS: Brett Cannon, to replicate please copy and paste lines in correct order :-)

    @benjimin benjimin mannequin changed the title relative import_from without parent relative import without parent Jun 27, 2019
    @benjimin benjimin mannequin reopened this Jun 27, 2019
    @brettcannon
    Copy link
    Member

    Please open a separate issue for the relative import issue.

    @brettcannon brettcannon added 3.8 only security fixes 3.9 only security fixes and removed 3.7 (EOL) end of life labels Jun 28, 2019
    @brettcannon
    Copy link
    Member

    I opened bpo-37444 for the relative import beyond top-level package issue.

    @brettcannon
    Copy link
    Member

    The code doing the sanity check for importlib can be found at

    if level > 0:
    if not isinstance(package, str):
    raise TypeError('__package__ not set to a string')
    elif not package:
    raise ImportError('attempted relative import with no known parent '
    'package')
    .

    @brettcannon brettcannon changed the title relative import without parent relative import without parent succeeds with builtins.__import__ Jun 28, 2019
    @brettcannon
    Copy link
    Member

    If you run with -Xdev/warnings turned on you get an idea of what's happening:

    >>> builtins.__import__('', globals(), locals(), ('foo',), 1)
    <stdin>:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
    <module '__main__' (built-in)>

    The check is being done in resolve_name() in import.c (

    resolve_name(PyThreadState *tstate, PyObject *name, PyObject *globals, int level)
    ). My guess is there's an off-by-one error in the sanity check logic for attempting a relative import beyond the top-level package.

    @benjimin
    Copy link
    Mannequin Author

    benjimin mannequin commented Jun 28, 2019

    I'll look into this further and open a PR initially for the regression tests.

    @brettcannon brettcannon self-assigned this Aug 2, 2019
    @brettcannon
    Copy link
    Member

    New changeset 92420b3 by Brett Cannon (Ben Lewis) in branch 'master':
    bpo-37409: fix relative import with no parent (bpo-14956)
    92420b3

    @brettcannon
    Copy link
    Member

    New changeset 0a6693a by Brett Cannon in branch '3.8':
    [3.8] bpo-37409: fix relative import with no parent (GH-14956) (GH-15913)
    0a6693a

    @miss-islington
    Copy link
    Contributor

    New changeset f3480ad by Miss Islington (bot) in branch '3.7':
    [3.8] bpo-37409: fix relative import with no parent (GH-14956) (GH-15913)
    f3480ad

    @tirkarthi
    Copy link
    Member

    The test added seems to have created an ImportWarning in test_builtin.BuiltinTest.test_import .

    ./python.exe -Wall -m unittest -v test.test_builtin.BuiltinTest.test_import
    test_import (test.test_builtin.BuiltinTest) ... /Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/case.py:202: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
    callable_obj(*args, **kwargs)
    ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.002s

    OK

    @tirkarthi
    Copy link
    Member

    The test is same as below and given that __spec__ an __name__ are passed as None where ImportWarning is raised in Lib/importlib/_bootstrap.py 1074 . can we just use self.assertWarns(ImportWarning) in the test?

    >>> __import__('', {'__package__': None, '__spec__': None, '__name__': '__main__'}, locals={}, fromlist=('foo',), level=1)
    <stdin>:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: attempted relative import with no known parent package

    @brettcannon
    Copy link
    Member

    Thanks for catching the warning and the fix, Karthikeyan!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants