Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inspect.isgeneratorfunction fails on hand-created methods #77442

Closed
jdemeyer opened this issue Apr 11, 2018 · 12 comments
Closed

inspect.isgeneratorfunction fails on hand-created methods #77442

jdemeyer opened this issue Apr 11, 2018 · 12 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@jdemeyer
Copy link
Contributor

BPO 33261
Nosy @rhettinger, @terryjreedy, @ncoghlan, @encukou, @jdemeyer
PRs
  • bpo-33261: guard access to __code__ attribute in inspect #6448
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-04-03.14:52:11.710>
    created_at = <Date 2018-04-11.08:59:53.781>
    labels = ['3.7', '3.8', 'type-bug', 'library']
    title = 'inspect.isgeneratorfunction fails on hand-created methods'
    updated_at = <Date 2019-04-03.14:52:11.709>
    user = 'https://github.com/jdemeyer'

    bugs.python.org fields:

    activity = <Date 2019-04-03.14:52:11.709>
    actor = 'petr.viktorin'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-04-03.14:52:11.710>
    closer = 'petr.viktorin'
    components = ['Library (Lib)']
    creation = <Date 2018-04-11.08:59:53.781>
    creator = 'jdemeyer'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 33261
    keywords = ['patch']
    message_count = 12.0
    messages = ['315187', '315272', '315284', '315287', '315288', '315302', '315305', '315312', '315438', '327401', '338968', '339335']
    nosy_count = 5.0
    nosy_names = ['rhettinger', 'terry.reedy', 'ncoghlan', 'petr.viktorin', 'jdemeyer']
    pr_nums = ['6448']
    priority = 'normal'
    resolution = None
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue33261'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    @jdemeyer
    Copy link
    Contributor Author

    The inspect functions isgeneratorfunction, iscoroutinefunction, isasyncgenfunction can fail on methods that do not have a __code__ attribute:

    >>> from types import MethodType
    >>> class Callable:
    ...     def __call__(self, *args):
    ...         return args
    >>> m = MethodType(Callable(), 42)
    >>> m()
    (42,)
    >>> import inspect
    >>> inspect.iscoroutinefunction(m)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib64/python3.6/inspect.py", line 186, in iscoroutinefunction
        object.__code__.co_flags & CO_COROUTINE)
    AttributeError: 'Callable' object has no attribute '__code__'

    This was discovered while working on PEP-575, but it is really an independent issue that should be fixed anyway.

    @jdemeyer jdemeyer added 3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Apr 11, 2018
    @terryjreedy
    Copy link
    Member

    Nick and Raymond, I added both of you as nosy because, among other reasons, you commented on the latest version of PEP-575.

    I agree that there is a bug in current CPython, in that the is* functions should return, not raise, but I am not sure where is it, and hence if the PR is the right fix. I am wondering if, instead, the bug is in m, the object returned by MethodType, or in attribute lookup thereupon.

    MethodType is the type of bound user-defined (Python-coded) functions and ismethod is true for instances thereof. Consider 4 other bound methods, two 'normal'(callable is method of instance class) and two, like Jeroen's, 'odd' (callable not related to int).

    >>> cm = Callable().__call__
    >>> cm.__code__
    <code object __call__ at 0x00000202C1C53780, file "<pyshell#5>", line 2>
    >>> cm.__name__
    '__call__'
    
    >>> cm2 = MethodType(Callable.__call__, Callable())
    >>> cm2.__code__
    <code object __call__ at 0x00000202C1C53780, file "<pyshell#5>", line 2>
    >>> cm2.__name__
    '__call__'
    
    >>> m2 = MethodType(Callable.__call__, 42)
    >>> m2.__code__
    <code object __call__ at 0x00000202C1C53780, file "<pyshell#5>", line 2>
    >>> m2.__name__
    '__call__'
    >>> m2()
    ()
    
    >>> m3 = MethodType(Callable().__call__, 42)
    >>> m3.__code__
    <code object __call__ at 0x00000202C1C53780, file "<pyshell#5>", line 2>
    >>> m3.__name__
    '__call__'
    >>> m3()
    (42,)
    
    >>> m = MethodType(Callable(), 42)
    >>> m.__code__
    ... AttributeError: 'Callable' object has no attribute '__code__'
    >>> m.__name__
    ... AttributeError: 'Callable' object has no attribute '__name__'
    >>> m()
    (42,)

    They all have the same attributes exposed by dir(), which omits both'__name__', promised in the docstring for ismethod*, and '__code__', assumed by the is--- functions under discussion.

    >>> dir(cm) == dir(cm2) == dir(m) == dir(m2) == dir(m3)
    True
    >>> ('__code__' in dir(cm)) or ('__name__' in dir(cm))
    False

    However, accessing those names anyway, as (hidden) attributes, works for all but m. Should doing so also work for m? (If not, the ismethod docstring should not 'guarantee' .__name__.)

    • " Instance method objects provide these attributes:
      __doc__ documentation string
      __name__ name with which this method was defined
      ..."

    @ncoghlan
    Copy link
    Contributor

    The inspect functions throwing an exception when handed a method wrapping a non-function callable instead of returning False is a definite bug.

    The behaviour of MethodType when handed a non-function callable is cryptic, but not actually a bug: the problem is that it expects to be able to delegate accesses to __name__ and __code__ to the underlying callable.

    A docs bug to clarify exactly which accesses are going to get delegated would make sense (and perhaps make MethodType.__dir__ a bit smarter about that delegation), but it's a separate issue from the one Jeroen has reported here.

    @jdemeyer
    Copy link
    Contributor Author

    I am wondering if, instead, the bug is in m, the object returned by MethodType, or in attribute lookup thereupon.

    What would you expect m.__code__ to return then? Methods support arbitrary callables and certainly not all callables have a meaningful __code__.

    @jdemeyer
    Copy link
    Contributor Author

    The only attributes that a method is guaranteed to have are __func__ and __self__ (which are the arguments passed to the MethodType constructor).

    All other attributes are looked up through __func__ by the C version of the following Python code:

    def __getattr__(self, attr):
        return getattr(self.__func__, attr)

    @terryjreedy
    Copy link
    Member

    I would like python_coded_callable.__code__ to be the code object executed when python_coded_callable is called, just as expected by the isxyz author(s). It has to exist somewhere. Methods m and m3 both return 42 when called, and both have the same code object.

    >>> m3.__code__
    <code object __call__ at 0x000001431C797E40, file "F:\Python\a\tem2.py", line 3>
    >>> m3.__func__.__code__
    <code object __call__ at 0x000001431C797E40, file "F:\Python\a\tem2.py", line 3>
    >>> m.__func__.__call__.__code__
    <code object __call__ at 0x000001431C797E40, file "F:\Python\a\tem2.py", line 3>

    The fact that m requires an additional level of indirection is an implementation detail that I don't think necessarily has to involve users.

    @jdemeyer
    Copy link
    Contributor Author

    I would like python_coded_callable.__code__ to be the code object executed when python_coded_callable is called

    First of all, that doesn't answer the question of what to do with non-Python coded callables where there is no __code__ object at all.

    Second, are you *really* sure that you want that? That would mean adding a __code__ attribute to all callable Python classes or adding a __code__ descriptor to "type".

    @ncoghlan
    Copy link
    Contributor

    Terry, if you'd like to continue that discussion, please open a new enhancement request for 3.8+ against the inspect module asking for the affected introspection functions to recursively search for relevant attributes, the same way inspect.signature digs inside different object types to look for signature information.

    However, the fact the inspect module doesn't implement that search for relevant attributes today is *not* a bug - it's a past design decision that could potentially stand to be revisited.

    By contrast, the fact the affected functions throw AttributeError instead of returning False for a missing attribute *is* a bug.

    @jdemeyer
    Copy link
    Contributor Author

    Can we please go back to the original issue? If you think that __code__ should be an alias for __call__.__code__, that is a different issue.

    On #6448 (comment) I posted an alternative solution for the problem. Is either the original PR or the new variant acceptable?

    @jdemeyer
    Copy link
    Contributor Author

    jdemeyer commented Oct 9, 2018

    Can somebody please review PR 6448?

    @encukou
    Copy link
    Member

    encukou commented Mar 27, 2019

    I just reviewed, and I plan to merge it if I don't see any pushback from the others.
    Sorry for the extreme delay.

    @encukou
    Copy link
    Member

    encukou commented Apr 2, 2019

    New changeset fcef60f by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-33261: guard access to __code__ attribute in inspect (GH-6448)
    fcef60f

    @encukou encukou closed this as completed Apr 3, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants