classification
Title: Inheriting from class that defines __new__ causes inspect.signature to always return (*args, **kwargs) for constructor
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: FFY00, ezyang, gvanrossum, hongweipeng, jonathan.slenders, lukasz.langa, mauvilsa, miss-islington, ralf.gommers, serhiy.storchaka, yselivanov
Priority: normal Keywords: patch

Created on 2020-06-07 03:37 by ezyang, last changed 2021-07-17 08:37 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23336 closed hongweipeng, 2020-11-17 08:46
PR 27177 merged hongweipeng, 2021-07-16 06:31
PR 27189 merged miss-islington, 2021-07-16 13:05
PR 27193 merged lukasz.langa, 2021-07-16 15:25
PR 27209 merged miss-islington, 2021-07-17 08:04
Messages (17)
msg370870 - (view) Author: Edward Yang (ezyang) * Date: 2020-06-07 03:37
Consider the following program:

```
import inspect
from typing import Generic, TypeVar

T = TypeVar('T')

class A(Generic[T]):
    def __init__(self) -> None:
        pass

print(inspect.signature(A))
```

I expect inspect.signature to return () as the signature of the constructor of this function. However, I get this:

```
$ python3 foo.py
(*args, **kwds)
```

Although it is true that one cannot generally rely on inspect.signature to always give the most accurate signature (because there may always be decorator or metaclass shenanigans getting in the way), in this particular case it seems especially undesirable because Python type annotations are supposed to be erased at runtime, and yet here inheriting from Generic (simply to add type annotations) causes a very clear change in runtime behavior.
msg370888 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-07 11:11
It is not special for Generic, but happens with every type implementing __new__.

class A:
    def __new__(cls, a=1, *args, **kwargs):
        return object.__new__(cls)

class B(A):
    def __init__(self, b):
        pass

import inspect
print(inspect.signature(B))

The above example prints "(a=1, *args, **kwargs)" instead of "(b)".
msg370958 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-06-08 04:58
So does that make this "not a bug"? Or is there something to document? For technical reasons we can't just add a __init__ method to Generic, and I doubt that it's feasible to change inspect.signature().
msg370970 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-08 09:25
I think that inspect.signature() could be made more smart. It should take into account signatures of both __new__ and __init__ and return the most specific compatible signature.
msg371023 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-06-08 17:16
Changing the topic to not point fingers at Generic.
msg387139 - (view) Author: Jonathan Slenders (jonathan.slenders) * Date: 2021-02-17 10:49
The following patch to inspect.py solves the issue that inspect.signature() returns the wrong signature on classes that inherit from Generic. Not 100% sure though if this implementation is the cleanest way possible. I've been looking into attaching a __wrapped__ to Generic as well, without success. I'm not very familiar with the inspect code.

To me, this fix is pretty important. ptpython, a Python REPL, has the ability to show the function signature of what the user is currently typing, and with codebases that have lots of generics, there's nothing really useful we can show.


 $ diff inspect.old.py  inspect.py  -p
*** inspect.old.py      2021-02-17 11:35:50.787234264 +0100
--- inspect.py  2021-02-17 11:35:10.131407202 +0100
*************** import sys
*** 44,49 ****
--- 44,50 ----
  import tokenize
  import token
  import types
+ import typing
  import warnings
  import functools
  import builtins
*************** def _signature_get_user_defined_method(c
*** 1715,1720 ****
--- 1716,1725 ----
      except AttributeError:
          return
      else:
+         if meth in (typing.Generic.__new__, typing.Protocol.__new__):
+             # Exclude methods from the typing module.
+             return
+
          if not isinstance(meth, _NonUserDefinedCallables):
              # Once '__signature__' will be added to 'C'-level
              # callables, this check won't be necessary


***

For those interested, the following monkey-patch has the same effect:

def monkey_patch_typing() -> None:
    import inspect, typing
    def _signature_get_user_defined_method(cls, method_name):
        try:
            meth = getattr(cls, method_name)
        except AttributeError:
            return
        else:
            if meth in (typing.Generic.__new__, typing.Protocol.__new__):
                # Exclude methods from the typing module.
                return

            if not isinstance(meth, inspect._NonUserDefinedCallables):
                # Once '__signature__' will be added to 'C'-level
                # callables, this check won't be necessary
                return meth

    inspect._signature_get_user_defined_method = _signature_get_user_defined_method

monkey_patch_typing()
msg387184 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-02-17 19:37
I doubt that solution is correct, given that we already established that the problem is *not* specific to Generic.
msg397248 - (view) Author: Mauricio Villegas (mauvilsa) Date: 2021-07-10 12:45
I think this is affecting me or it is a new similar issue. And it is not only for python 3.9, but also from 3.6 up. I am working on making code configurable based on signatures (see https://jsonargparse.readthedocs.io/en/stable/#classes-methods-and-functions). Now we need this to work for datetime.timedelta which defines parameters in __new__ instead of __init__. The following happens:

>>> from datetime import timedelta
>>> import inspect
>>> inspect.signature(timedelta.__new__)
<Signature (*args, **kwargs)>
>>> inspect.signature(timedelta.__init__)
<Signature (self, /, *args, **kwargs)>
>>> inspect.signature(timedelta)
...
ValueError: no signature found for builtin type <class 'datetime.timedelta'>

I am expecting to get parameters for days, seconds, microseconds, milliseconds, minutes, hours and weeks, see https://github.com/python/cpython/blob/bfe544d2f2c2e7a7c03a764bed3276a1e27a0f5c/Lib/datetime.py#L461-L462.

Hopefully this gives some insight into what should be done. Independent from the fix, I would like to know if currently there is any way I can get the signature for __new__ that I could use until there is a proper fix for it.
msg397470 - (view) Author: Mauricio Villegas (mauvilsa) Date: 2021-07-14 06:03
I created another issue since the problem appears to be a bit different: https://bugs.python.org/issue44618
msg397606 - (view) Author: hongweipeng (hongweipeng) * Date: 2021-07-16 09:25
>>> from datetime import timedelta as a
>>> from _datetime import timedelta as b
>>> a is b
True
>>>

`timedelta` is a C-level class, so inspect.signature(timedelta) is the same with inspect.signature(int).

But `signature` allow C-level function such as `inspect.signature(len)`, I think one way to improve is to use the C-level function when the python-level function cannot be found.
msg397617 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 13:04
New changeset 6aab5f9bf303a8e4cd8377fabcdcb499e0541f9a by Weipeng Hong in branch 'main':
bpo-40897:Give priority to using the current class constructor in `inspect.signature` (#27177)
https://github.com/python/cpython/commit/6aab5f9bf303a8e4cd8377fabcdcb499e0541f9a
msg397619 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 13:14
We won't be backporting this fix to 3.9 due to larger changes between versions.
msg397623 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 13:26
New changeset 948e39a866ccf33b4e30668c3f88a95a65966159 by Miss Islington (bot) in branch '3.10':
bpo-40897:Give priority to using the current class constructor in `inspect.signature` (GH-27177) (#27189)
https://github.com/python/cpython/commit/948e39a866ccf33b4e30668c3f88a95a65966159
msg397633 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-16 15:16
On second thought it's a bummer not to fix this in 3.9.x that will still be the only stable version until October. I'll refactor the relevant part of inspect.py in 3.9 to make the backport applicable.
msg397705 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-17 08:04
New changeset ed2db9b5940ccbc0bc72d01bf62d8b3095ccef21 by Łukasz Langa in branch '3.9':
bpo-40897: Partially backport GH-22583's refactor of inspect.py to allow bugfix backports (#27193)
https://github.com/python/cpython/commit/ed2db9b5940ccbc0bc72d01bf62d8b3095ccef21
msg397708 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-17 08:36
New changeset df7c62980d15acd3125dfbd81546dad359f7add7 by Miss Islington (bot) in branch '3.9':
bpo-40897:Give priority to using the current class constructor in `inspect.signature` (GH-27177) (GH-27209)
https://github.com/python/cpython/commit/df7c62980d15acd3125dfbd81546dad359f7add7
msg397709 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-17 08:37
Thanks! ✨ 🍰 ✨
History
Date User Action Args
2021-07-17 08:37:04lukasz.langasetstatus: open -> closed
resolution: fixed
messages: + msg397709

stage: patch review -> resolved
2021-07-17 08:36:39lukasz.langasetmessages: + msg397708
2021-07-17 08:04:42miss-islingtonsetpull_requests: + pull_request25747
2021-07-17 08:04:22lukasz.langasetmessages: + msg397705
2021-07-16 15:25:14lukasz.langasetpull_requests: + pull_request25728
2021-07-16 15:16:11lukasz.langasetmessages: + msg397633
2021-07-16 13:26:05lukasz.langasetmessages: + msg397623
2021-07-16 13:14:54lukasz.langasetmessages: + msg397619
versions: + Python 3.10, Python 3.11
2021-07-16 13:05:19miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25725
2021-07-16 13:04:58lukasz.langasetnosy: + lukasz.langa
messages: + msg397617
2021-07-16 09:25:23hongweipengsetmessages: + msg397606
2021-07-16 06:31:27hongweipengsetpull_requests: + pull_request25714
2021-07-14 06:03:29mauvilsasetmessages: + msg397470
2021-07-10 12:45:29mauvilsasetnosy: + mauvilsa
messages: + msg397248
2021-02-17 19:37:17gvanrossumsetmessages: + msg387184
2021-02-17 10:49:22jonathan.slenderssetnosy: + jonathan.slenders
messages: + msg387139
2020-11-17 08:46:55hongweipengsetkeywords: + patch
nosy: + hongweipeng

pull_requests: + pull_request22225
stage: patch review
2020-06-08 17:16:39gvanrossumsetnosy: - levkivskyi

messages: + msg371023
title: Inheriting from Generic causes inspect.signature to always return (*args, **kwargs) for constructor (and all subclasses) -> Inheriting from class that defines __new__ causes inspect.signature to always return (*args, **kwargs) for constructor
2020-06-08 17:12:47FFY00setnosy: + FFY00
2020-06-08 09:25:51serhiy.storchakasetmessages: + msg370970
2020-06-08 04:58:18gvanrossumsetmessages: + msg370958
2020-06-07 18:59:17ralf.gommerssetnosy: + ralf.gommers
2020-06-07 11:11:40serhiy.storchakasetnosy: + serhiy.storchaka, yselivanov
messages: + msg370888
2020-06-07 04:17:26xtreaksetnosy: + gvanrossum, levkivskyi
2020-06-07 03:37:38ezyangcreate