This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Improve help() by making typing.overload() information accessible at runtime
Type: Stage: patch review
Components: Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: AlexWaygood, DiddiLeija, JelleZijlstra, Spencer Brown, gvanrossum, kj, rhettinger, ronaldoussoren, sobolevn
Priority: normal Keywords: patch

Created on 2021-09-04 18:03 by rhettinger, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 31716 open JelleZijlstra, 2022-03-07 02:16
Messages (14)
msg401052 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-09-04 18:03
Python's help() function does not display overloaded function signatures.

For example, this code:

    from typing import Union

    class Smudge(str):

        @overload
        def __getitem__(self, index: int) -> str:
            ...

        @overload
        def __getitem__(self, index: slice) -> 'Smudge':
            ...

        def __getitem__(self, index: Union[int, slice]) -> Union[str, 'Smudge']:
            'Return a smudged character or characters.' 
            if isinstance(index, slice):
                start, stop, step = index.indices(len(self))
                values = [self[i] for i in range(start, stop, step)]
                return Smudge(''.join(values))
            c = super().__getitem__(index)
            return chr(ord(c) ^ 1)


Currently gives this help:

    __getitem__(self, index: Union[int, slice]) -> Union[str, ForwardRef('Smudge')]
        Return a smudged character or characters.


What is desired is:

    __getitem__(self, index: int) -> str
    __getitem__(self, index: slice) -> ForwardRef('Smudge')
        Return a smudged character or characters.

The overload() decorator is sufficient for informing a static type checker but insufficient for informing a user or editing tool.
msg401074 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2021-09-05 07:45
I agree that this would be nice to have, but wonder how help() could access that information.  The two @overload definitions will be overwritten by the non-overload one at runtime, and hence will ever been seen by help().
msg401080 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2021-09-05 11:39
There is a similar issue with `functools.singledispatch`

```
>>> from functools import singledispatch
>>> @singledispatch
... def flip(x: str) -> int:
... 	"""Signature when given a string"""
... 	return int(x)
... 
>>> @flip.register
... def _(x: int) -> str:
... 	"""Signature when given an int"""
... 	return str(x)
... 
>>> flip(5)
'5'
>>> flip('5')
5
>>> help(flip)
Help on function flip in module __main__:
flip(x: str) -> int
    Signature when given a string
```
msg401094 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-09-05 16:14
> The two @overload definitions will be overwritten by 
> the non-overload one at runtime, and hence will ever 
> been seen by help().

We can fix this by adding an __overloads__ attribute.  The overload decorator can accumulate the chain in an external namespace and function creation can move that accumulation into the new attribute.

----- Proof of concept -----

from typing import Union, _overload_dummy

def create_function(func):
    namespace = func.__globals__
    key = f'__overload__{func.__qualname__}__'
    func.__overloads__ = namespace.pop(key, [])
    return func

def overload(func):
    namespace = func.__globals__
    key = f'__overload__{func.__qualname__}__'
    namespace[key] = func.__overloads__ + [func.__annotations__]
    return _overload_dummy

class Smudge(str):

    @overload
    @create_function
    def __getitem__(self, index: int) -> str:
        ...

    @overload
    @create_function
    def __getitem__(self, index: slice) -> 'Smudge':
        ...

    @create_function
    def __getitem__(self, index: Union[int, slice]) -> Union[str, 'Smudge']:
        'Return a smudged character or characters.' 
        if isinstance(index, slice):
            start, stop, step = index.indices(len(self))
            values = [self[i] for i in range(start, stop, step)]
            return Smudge(''.join(values))
        c = super().__getitem__(index)
        return chr(ord(c) ^ 1)

    @create_function
    def other_method(self, x:str) -> tuple:
        pass

print(f'{Smudge.__getitem__.__annotations__=}')
print(f'{Smudge.__getitem__.__overloads__=}')
print(f'{Smudge.other_method.__annotations__=}') 
print(f'{Smudge.other_method.__overloads__=}')
msg401097 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-09-05 16:53
Note, I'm not proposing a create_function() decorator.  That is just for the proof of concept.  The actual logic would go into normal function creation, the same place that __annotations__ gets added.

Also, there may be a better place than func.__globals__ to accumulate the overloads.  For the proof-of-concept, it was just the easiest way to go.
msg413677 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-02-21 20:27
I made a similar suggestion in issue46821 (thanks Alex for pointing me to this older issue):

Currently, the implementation of @overload (https://github.com/python/cpython/blob/59585d6b2ea50d7bc3a9b336da5bde61367f527c/Lib/typing.py#L2211) simply returns a dummy function and throws away the decorated function. This makes it virtually impossible for type checkers using the runtime function object to find overloads specified at runtime.

In pyanalyze, I worked around this by providing a custom @overload decorator, working something like this:

_overloads: dict[str, list[Callable]] = {}

def _get_key(func: Callable) -> str:
    return f"{func.__module__}.{func.__qualname__}"

def overload(func):
    key = _get_key(func)
    _overloads.setdefault(key, []).append(func)
    return _overload_dummy

def get_overloads_for(func):
    key = _get_key(func)
    return _overloads.get(key, [])

A full implementation will need more error handling.

I'd like to add something like this to typing.py so that other tools can also use this information.

---

With my suggested solution, help() would need to call typing.get_overloads_for() to get any overloads for the function. Unlike Raymond's suggestion, we would not need to change the function creation machinery.
msg413678 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-02-21 20:53
Sounds good to me. (I don’t care what happens at runtime but I want to
support the folks who do.)--
--Guido (mobile)
msg413684 - (view) Author: Spencer Brown (Spencer Brown) * Date: 2022-02-21 22:01
I'm not sure a get_overloads() function potentially called after the fact would fully work - there's the tricky case of nested functions, where the overload list would need to be somehow cleared to ensure every instantiation doesn't endlessly append to the same list. It's probably also desirable to weakref it (or make it an attribute) so they can be decrefed if the function isn't being used.
msg413685 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-02-21 23:03
I'm OK with not fully supporting overloads created in nested functions; that's a pretty marginal use case. But it's true that my proposed implementation would create a memory leak if someone does do that. I don't immediately see a way to fix that with weakrefs. Maybe we need to put something in the defining namespace, as Raymond suggested.
msg413695 - (view) Author: Spencer Brown (Spencer Brown) * Date: 2022-02-22 06:51
Had a potential thought. Since the only situation we care about is overload being used on function definitions in lexical order, valid calls are only that on definitions with ascending co_firstlineno counts. Expanding on Jelle's solution, the overload() decorator could compare the current function's line number to the first in the list, and if it's <= clear out the list (we're re-defining). Then repeated re-definitions wouldn't duplicate overloads. 

The other change I'd suggest is to make get_overloads_for() first check __overloads__, then only if not present pop from the _overloads dict and assign to that attribute. That way if code calls get_overloads_for() at least once, the function will be referring to the actual overloads created at the same time. It'd also get garbage collected then when the function dies. It also means you could manually assign to add overloads to any callable.
msg413751 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-02-22 22:13
I'd dearly like better introspection tools for functions decorated with @overload, but I'd rather have a solution where:

- inspect.signature doesn't have to import typing. That doesn't feel worth it for users who aren't using typing.overload, but inspect.signature would have to import typing whether or not @overload was being used, in order to *check* whether @overload was being used.
- The solution could be reused by, and generalised to, other kinds of functions that have multiple signatures.

If we create an __overloads__ dunder that stored the signatures of multi-signature functions, as Raymond suggests, inspect.signature could check that dunder to examine whether the function is a multi-dispatch signature, and change its representation of the function accordingly. This kind of solution could be easily reused by other parts of the stdlib, like @functools.singledispatch, and by third-party packages such as plum-dispatch, multipledispatch, and Nikita's dry-python/classes library.

So, while it would undoubtedly be more complex to implement, I much prefer Raymond's suggested solution.
msg414561 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-03-05 03:45
We could make my proposed overload registry more reusable by putting it in a different module, probably functools. (Another candidate is inspect, but inspect.py imports functools.py, so that would make it difficult to use the registry for functools.singledispatch.)

We could then bill it as a "variant registry", with an API like this:

def register_variant(key: str, variant: Callable) -> None: ...
def get_variants(key: str) -> list[Callable]: ...
def get_key_for_callable(callable: Callable) -> str | None: ...

@overload could then call register_variant() to register each overload, and code that wants a list of overloads (pydoc, inspect.signature, runtime type checkers) could call get_variants().

get_key_for_callable() essentially does f"{callable.__qualname__}.{callable.__name__}", but returns None for objects it can't handle. It will also support at least classmethods and staticmethods.

I will prepare a PR implementing this idea.
msg414566 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-05 08:23
The latest plan sounds good to me. I have some Thoughts on the proposed API, but it will be easier to express those as part of a PR review. Looking forward to seeing the PR!
msg416226 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-28 22:36
Looks like there may be a new plan where we solve a smaller problem (overloads) in the context of typing only.
History
Date User Action Args
2022-04-11 14:59:49adminsetgithub: 89263
2022-03-28 22:36:21gvanrossumsetmessages: + msg416226
2022-03-28 17:54:17gvanrossumsetnosy: + gvanrossum
2022-03-07 02:16:38JelleZijlstrasetkeywords: + patch
stage: patch review
pull_requests: + pull_request29832
2022-03-05 08:23:30AlexWaygoodsetmessages: + msg414566
2022-03-05 03:45:06JelleZijlstrasetmessages: + msg414561
2022-02-22 22:13:23AlexWaygoodsetmessages: + msg413751
2022-02-22 16:44:59gvanrossumsetnosy: - gvanrossum
2022-02-22 13:47:50AlexWaygoodsetnosy: + sobolevn
2022-02-22 06:51:20Spencer Brownsetmessages: + msg413695
2022-02-21 23:03:43JelleZijlstrasetmessages: + msg413685
2022-02-21 22:02:00Spencer Brownsetnosy: + Spencer Brown
messages: + msg413684
2022-02-21 20:53:06gvanrossumsetmessages: + msg413678
2022-02-21 20:27:20JelleZijlstrasetnosy: + JelleZijlstra
messages: + msg413677
2022-02-21 20:23:36JelleZijlstralinkissue46821 superseder
2021-09-05 16:53:27rhettingersetmessages: + msg401097
2021-09-05 16:16:56rhettingersettitle: Teach help about typing.overload() -> Improve help() by making typing.overload() information accessible at runtime
versions: + Python 3.11
2021-09-05 16:14:44rhettingersetmessages: + msg401094
2021-09-05 12:39:03DiddiLeijasetnosy: + DiddiLeija
2021-09-05 11:39:08AlexWaygoodsetnosy: + AlexWaygood
messages: + msg401080
2021-09-05 08:29:08kjsetnosy: + gvanrossum, kj
2021-09-05 07:45:09ronaldoussorensetnosy: + ronaldoussoren
messages: + msg401074
2021-09-04 18:03:28rhettingercreate