Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add __parameters__ and __getitem__ in TypeVar and ParamSpec #88959

Open
serhiy-storchaka opened this issue Jul 31, 2021 · 9 comments
Open

Add __parameters__ and __getitem__ in TypeVar and ParamSpec #88959

serhiy-storchaka opened this issue Jul 31, 2021 · 9 comments
Labels
3.11 only security fixes stdlib Python modules in the Lib dir topic-typing type-feature A feature request or enhancement

Comments

@serhiy-storchaka
Copy link
Member

BPO 44796
Nosy @gvanrossum, @serhiy-storchaka, @JelleZijlstra, @sobolevn, @Fidget-Spinner, @AlexWaygood
PRs
  • gh-88959: Add __parameters__ and __getitem__ in TypeVar and ParamSpec #27511
  • bpo-44796: Unify TypeVar and ParamSpec substitution #31143
  • Dependencies
  • bpo-44794: Merge tests for typing.Callable and collection.abc.Callable
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2021-07-31.13:31:29.771>
    labels = ['type-feature', 'library', '3.11']
    title = 'Add __parameters__ and __getitem__ in TypeVar and ParamSpec'
    updated_at = <Date 2022-03-12.13:54:12.028>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2022-03-12.13:54:12.028>
    actor = 'serhiy.storchaka'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2021-07-31.13:31:29.771>
    creator = 'serhiy.storchaka'
    dependencies = ['44794']
    files = []
    hgrepos = []
    issue_num = 44796
    keywords = ['patch']
    message_count = 9.0
    messages = ['398640', '411555', '412464', '412483', '412484', '412575', '412704', '414895', '414992']
    nosy_count = 6.0
    nosy_names = ['gvanrossum', 'serhiy.storchaka', 'JelleZijlstra', 'sobolevn', 'kj', 'AlexWaygood']
    pr_nums = ['27511', '31143']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue44796'
    versions = ['Python 3.11']

    @serhiy-storchaka
    Copy link
    Member Author

    Adding __parameters__ and __getitem__ in TypeVar and ParamSpec allows to generalize and simplify the code (especially the C code) and allows to add more runtime checks. It may open ways for further simplification.

    Unfortunately it is not compatible with bpo-44098, so the latter changes should be reverted.

    @serhiy-storchaka serhiy-storchaka added 3.11 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Jul 31, 2021
    @JelleZijlstra
    Copy link
    Member

    The linked PR makes T = TypeVar("T"); T[int] work. That's not currently meaningful syntax to the type checker, but we may want it in the future for higher-kinded types. I would rather not make this change without a clear static typing use case in mind.

    @serhiy-storchaka
    Copy link
    Member Author

    It does not have a use case of T[int] in mind. It is an implementation detail which simplifies the code, makes runtime checks more strict and makes types.GenericAlias more similar to typing._GenericAlias.

    For example, currently:

    >>> A = List[T]
    >>> B = list[T]
    >>> A[None]
    typing.List[NoneType]
    >>> B[None]
    list[None]
    >>> A['X']
    typing.List[ForwardRef('X')]
    >>> B['X']
    list['X']

    With the proposed change:

    >>> B[None]
    list[NoneType]
    >>> B['X']
    list[ForwardRef('X')]

    Meanless expressions like A[42], A[Final], A[Final[int]], A[ClassVar], A[ClassVar[int]], etc which are silently accepted will now be errors.

    The code simplification (especially for the C code) is my primary goal, and the rest is a nice side effect. It is possible to add more strict runtime checks and conversions without making TypeVar and ParamSpec subscriptable, but the code will not be so simple.

    @gvanrossum
    Copy link
    Member

    I don't think that changing list[None] to list[NoneType] in the output is an improvement. I do appreciate the reduction in C code though!

    @JelleZijlstra
    Copy link
    Member

    I'd also be wary about making changes that introduce additional runtime checks. As we've seen with the PEP-612 and 646 implementations, those can impede future iteration on typing.

    @serhiy-storchaka
    Copy link
    Member Author

    I have created an alternative PR 31143 which adds special method __typing_subst__ in TypeVar and ParamSpec instead of __parameters__ and __getitem__. It has almost the same effect except making TypeVar and ParamSpec and subscriptable. The Python code in typing.py and _collections_abc.py is simplified, but the C code is little changed.

    But on other hand, the reasons for PR 27511:

    1. We already implemented __getitem__ for a Concatenate alias, although it is never documented. It is not even necessary, it is a pure implementation detail which makes the code for nested variables substitution simpler. Adding __parameters__ and __getitem__ in TypeVar and ParamSpec would have the same effect on simplification of the code.

    2. There is a use case for subscription of TypeVar. For example:

    if version < 2:
       RetType = T
    elif version < 3:
       RetType = Optional[T]
    else:
       RetType = Optional[T] | Literal[Unknown]
    
    def get_int(precision: int) -> RetType[int]: ...
    def get_str(encoding: str) -> RetType[str]: ...
    def get_any(converter: Callable[[str], T]) -> RetType[T]: ...

    @gvanrossum
    Copy link
    Member

    Super subtle stuff. Tonight I do not have time to really dive into this, but I think Serhiy is on to something. KJ or Jelle, do you have the guts to dive in here?

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset b6a5d85 by Serhiy Storchaka in branch 'main':
    bpo-44796: Unify TypeVar and ParamSpec substitution (GH-31143)
    b6a5d85

    @serhiy-storchaka
    Copy link
    Member Author

    PR 31143 simplified some code without changing the common API:

    Objects/genericaliasobject.c | 46 ++++++++++++++--------------------------------
    1 file changed, 14 insertions(+), 32 deletions(-)
    Lib/_collections_abc.py | 63 +++++++++------------------------------------------------------
    1 file changed, 9 insertions(+), 54 deletions(-)

    PR 27511 will simplify it even more:

    Objects/genericaliasobject.c | 60 ++++++++++++++++++------------------------------------------
    1 file changed, 18 insertions(+), 42 deletions(-)

    (Lib/typing.py will be simplified too, even if this does not reduce the number of lines).

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 only security fixes stdlib Python modules in the Lib dir topic-typing type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants