Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

typing.Literal[True] is implicitly converted to typing.Literal[1] #89842

Closed
vanburgerberg mannequin opened this issue Oct 30, 2021 · 9 comments
Closed

typing.Literal[True] is implicitly converted to typing.Literal[1] #89842

vanburgerberg mannequin opened this issue Oct 30, 2021 · 9 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@vanburgerberg
Copy link
Mannequin

vanburgerberg mannequin commented Oct 30, 2021

BPO 45679
Nosy @gvanrossum, @rhettinger, @serhiy-storchaka, @miss-islington, @sobolevn, @Fidget-Spinner, @vanburgerberg
PRs
  • bpo-45679: Do not cache typing.Literal with more than one type arg #29333
  • bpo-45679: Fix caching of multi-value typing.Literal #29334
  • bpo-45679: add tuple tests with lru_cache to test_functools #29339
  • [3.10] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) #29340
  • [3.9] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) #29342
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-11-03.09:45:01.789>
    created_at = <Date 2021-10-30.15:33:26.761>
    labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
    title = 'typing.Literal[True] is implicitly converted to typing.Literal[1]'
    updated_at = <Date 2021-11-05.19:52:54.863>
    user = 'https://github.com/vanburgerberg'

    bugs.python.org fields:

    activity = <Date 2021-11-05.19:52:54.863>
    actor = 'rhettinger'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-11-03.09:45:01.789>
    closer = 'kj'
    components = ['Library (Lib)']
    creation = <Date 2021-10-30.15:33:26.761>
    creator = 'vanburgerberg'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45679
    keywords = ['patch']
    message_count = 9.0
    messages = ['405371', '405375', '405376', '405377', '405394', '405395', '405592', '405598', '405825']
    nosy_count = 7.0
    nosy_names = ['gvanrossum', 'rhettinger', 'serhiy.storchaka', 'miss-islington', 'sobolevn', 'kj', 'vanburgerberg']
    pr_nums = ['29333', '29334', '29339', '29340', '29342']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue45679'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @vanburgerberg
    Copy link
    Mannequin Author

    vanburgerberg mannequin commented Oct 30, 2021

    When you create Literal[1] annotation and then create Literal[True] annotation, in the second case you will actually get Literal[1] instead. This is happening because typing performs caching of the outcome of parameterizing generics and hash(True) is equal to hash(1). I think this behavior is incorrect and may lead to unexpected results.

    Why is this inaccuracy important?
    Consider the following example:

    from typing import Literal
    
    SomeUsefulAlias = Literal[1, "abc"]
    
    def func(arg: Literal[True, "abc"]):
        if arg is not True:
            arg.capitalize()

    If we look at func.__annotations__["arg"], we will see Literal[1, 'abc']. According to the new annotation, we can pass the value 1 to func, and this will lead to an attribute error, as you've already understood. Thus, proper runtime type checking cannot be performed.

    @vanburgerberg vanburgerberg mannequin added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Oct 30, 2021
    @sobolevn
    Copy link
    Member

    This happens to all lru_cache calls, where we deal with container types.

    Example:

    from functools import lru_cache
    
    @lru_cache(typed=True)
    def test(arg):
        return str(arg)
    
    print(test((1, 'a')))  # (1, 'a')
    print(test((True, 'a')))  # (1, 'a')
    

    Moveover, this behavior is not tested in test_functools: https://github.com/python/cpython/blame/aae18a17401dc36917c0f64f971d60ab1a5b477e/Lib/test/test_functools.py#L1482-L1496

    Original issue: https://bugs.python.org/issue13227
    Original commit: cd9fdfd

    @sobolevn
    Copy link
    Member

    I see several ways of solving this:

    1. Make lru_cache(typed=True) to look into iterable internals. This might be slow. And very slow for large iterables. Maybe even infinite for infinite ones
    2. Adapt Literal / possibly other types to be treated differently. Because, most of the time, Literal take a small amount of args

    I am going to send a PR with the second option.

    @serhiy-storchaka
    Copy link
    Member

    I agree. lru_cache(typed=True) itself should not look into iterable internals. It would be not only slow, but a change of semantic.

    The simplest way to solve this issue is to remove caching of __getitem__(). The more sophisticated way is to move caching to lower level and apply it to a function with a var-positional parameter.

    @serhiy-storchaka
    Copy link
    Member

    New changeset 634984d by Serhiy Storchaka in branch 'main':
    bpo-45679: Fix caching of multi-value typing.Literal (GH-29334)
    634984d

    @miss-islington
    Copy link
    Contributor

    New changeset 3997f3c by Miss Islington (bot) in branch '3.10':
    bpo-45679: Fix caching of multi-value typing.Literal (GH-29334)
    3997f3c

    @Fidget-Spinner
    Copy link
    Member

    New changeset bbcf06b by Serhiy Storchaka in branch '3.9':
    [3.9] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) (GH-29342)
    bbcf06b

    @Fidget-Spinner
    Copy link
    Member

    Closing this issue as the bug has been solved.

    @nikita could you please open a new issue for your tests PR and link to that instead? It seems like an enhancement over the current test suite for lru_cache. Thanks!

    @rhettinger
    Copy link
    Contributor

    New changeset 60b5333 by Nikita Sobolev in branch 'main':
    bpo-45679: add tuple tests with lru_cache to test_functools (GH-29339)
    60b5333

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants