This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Annotated and Optional get_type_hints buggy interaction
Type: behavior Stage: resolved
Components: Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: JelleZijlstra, gvanrossum, kj, med2277, sobolevn
Priority: normal Keywords: patch

Created on 2021-12-29 18:38 by med2277, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 30304 merged sobolevn, 2021-12-30 18:40
Messages (8)
msg409319 - (view) Author: Mehdi2277 (med2277) Date: 2021-12-29 18:38
This is two closely related issues with get_type_hints on an optional annotated member. I'm testing with Annotated/get_type_hints from typing extensions on 3.8 and assuming they're backport equivalent to current behavior.

The first issue is get_type_hints has inconsistent behavior depending on whether annotation comes from a function with a None default or an attribute with a None default.

class Foo:
    def __init__(
        self,
        x: 
            Annotated[Optional[str], "doc string"] = None,
    ):
        ...

class Foo2:
    x: Annotated[Optional[str], "doc string"] = None

get_type_hints(Foo.__init__) 
# {'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}

get_type_hints(Foo2)
# {'x': typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string']}

Attributes with a None default are not wrapped by get_type_hints, but function parameters. Which of the two behaviors is correct I don't know, but I'd expect the two to be equivalent annotations.

The second issue is for function arguments with a None default the optional wrapper happens even if the type inside annotated already has optional. Example,

from typing_extensions import Annotated, get_type_hints

class Foo:
    def __init__(
        self,
        x: 
            Annotated[Optional[str], "doc string"] = None,
    ):
        ...

get_type_hints(Foo.__init__, include_extras=True)
# {'x': typing.Union[typing_extensions.Annotated[typing.Union[str, NoneType], 'doc string'], NoneType]}


For Annotated types I would expect any type rules like wrapping to apply only to the first argument and not the entire annotation. I mainly ran into this for a runtime type introspection library (similar in spirit to pydantic).

As a note include_extras being True or False while it changes type is an issue in either case. With include_extras as False the Annotated goes away, but the type still gets double wrapped as an Optional.
msg409325 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-12-29 19:54
Could you try with 3.10 and the stdlib typing.Annotated please? There might
be changes (in the past a default of None automatically caused an Optional
to be added, but we changed our minds.--
--Guido (mobile)
msg409340 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2021-12-30 00:38
I can verify that this happens on `3.10` and `main` branches:

```
from typing import Annotated, Optional, get_type_hints

class Foo:
    def __init__(self, x: Annotated[Optional[str], "d"] = None): ...

class Foo2:
    x: Annotated[Optional[str], "d"] = None

print(get_type_hints(Foo.__init__, include_extras=False))  # ok
# {'x': typing.Optional[str]}
print(get_type_hints(Foo2, include_extras=False))  # ok
# {'x': typing.Optional[str]}

print(get_type_hints(Foo.__init__, include_extras=True))  # not ok?
# {'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}
print(get_type_hints(Foo2, include_extras=True))  # ok
# {'x': typing.Annotated[typing.Optional[str], 'd']}
```

Notice that 3rd case does not look correct: `{'x': typing.Optional[typing.Annotated[typing.Optional[str], 'd']]}`

In my opinion it should be `{'x': typing.Annotated[typing.Optional[str], 'd']}`

I will look into it! :)
msg409341 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2021-12-30 00:39
And on 3.9 as well.
msg409357 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2021-12-30 10:38
As Guido said, the root cause of this problem is because `None` default automatically adds `Optional` to the resulting type.

Source: https://github.com/python/cpython/blob/8d7644fa64213207b8dc6f555cb8a02bfabeced2/Lib/typing.py#L1854-L1856

So, what happens there:
- correct `value` is passed to `_eval_type`, correct result `typing.Annotated[typing.Optional[str], 'd']` is returned at this point
- then `if name in defaults and defaults[name] is None:` adds extra `Optional` annotation on top of `Annotated`

> in the past a default of None automatically caused an Optional
to be added, but we changed our minds

Guido, are you talking about https://github.com/python/typing/issues/275 ?

Now all type-checkers (AFAIK) support something similar to `--no-implicit-optional` mode.

Having this in mind, I see different solutions to the current problem:
1. Remove `Optional` inference with `None` default. This is going to be a some-what breaking change. The only positive side of this is that we can really simplify our code (mainly because the other solution is to complicate our code even more).
2. Or we can change this place to explicitly check for `Annotated` type and its internal type. This should be the easiest to write and backport. But, it still has some complexity to it. I think that this is a better solution: we don't break existing behavior, change is local and pretty trivial.

Also caused by this:
- https://bugs.python.org/issue42921
- https://bugs.python.org/issue42288
msg409375 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-12-30 17:21
Yes, we changed PEP 484 in https://github.com/python/peps/pull/689.

So get_type_hints() should follow suit.
msg414328 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-03-02 05:30
New changeset 20a1c8ee4bcb1c421b7cca1f3f5d6ad7ce30a9c9 by Nikita Sobolev in branch 'main':
bpo-46195: Do not add `Optional` in `get_type_hints` (GH-30304)
https://github.com/python/cpython/commit/20a1c8ee4bcb1c421b7cca1f3f5d6ad7ce30a9c9
msg414329 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-03-02 05:31
This is now fixed in 3.11, but we'll leave 3.10 and 3.9 alone. Thanks for your bug report!
History
Date User Action Args
2022-04-11 14:59:54adminsetgithub: 90353
2022-03-02 05:31:58JelleZijlstrasetstatus: open -> closed
resolution: fixed
messages: + msg414329

stage: patch review -> resolved
2022-03-02 05:30:27JelleZijlstrasetnosy: + JelleZijlstra
messages: + msg414328
2021-12-30 18:40:26sobolevnsetkeywords: + patch
stage: patch review
pull_requests: + pull_request28517
2021-12-30 17:21:30gvanrossumsetmessages: + msg409375
2021-12-30 10:38:57sobolevnsetmessages: + msg409357
2021-12-30 00:39:09sobolevnsetmessages: + msg409341
versions: + Python 3.9
2021-12-30 00:38:21sobolevnsetversions: + Python 3.10, Python 3.11, - Python 3.8
2021-12-30 00:38:12sobolevnsetnosy: + sobolevn
messages: + msg409340
2021-12-29 19:54:58gvanrossumsetmessages: + msg409325
2021-12-29 19:10:03med2277settype: behavior
2021-12-29 18:38:47med2277create