This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [DOC] typing.get_type_hints() raises for type aliases with forward references
Type: behavior Stage: resolved
Components: Documentation, Library (Lib) Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, gvanrossum, kj, lukasz.langa, mhils, miss-islington
Priority: normal Keywords: patch

Created on 2021-08-16 17:15 by mhils, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27859 merged mhils, 2021-08-20 12:16
PR 27860 merged miss-islington, 2021-08-20 14:37
Messages (13)
msg399660 - (view) Author: Maximilian Hils (mhils) * Date: 2021-08-16 17:15
Someone reported this rather interesting issue where typing.get_type_hints crashes on type aliases with forward references. The original report is at https://github.com/mitmproxy/pdoc/issues/290. Here's an extended minimal example:

foo.py:
```
import typing

FooList1: typing.TypeAlias = list["Foo"]
FooList2: typing.TypeAlias = typing.List["Foo"]

class Foo:
    pass

```

bar.py:
```
import typing
import foo

def func1(x: foo.FooList1):
    pass

def func2(x: foo.FooList2):
    pass

print(typing.get_type_hints(func1))  # {'x': list['Foo']}
print(typing.get_type_hints(func2))  # NameError: name 'Foo' is not defined.

```


Observations:

1. func1 doesn't crash, but also doesn't resolve the forward reference. I am not sure if this expected behavior.
   If it isn't, this should eventually run in the same problem as func2.
2. func2 crashes because "Foo" is evaluated in the context of bar.py (where class Foo is unknown) and not in the context of foo.py. ForwardRef._evaluate would
   somehow need to know in which context it was defined. #41249 (TypedDict inheritance doesn't work with get_type_hints) introduced 
   ForwardRef.__forward_module__, which would be a logical place for that information. I'm not sure if it is a good idea
   to use __forward_module__ more widely.
3. This may end up as quite a bit of complexity for an edge case, I'm fine if it is considered wontfix. 
   The reason I'm bringing it up is that PEP 613 (Explicit Type Aliases) decidedly allows forward references in type aliases.


For the record, PEP 563 (postponed evaluations) does not change the outcome here. However, postponed evaluations often make it possible to avoid the forward references by declaring the aliases last.
msg399662 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-08-16 17:45
> 1. func1 doesn't crash, but also doesn't resolve the forward reference. I am not sure if this expected behavior.

Good observation! That's indeed the current behavior. The reason is a little subtle - list[...] is using a special builtin type (https://docs.python.org/3/library/stdtypes.html#types-genericalias), while typing.List is the one from typing. The typing version wraps all strings in ForwardRef(), while the builtin version shouldn't import from typing, so it doesn't have that luxury. get_type_hints could probably support the builtin version, though I'd imagine we'd need some rework.

Re: 2. You're right there too. Storing the defined module in __forward_module__ is an interesting proposal.

For your specific use case (where the user is using Python 3.6), you could pass in globalns and localns to get_type_hints as a temporary workaround. Off the top of my head:
get_type_hints(func2, globalns=foo.__dict__) might work. Would that work for your library?
msg399664 - (view) Author: Maximilian Hils (mhils) * Date: 2021-08-16 18:06
> For your specific use case (where the user is using Python 3.6), you could pass in globalns and localns to get_type_hints as a temporary workaround. Off the top of my head:
get_type_hints(func2, globalns=foo.__dict__) might work. Would that work for your library?

I guess the hard part is knowing that the type annotation comes from `foo`. In the example here we can of course hardcode it, but that doesn't work in the general case or for pdoc, the documentation generator I'm working on (https://pdoc.dev). I have experimented quite a bit with walking the AST to figure out where type aliases are imported from to then re-executing ForwardRefs with that globalns. Long story short, trying to reverse-engineer __forward_module__ quickly becomes a tangled hot mess where you need to adjust for import aliases, reimports, and so on.
msg399672 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-08-16 19:21
Is GH-27017 at all relevant here?
msg399675 - (view) Author: Maximilian Hils (mhils) * Date: 2021-08-16 20:12
@Guido van Rossum: Yes, GH-27017 is the same as #41249 in the initial post. There are also some cases where we don't even have a ForwardRef though:

foo.py:
```
import typing

FooType: typing.TypeAlias = "Foo"

class Foo:
    pass

```

bar.py:
```
import typing
import foo

def func3(x: foo.FooType):
    pass

print(typing.get_type_hints(func3))  # NameError: name 'Foo' is not defined.

```

In this example, `FooType` is just a regular str (with unknown origin). That's a case where get_type_hints just needs to give up?
msg399677 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-08-16 20:22
I think it's fine for get_type_hints() to give up in some cases. It's not fine if it were to look in the wrong namespace (e.g. if the caller of get_type_hints() were to happen to have an unrelated class named "Foo" in its namespace, that should *not* be found if the type alias was defined in a different file).
msg399746 - (view) Author: Maximilian Hils (mhils) * Date: 2021-08-17 12:24
Thank you Ken and Guido for the super quick feedback!

> It's not fine if it were to look in the wrong namespace

That's a really good point. Here's another terrible example:


foo.py:
```
import typing

FooData: typing.TypeAlias = "Data"

class Data:
    pass

```

bar.py:
```
import typing
import foo

BarData: typing.TypeAlias = "Data"

def func1(x: foo.FooData, y: BarData):
    pass

class Data:
    pass

print(typing.get_type_hints(func1))  # incorrectly uses bar.Data twice.

```


I don't see how we could distinguish FooData from BarData without static analysis. Changing get_type_hints to not pick the unrelated class with the same name would essentially mean not using func.__globals__, which breaks almost all ForwardRef evaluation. This doesn't seem viable.

Potential alternatives:

1. Accept the current state: get_type_hints does not work well with type aliases that use forward references.
2. Fix it at least for cases where a ForwardRef is constructed immediately (typing.List["Foo"], but not "Foo" or list["Foo"]), similar to how it's been done for #41249. I have made a very basic proof-of-concept at https://github.com/mhils/cpython/commit/4adbcf088d2857166b579f7dd2954ff9981fc7db, but it's a mess (see commit comments).
3. Deprecate/discourage use of forward references in type aliases and/or change the syntax for [forward references in] type aliases so that their origin can be tracked.

In summary, it seems like there are no really good solutions here. I'm fine if this ends up as a pragmatic wontfix, maybe with a comment added somewhere in the docs or in PEP 613.
msg399774 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-08-17 17:32
Maximilian, I think your last analysis is spot on. The problem is specific to type aliases and how at runtime they do not carry information about scope with them. The current type alias syntax (both the original form and the form using "x: TypeAlias = ...") was designed for static analysis only.

I suppose we could fix cases like

A = List["Foo"]

but we can't fix for example

A: TypeAlias = "Foo"

since at runtime this just ends up creating a variable A whose value is the string "Foo", and if you import and use that in another module, all you have is the value "Foo".

I think we have to accept this as a limitation of type aliases when combined with runtime access to types (it's not just get_type_hints(), but any mechanism that introspects types at runtime).

I'm reluctant to fix the List["Foo"] case, if only because that case is being phased out in favor of list["Foo"] anyway, and because it depends on typing.List being special -- we can't do the same kind of fixup for user-defined classes (e.g. C["Foo"]).
msg399829 - (view) Author: Maximilian Hils (mhils) * Date: 2021-08-18 09:49
Thanks Guido! I agree on not pursuing the List["Foo"] case for the reasons you mentioned.

Let me know if you think it'd be useful to mention this limitation briefly in one of the relevant PEPs or somewhere else. I'm not sure if it meets the bar for notability, you probably have a better gut feeling for this.

Other than that I'd propose we close this here as wontfix. Thank you again for the very useful feedback! :)
msg399850 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-08-18 15:22
We could mention this in the docs for one or more of the following:

- type aliases (old or new syntax)
- forward references
- get_type_hints()

Ken Jin, can you guide Maximilian towards a successful doc update PR?
msg399852 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-08-18 15:52
> Ken Jin, can you guide Maximilian towards a successful doc update PR?

It seems that Maximilian has already made some contributions to CPython, so I'm sure he's somewhat familiar with our workflow :). Nonetheless, @Maximilian if you need any help, please do ping me, I'll be happy to.

We could add a ..note: here https://docs.python.org/3/library/typing.html#typing.get_type_hints. The document is at https://github.com/python/cpython/blob/main/Doc/library/typing.rst. The Python docs style guide is at https://devguide.python.org/documenting/#style-guide.

Thanks Max for your interest in improving CPython!

PS: I've changed the affected versions to the ones we still bugfix (3.9 and up).
msg399974 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-08-20 14:36
New changeset 16b9be4861e007ad483611ba0479feb2b90ea783 by Maximilian Hils in branch 'main':
bpo-44926: `get_type_hints`: Add note about type aliases with forward refs (#27859)
https://github.com/python/cpython/commit/16b9be4861e007ad483611ba0479feb2b90ea783
msg400086 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-08-22 18:30
New changeset 9ffe582a018a2babd14e874ae2b169370633fe33 by Miss Islington (bot) in branch '3.10':
bpo-44926: `get_type_hints`: Add note about type aliases with forward refs (GH-27859) (GH-27860)
https://github.com/python/cpython/commit/9ffe582a018a2babd14e874ae2b169370633fe33
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 89089
2021-10-18 07:55:48iritkatrielsetstatus: open -> closed
assignee: docs@python
type: crash -> behavior
title: typing.get_type_hints() raises for type aliases with forward references -> [DOC] typing.get_type_hints() raises for type aliases with forward references
components: + Documentation

nosy: + docs@python
versions: - Python 3.9
resolution: fixed
stage: patch review -> resolved
2021-08-22 18:30:10lukasz.langasetnosy: + lukasz.langa
messages: + msg400086
2021-08-20 14:37:01miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26318
2021-08-20 14:36:59gvanrossumsetmessages: + msg399974
2021-08-20 12:16:28mhilssetkeywords: + patch
stage: patch review
pull_requests: + pull_request26317
2021-08-18 15:52:44kjsetmessages: + msg399852
versions: + Python 3.11, - Python 3.7, Python 3.8
2021-08-18 15:22:21gvanrossumsetmessages: + msg399850
2021-08-18 09:49:33mhilssetmessages: + msg399829
2021-08-17 17:32:19gvanrossumsetmessages: + msg399774
2021-08-17 12:24:12mhilssetmessages: + msg399746
2021-08-16 20:22:21gvanrossumsetmessages: + msg399677
2021-08-16 20:12:48mhilssetmessages: + msg399675
2021-08-16 19:21:45gvanrossumsetmessages: + msg399672
2021-08-16 18:06:27mhilssetmessages: + msg399664
2021-08-16 17:45:54kjsetnosy: + gvanrossum, kj
messages: + msg399662
2021-08-16 17:15:16mhilscreate