This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Postponed annotations break inspection of dataclasses
Type: behavior Stage: resolved
Components: Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: yselivanov Nosy List: drhagen, eric.smith, gregory.p.smith, gvanrossum, ivan, levkivskyi, lukasz.langa, methane, ned.deily, scoder, vlad, yselivanov
Priority: deferred blocker Keywords: patch

Created on 2018-09-23 10:45 by drhagen, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9518 merged yselivanov, 2018-09-23 18:25
PR 17531 merged miss-islington, 2019-12-09 14:54
PR 17532 merged miss-islington, 2019-12-09 14:55
Messages (20)
msg326148 - (view) Author: David Hagen (drhagen) Date: 2018-09-23 10:45
The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods.

For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.)  

```
from dataclasses import dataclass
from typing import get_type_hints

class Foo:
    pass

@dataclass
class Bar:
    foo: Foo

print(get_type_hints(Bar.__init__))
```

In Python 3.6 and 3.7, this does what is expected; it prints `{'foo': <class '__main__.Foo'>, 'return': <class 'NoneType'>}`.

However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error:

```
NameError: name 'Foo' is not defined
```

I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined.

I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once.

Original mailing list discussion: https://mail.python.org/pipermail/python-dev/2018-September/155289.html
msg326164 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-09-23 16:35
[Adding to nosy people who were on the original email]

Copying (part of) my response from the email thread:

These work:
print(get_type_hints(Bar.__init__, globals()))
print(get_type_hints(Bar.__init__, Bar.__module__))

But I agree that maybe doing something with dataclasses to address this would be good. Especially as the first one requires being in the same module as Foo.

See this for Yury's self-described "hack-ish fix we can use" until we do something better:

https://gist.github.com/1st1/37fdd3cc84cd65b9af3471b935b722df
msg326170 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-23 18:31
> See this for Yury's self-described "hack-ish fix we can use" until we do something better:

Actually, I think I found a better solution that doesn't require any changes to anything besides dataclasses.

Currently, dataclasses uses 'exec()' function to dynamically create methods like '__init__'.  The generated code for '__init__' needs to access MISSING and _HAS_DEFAULT_FACTORY constants from the dataclasses module.  To do that, we compile the code with 'exec()' with globals set to a dict with {MISSING, _HAS_DEFAULT_FACTORY} keys in it.  This does the trick, but '__init__.__globals__' ends up pointing to that custom dict, instead of pointing to the module's dict.

The other way around is to use a closure around __init__ to inject MISSING and _HAS_DEFAULT_FACTORY values *and* to compile the code in a proper __dict__ of the module the dataclass was defined in.  Please take a look at the PR.
msg326171 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-23 18:33
And FWIF I don't think we need to use lambdas for annotations to solve issues like this one.
msg326266 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-09-24 16:45
Ned: I'm marking this as a release blocker because I'd like to get it in 3.7.1. If I can't do that in the next 5 hours or so, I'll remove the release blocker tag.
msg326307 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-09-25 01:07
Unfortunately, I'm not going to be able to give this the attention it deserves before the 3.7.1 cutoff. The changes are sufficiently tricky that I want to make sure I think through the issue in the appropriate detail.

Ned: I've made it not a release blocker. Please remove yourself as nosy if you don't care about this issue. I don't plan on making it a release blocker again, I just did that because it was so near the deadline.

Yury: Sorry for not being able to get delve in to it sufficiently. Thanks for your work on it.

I should be able to review this in the next few days, so we can get this in to 3.7.2.
msg326308 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-25 01:49
NP, Eric, take your time. I agree that the PR isn't simple and needs a very careful review.
msg326314 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-09-25 04:02
fwiw, agreed that this should wait for 3.7.2.
msg327059 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2018-10-04 15:55
Yury, thanks for your patch. I'll review it soon.

Please note that postponed annotations only reveal a problem that we already had if anybody used a string forward reference:

>>> from dataclasses import dataclass
>>> from typing import get_type_hints
>>> class C:
...   pass
...
>>> @dataclass
... class D:
...   c: C
...
>>> @dataclass
... class E:
...   c: "C"
...
>>> get_type_hints(C.__init__)
{}
>>> get_type_hints(D.__init__)
{'c': <class '__main__.C'>, 'return': <class 'NoneType'>}
>>> get_type_hints(E.__init__)
Traceback (most recent call last):
...
NameError: name 'C' is not defined
msg327062 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-10-04 16:06
> Please note that postponed annotations only reveal a problem that we already had if anybody used a string forward reference:

Yeah, makes sense. It's cool that the PR fixes string forward references as well.
msg337014 - (view) Author: Vlad Shcherbina (vlad) * Date: 2019-03-02 16:20
Any chance this could get into 3.7.3?
msg337237 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-03-05 19:15
I'm finally getting time to look at this. I'll see what I can do.
msg355258 - (view) Author: David Hagen (drhagen) Date: 2019-10-23 19:27
This PR has been sitting for a while. Any chance we can bring it over the finish line?
msg355268 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-10-23 21:53
FWIW we discussed my patch with Eric at PyCon 2019 and I think he had no issues with it.  Eric, can I merge it?  Should we backport to 3.8?
msg355269 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-10-23 22:27
Yury: I'm okay with merging. If I recall, you were going to add a comment or two about the approach (a closure, if I remember correctly).

I think we should backport to 3.8 and 3.7: it's a bug.
msg355270 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-10-23 22:29
> Yury: I'm okay with merging. If I recall, you were going to add a comment or two about the approach (a closure, if I remember correctly).

Sure, I'll rebase and add a comment in a couple of days.  I'll elevate the status so that I don't forget before 3.8.1
msg357987 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-07 20:39
>  I'll elevate the status so that I don't forget before 3.8.1.

Ping, 3.8.1 cutoff is coming up very soon.
msg358106 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-12-09 14:54
New changeset d219cc4180e7589807ebbef7421879f095e72a98 by Łukasz Langa (Yury Selivanov) in branch 'master':
bpo-34776: Fix dataclasses to support __future__ "annotations" mode (#9518)
https://github.com/python/cpython/commit/d219cc4180e7589807ebbef7421879f095e72a98
msg358112 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-12-09 16:07
New changeset 0d57db27f2c563b6433a220b646b50bdeff8a1f2 by Łukasz Langa (Miss Islington (bot)) in branch '3.8':
bpo-34776: Fix dataclasses to support __future__ "annotations" mode (GH-9518) (#17531)
https://github.com/python/cpython/commit/0d57db27f2c563b6433a220b646b50bdeff8a1f2
msg358113 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-12-09 16:07
New changeset 66d7a5d58a88bce312bc4668f2cc54c9488c5bd8 by Łukasz Langa (Miss Islington (bot)) in branch '3.7':
bpo-34776: Fix dataclasses to support __future__ "annotations" mode (GH-9518) (#17532)
https://github.com/python/cpython/commit/66d7a5d58a88bce312bc4668f2cc54c9488c5bd8
History
Date User Action Args
2022-04-11 14:59:06adminsetgithub: 78957
2019-12-09 16:41:55lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-12-09 16:07:58lukasz.langasetmessages: + msg358113
2019-12-09 16:07:55lukasz.langasetmessages: + msg358112
2019-12-09 14:55:00miss-islingtonsetpull_requests: + pull_request17010
2019-12-09 14:54:52miss-islingtonsetpull_requests: + pull_request17009
2019-12-09 14:54:27lukasz.langasetmessages: + msg358106
2019-12-07 20:39:18ned.deilysetmessages: + msg357987
2019-10-23 22:29:27yselivanovsetpriority: deferred blocker
assignee: eric.smith -> yselivanov
messages: + msg355270
2019-10-23 22:27:46eric.smithsetmessages: + msg355269
2019-10-23 21:53:11yselivanovsetmessages: + msg355268
2019-10-23 19:44:16gregory.p.smithsetversions: + Python 3.9
2019-10-23 19:27:43drhagensetmessages: + msg355258
2019-03-05 19:15:49eric.smithsetmessages: + msg337237
2019-03-02 16:20:28vladsetnosy: + vlad
messages: + msg337014
2018-10-04 16:06:57yselivanovsetmessages: + msg327062
2018-10-04 15:55:03lukasz.langasetmessages: + msg327059
2018-09-28 06:27:41scodersetnosy: + scoder
2018-09-27 17:06:43methanesetnosy: + methane
2018-09-27 16:47:02levkivskyisetnosy: + levkivskyi
2018-09-25 04:02:43gregory.p.smithsetmessages: + msg326314
2018-09-25 04:01:50gregory.p.smithsetnosy: + gregory.p.smith

versions: + Python 3.8
2018-09-25 01:49:35yselivanovsetmessages: + msg326308
2018-09-25 01:07:46eric.smithsetpriority: release blocker -> (no value)

messages: + msg326307
2018-09-24 16:45:53eric.smithsetpriority: normal -> release blocker

nosy: + ned.deily
messages: + msg326266

assignee: eric.smith
2018-09-23 18:33:27yselivanovsetmessages: + msg326171
2018-09-23 18:31:42yselivanovsetmessages: + msg326170
2018-09-23 18:25:32yselivanovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8923
2018-09-23 16:35:44eric.smithsetnosy: + ivan, eric.smith, gvanrossum, lukasz.langa, yselivanov
messages: + msg326164
2018-09-23 10:45:29drhagencreate