Title: Postponed annotations break inspection of dataclasses
Type: behavior Stage: patch review
Components: Versions: Python 3.8, Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: drhagen, eric.smith, gregory.p.smith, gvanrossum, inada.naoki, ivan, levkivskyi, lukasz.langa, ned.deily, scoder, vlad, yselivanov
Priority: Keywords: patch

Created on 2018-09-23 10:45 by drhagen, last changed 2019-03-05 19:15 by eric.smith.

Pull Requests
URL Status Linked Edit
PR 9518 open yselivanov, 2018-09-23 18:25
Messages (12)
msg326148 - (view) Author: David Hagen (drhagen) Date: 2018-09-23 10:45
The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods.

For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.)  

from dataclasses import dataclass
from typing import get_type_hints

class Foo:

class Bar:
    foo: Foo


In Python 3.6 and 3.7, this does what is expected; it prints `{'foo': <class '__main__.Foo'>, 'return': <class 'NoneType'>}`.

However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error:

NameError: name 'Foo' is not defined

I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined.

I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once.

Original mailing list discussion:
msg326164 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-09-23 16:35
[Adding to nosy people who were on the original email]

Copying (part of) my response from the email thread:

These work:
print(get_type_hints(Bar.__init__, globals()))
print(get_type_hints(Bar.__init__, Bar.__module__))

But I agree that maybe doing something with dataclasses to address this would be good. Especially as the first one requires being in the same module as Foo.

See this for Yury's self-described "hack-ish fix we can use" until we do something better:
msg326170 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-23 18:31
> See this for Yury's self-described "hack-ish fix we can use" until we do something better:

Actually, I think I found a better solution that doesn't require any changes to anything besides dataclasses.

Currently, dataclasses uses 'exec()' function to dynamically create methods like '__init__'.  The generated code for '__init__' needs to access MISSING and _HAS_DEFAULT_FACTORY constants from the dataclasses module.  To do that, we compile the code with 'exec()' with globals set to a dict with {MISSING, _HAS_DEFAULT_FACTORY} keys in it.  This does the trick, but '__init__.__globals__' ends up pointing to that custom dict, instead of pointing to the module's dict.

The other way around is to use a closure around __init__ to inject MISSING and _HAS_DEFAULT_FACTORY values *and* to compile the code in a proper __dict__ of the module the dataclass was defined in.  Please take a look at the PR.
msg326171 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-23 18:33
And FWIF I don't think we need to use lambdas for annotations to solve issues like this one.
msg326266 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-09-24 16:45
Ned: I'm marking this as a release blocker because I'd like to get it in 3.7.1. If I can't do that in the next 5 hours or so, I'll remove the release blocker tag.
msg326307 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-09-25 01:07
Unfortunately, I'm not going to be able to give this the attention it deserves before the 3.7.1 cutoff. The changes are sufficiently tricky that I want to make sure I think through the issue in the appropriate detail.

Ned: I've made it not a release blocker. Please remove yourself as nosy if you don't care about this issue. I don't plan on making it a release blocker again, I just did that because it was so near the deadline.

Yury: Sorry for not being able to get delve in to it sufficiently. Thanks for your work on it.

I should be able to review this in the next few days, so we can get this in to 3.7.2.
msg326308 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-25 01:49
NP, Eric, take your time. I agree that the PR isn't simple and needs a very careful review.
msg326314 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-09-25 04:02
fwiw, agreed that this should wait for 3.7.2.
msg327059 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2018-10-04 15:55
Yury, thanks for your patch. I'll review it soon.

Please note that postponed annotations only reveal a problem that we already had if anybody used a string forward reference:

>>> from dataclasses import dataclass
>>> from typing import get_type_hints
>>> class C:
...   pass
>>> @dataclass
... class D:
...   c: C
>>> @dataclass
... class E:
...   c: "C"
>>> get_type_hints(C.__init__)
>>> get_type_hints(D.__init__)
{'c': <class '__main__.C'>, 'return': <class 'NoneType'>}
>>> get_type_hints(E.__init__)
Traceback (most recent call last):
NameError: name 'C' is not defined
msg327062 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-10-04 16:06
> Please note that postponed annotations only reveal a problem that we already had if anybody used a string forward reference:

Yeah, makes sense. It's cool that the PR fixes string forward references as well.
msg337014 - (view) Author: Vlad Shcherbina (vlad) * Date: 2019-03-02 16:20
Any chance this could get into 3.7.3?
msg337237 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-03-05 19:15
I'm finally getting time to look at this. I'll see what I can do.
Date User Action Args
2019-03-05 19:15:49eric.smithsetmessages: + msg337237
2019-03-02 16:20:28vladsetnosy: + vlad
messages: + msg337014
2018-10-04 16:06:57yselivanovsetmessages: + msg327062
2018-10-04 15:55:03lukasz.langasetmessages: + msg327059
2018-09-28 06:27:41scodersetnosy: + scoder
2018-09-27 17:06:43inada.naokisetnosy: + inada.naoki
2018-09-27 16:47:02levkivskyisetnosy: + levkivskyi
2018-09-25 04:02:43gregory.p.smithsetmessages: + msg326314
2018-09-25 04:01:50gregory.p.smithsetnosy: + gregory.p.smith

versions: + Python 3.8
2018-09-25 01:49:35yselivanovsetmessages: + msg326308
2018-09-25 01:07:46eric.smithsetpriority: release blocker ->

messages: + msg326307
2018-09-24 16:45:53eric.smithsetpriority: normal -> release blocker

nosy: + ned.deily
messages: + msg326266

assignee: eric.smith
2018-09-23 18:33:27yselivanovsetmessages: + msg326171
2018-09-23 18:31:42yselivanovsetmessages: + msg326170
2018-09-23 18:25:32yselivanovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8923
2018-09-23 16:35:44eric.smithsetnosy: + ivan, eric.smith, gvanrossum, lukasz.langa, yselivanov
messages: + msg326164
2018-09-23 10:45:29drhagencreate