Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

from __future__ import annotations makes dataclasses.Field.type a string, not type #83623

Closed
lopekpl mannequin opened this issue Jan 24, 2020 · 15 comments
Closed

from __future__ import annotations makes dataclasses.Field.type a string, not type #83623

lopekpl mannequin opened this issue Jan 24, 2020 · 15 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error

Comments

@lopekpl
Copy link
Mannequin

lopekpl mannequin commented Jan 24, 2020

BPO 39442
Nosy @ericvsmith, @methane, @drhagen, @lopekpl, @barisione, @ARF1, @davetapley

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-24.11:48:40.731>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'from __future__ import annotations makes dataclasses.Field.type a string, not type'
updated_at = <Date 2022-04-08.09:16:17.541>
user = 'https://github.com/lopekpl'

bugs.python.org fields:

activity = <Date 2022-04-08.09:16:17.541>
actor = 'barisione'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-01-24.11:48:40.731>
creator = 'lopek'
dependencies = []
files = []
hgrepos = []
issue_num = 39442
keywords = []
message_count = 13.0
messages = ['360611', '360613', '360617', '360628', '360640', '360641', '360760', '381398', '381399', '401549', '401558', '416963', '416965']
nosy_count = 8.0
nosy_names = ['eric.smith', 'methane', 'drhagen', 'lopek', 'barisione', 'ARF1', 'dcecile', 'davetapley']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39442'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

@lopekpl
Copy link
Mannequin Author

lopekpl mannequin commented Jan 24, 2020

I've checked this behaviour under Python 3.7.5 and 3.8.1.

from __future__ import annotations
from dataclasses import dataclass, fields

@dataclass
class Foo:
    x: int

print(fields(Foo)[0].type)

With annotations imported, the type field of Field class becomes a string with a name of a type, and the program outputs 'int'.

Without annotations, the type field of Field class is a type, and the program outputs <class 'int'>.

I found this out when using dataclasses_serialization module. Following code works fine when we remove import of annotations:

from __future__ import annotations
from dataclasses import dataclass
from dataclasses_serialization.json import JSONSerializer

@dataclass
class Foo:
    x: int

JSONSerializer.deserialize(Foo, {'x': 42})

TypeError: issubclass() arg 1 must be a class

@lopekpl lopekpl mannequin added 3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jan 24, 2020
@lopekpl lopekpl mannequin changed the title from __future__ import annotations breaks dataclasses.Field.type from __future__ import annotations makes dataclasses.Field.type a string, not type Jan 24, 2020
@lopekpl lopekpl mannequin changed the title from __future__ import annotations breaks dataclasses.Field.type from __future__ import annotations makes dataclasses.Field.type a string, not type Jan 24, 2020
@ericvsmith
Copy link
Member

Isn't that the entire point of "from __future__ import annotations"?

Also, please show the traceback when reporting errors so that I can see what's going on.

@lopekpl
Copy link
Mannequin Author

lopekpl mannequin commented Jan 24, 2020

Isn't that the entire point of "from __future__ import annotations"?
I'm not complaining about Foo.__annotations__ storing strings instead of types. I'm complaining about dataclass.Field.type being a string instead of type. I don't think the former needs to imply the latter. I'm trying to access Field objects at runtime, when it should already be possible to resolve the types, as far as I understand.

Also, please show the traceback when reporting errors so that I can see what's going on.

That's the error I get trying to use dataclasses_serialization module:

$ cat test.py 
from __future__ import annotations
from dataclasses import dataclass
from dataclasses_serialization.json import JSONSerializer
@dataclass
class Foo:
    x: int

JSONSerializer.deserialize(Foo, {'x': 42})
$ python3 test.py 
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dataclasses_serialization/serializer_base.py", line 125, in dict_to_dataclass
    for fld, fld_type in zip(flds, fld_types)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dataclasses_serialization/serializer_base.py", line 126, in <dictcomp>
    if fld.name in dct
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dataclasses_serialization/serializer_base.py", line 234, in deserialize
    if issubclass(cls, type_):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dataclasses_serialization/serializer_base.py", line 72, in issubclass
    return original_issubclass(cls, classinfo)
TypeError: issubclass() arg 1 must be a class
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    JSONSerializer.deserialize(Foo, {'x': 42})
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dataclasses_serialization/serializer_base.py", line 238, in deserialize
    return self.deserialization_functions[dataclass](cls, serialized_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/toolz/functoolz.py", line 303, in __call__
    return self._partial(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/dataclasses_serialization/serializer_base.py", line 131, in dict_to_dataclass
    cls
dataclasses_serialization.serializer_base.DeserializationError: Missing one or more required fields to deserialize {'x': 42} as <class '__main__.Foo'>

@ericvsmith
Copy link
Member

Well the type comes from the annotation, so this makes sense to me. If dataclasses were to call get_type_hints() for every field, it would defeat the purpose of PEP-563 (at least for dataclasses).

@drhagen
Copy link
Mannequin

drhagen mannequin commented Jan 24, 2020

Should dataclass.Field.type become a property that evaluates the annotation at runtime much in the same way that get_type_hints works?

@ericvsmith
Copy link
Member

Should dataclass.Field.type become a property that evaluates the annotation at runtime much in the same way that get_type_hints works?

I think not. But maybe a function that evaluates all of the field types. Or maybe an @DataClass parameter to cause it to happen at definition time.

At this point, this seems more like fodder for python-ideas.

@lopekpl
Copy link
Mannequin Author

lopekpl mannequin commented Jan 27, 2020

I thought of this behaviour as a bug, because PEP-563 mentions breaking "applications depending on arbitrary objects to be directly present in annotations", while it is also breaking users of dataclasses.fields(), that is a part of the standard library. But if it's not something worth fighting for, feel free to close this issue.

@ARF1
Copy link
Mannequin

ARF1 mannequin commented Nov 19, 2020

One problem I have with the current behaviour is that users of library code need to know the exact namespace in which a library has defined a dataclass.

An example is if a library writer had to deconflict the name of a type he used in a user-facing dataclass.

Below is a "typical" use case which will become very fragile to implement.(E.g. imagine the dataclass with dynamically generated fields, the implementation of which I have neglected for the sake of brevity.)

=== some_library_typing.py ===

mytype = str  # library author defines some type alias

=== some_library_module_a.py ===

from __future__ import annotations
import dataclasses
from some_library_typing import mytype as mytype_deconflicted

mytype = int

@dataclasses.dataclass
class MyClass:
    var1: mytype_deconflicted = 'foo'

    def method1(self, val: mytype) -> mytype:
        return val + 1

=== user_code.py ===

from __future__ import annotations
import dataclasses
from some_library_typing import mytype
from some_library_module_a import MyClass

inst = MyClass('bar')

for f in dataclasses.fields(inst):
    if f.type is mytype:
        print('mytype found')
        break
else:
    print('mytype not found')

The if f.type is mytype comparison obviously won't work any more. But neither will if f.type == 'mytype'. The user will have to be aware that the library author had to deconflict the identifier mytype to mytype_deconflicted to write his code.

Of course, the library writer could have written the following to make the code work:

=== some_library_module_a.py ===

from __future__ import annotations
import dataclasses
from some_library_typing import mytype as mytype_deconflicted

mytype = int

@dataclasses.dataclass
class MyClass:
    var1: mytype = 'foo'

    def method1(self, val: mytype)
        return val + 1

That is a phenomenally obscure and counter-intuitive way of writing code!

Whichever way one turns this, the current behaviour either seems to require library authors to take extraordinary care with their namespaces when defining dataclasses or forces them to write hard-to-read code or seems to require from users detailed knowledge about the implementation specifics of a library they use.

If this behaviour is kept as is, some clear warnings and guidance on how to deal with this in practice should be given in the docs. From what I can see in the 3.10 docs, that is not yet the case.

@ARF1 ARF1 mannequin added 3.9 only security fixes labels Nov 19, 2020
@ARF1
Copy link
Mannequin

ARF1 mannequin commented Nov 19, 2020

Another counter-intuitive behaviour is the different behaviour of dataclasses depending on whether they were defined with the decorator or the make_dataclass factory method:

from __future__ import annotations
import dataclasses

mytype = int

@dataclasses.dataclass
class MyClass1:
    foo: mytype = 1

MyClass2 = dataclasses.make_dataclass(
    f'MyClass2',
    [('foo', mytype, 1)]
)

print(dataclasses.fields(MyClass1)[0].type)
print(dataclasses.fields(MyClass2)[0].type)

Results in:

mytype
<class 'int'>

@davetapley
Copy link
Mannequin

davetapley mannequin commented Sep 10, 2021

I don't know if it helps, but I just ran in to this when I followed the advice at (1) because I wanted to type hint a method with the type of the enclosing class.

This broke a package I'm working on in parallel (2) because it uses dataclasses.fields internally.

I'm not sure what the advice would be here, should my package detect if the caller has from __future__ import annotations and do something?

(1) https://stackoverflow.com/questions/33533148/how-do-i-type-hint-a-method-with-the-type-of-the-enclosing-class/33533514#33533514
(2) https://pypi.org/project/dataclasses-configobj/

@methane
Copy link
Member

methane commented Sep 10, 2021

I think pydantic approach is the best practice.
See https://pydantic-docs.helpmanual.io/usage/postponed_annotations/

@barisione
Copy link
Mannequin

barisione mannequin commented Apr 8, 2022

This is particularly annoying if you are using Annotated with a dataclass.

For instance:

from __future__ import annotations

import dataclasses
from typing import Annotated, get_type_hints


@dataclasses.dataclass
class C:
    v: Annotated[int, "foo"]


v_type = dataclasses.fields(C)[0].type
print(repr(v_type))  # "Annotated[int, 'foo']"
print(repr(get_type_hints(C)["v"]))  # <class 'int'>
print(repr(eval(v_type)))  # typing.Annotated[int, 'foo']

In the code above it looks like the only way to get the Annotated so you get get its args is using eval. The problem is that, in non-trivial, examples, eval would not be simple to use as you need to consider globals and locals, see https://peps.python.org/pep-0563/#resolving-type-hints-at-runtime.

@barisione
Copy link
Mannequin

barisione mannequin commented Apr 8, 2022

Actually, sorry I realise I can pass include_extras to get_type_hints.
Still, it would be nicer not to have to do that.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@peku33
Copy link

peku33 commented Jan 10, 2023

Are there any plans for fixing this? The behavior described in documentation (type: The type of the field.) differs from actual returned type based on side effect outside of this module.

I understand there is a workaround using get_type_hints, but still this is a workaround and part of core library works not as expected.

@methane
Copy link
Member

methane commented Jan 10, 2023

Do not use from __future__ import annotations when you need actual type object.
This option won't be default. So no plan to fix it.

PEP 649 will be accepted, but you can not use it for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

4 participants