Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dataclasses.astuple (and .asdict) do deepcopy on all fields #88071

Closed
mandolaerik mannequin opened this issue Apr 21, 2021 · 14 comments
Closed

dataclasses.astuple (and .asdict) do deepcopy on all fields #88071

mandolaerik mannequin opened this issue Apr 21, 2021 · 14 comments
Assignees
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes docs Documentation in the Doc dir type-bug An unexpected behavior, bug, or error

Comments

@mandolaerik
Copy link
Mannequin

mandolaerik mannequin commented Apr 21, 2021

BPO 43905
Nosy @ericvsmith, @serhiy-storchaka, @miss-islington, @akulakov
PRs
  • bpo-43905: Expand dataclasses.astuple() and asdict() docs #26154
  • [3.10] bpo-43905: Expand dataclasses.astuple() and asdict() docs (GH-26154) #29851
  • [3.9] bpo-43905: Expand dataclasses.astuple() and asdict() docs (GH-26154) #29852
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ericvsmith'
    closed_at = <Date 2021-11-29.18:34:32.678>
    created_at = <Date 2021-04-21.12:19:17.095>
    labels = ['3.11', 'type-bug', '3.9', '3.10', 'docs']
    title = 'dataclasses.astuple (and .asdict) do deepcopy on all fields'
    updated_at = <Date 2021-11-29.18:44:26.912>
    user = 'https://bugs.python.org/mandolaerik'

    bugs.python.org fields:

    activity = <Date 2021-11-29.18:44:26.912>
    actor = 'andrei.avk'
    assignee = 'eric.smith'
    closed = True
    closed_date = <Date 2021-11-29.18:34:32.678>
    closer = 'eric.smith'
    components = ['Documentation']
    creation = <Date 2021-04-21.12:19:17.095>
    creator = 'mandolaerik'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43905
    keywords = ['patch']
    message_count = 14.0
    messages = ['391516', '391518', '391519', '391520', '394015', '394019', '396284', '407241', '407243', '407306', '407310', '407311', '407314', '407316']
    nosy_count = 5.0
    nosy_names = ['eric.smith', 'serhiy.storchaka', 'miss-islington', 'andrei.avk', 'mandolaerik']
    pr_nums = ['26154', '29851', '29852']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue43905'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Linked PRs

    @mandolaerik
    Copy link
    Mannequin Author

    mandolaerik mannequin commented Apr 21, 2021

    It seems that the 'dataclass.astuple' function does a deepcopy of all fields. This is not documented. Two problems:

    1. Dictionary keys that rely on object identity are ruined:

      import dataclasses
      @dataclasses.dataclass
      class Foo:
          key: object
      key = object()
      lut = {key: 5}
      (y,) = dataclasses.astuple(Foo(x))
      \# KeyError
      lut[y]
    2. dataclasses can only be converted to a tuple if all fields are serializable:

      import dataclasses
      @dataclasses.dataclass
      class Foo:
          f: object
      foo = Foo(open('test.py'))
      dataclasses.astuple(foo)

    ->

    TypeError: cannot pickle '_io.TextIOWrapper' object
    

    In my use case, I just want a list of all fields. I can do the following as a workaround:

    (getattr(foo, field.name) for field in dataclasses.fields(foo))

    Tested on Python 3.8.7 and 3.7.9.

    @mandolaerik mandolaerik mannequin added stdlib Python modules in the Lib dir 3.7 (EOL) end of life 3.8 only security fixes labels Apr 21, 2021
    @ericvsmith
    Copy link
    Member

    Unfortunately this can't be changed, although I suppose it should be documented.

    In general I think this API was a mistake, and should not have been added. There are just too many cases where it doesn't do what you want, or where it fails.

    I'd like to deprecate it and remove it (along with asdict), but I fear that would be too disruptive.

    Your approach seems reasonable to me for your use case.

    @serhiy-storchaka
    Copy link
    Member

    Why deepcopy is used at all? It is a very specific feature which should not be used by default. If you want to make a deep copy of fields, you can call copy.deepcopy() explicitly.

        copy.deepcopy(dataclasses.astuple(obj))

    or

        dataclasses.astuple(copy.deepcopy(obj))

    @ericvsmith
    Copy link
    Member

    The reason for the deep copying was to support changing a hierarchy of dataclasses into something that could be JSON serialized. But it didn't really work out. It recurses into dataclasses, namedtuples, lists, tuples, and dicts, and deep copies everything else.

    As I said, it's a design flaw.

    @mandolaerik
    Copy link
    Mannequin Author

    mandolaerik mannequin commented May 20, 2021

    Would it make sense to make dataclasses iterable, like so?

        def __iter__(self):
            return (getattr(self, field.name) for field in fields(self))

    With that in place, deprecating astuple would maybe be less disruptive?

    @ericvsmith
    Copy link
    Member

    No, iteration is explicitly a non-goal of PEP-557. See the section on namedtuple for why: https://www.python.org/dev/peps/pep-0557/#why-not-just-use-namedtuple

    @akulakov
    Copy link
    Contributor

    I've added a PR here: #26154

    @akulakov
    Copy link
    Contributor

    Eric: I've closed a similar issue about asdict() and now updating the title to keep track of both in this issue. Let me know if you want to keep them separate instead.

    @akulakov akulakov changed the title dataclasses.astuple does deepcopy on all fields dataclasses.astuple (and .asdict) do deepcopy on all fields Nov 29, 2021
    @akulakov akulakov changed the title dataclasses.astuple does deepcopy on all fields dataclasses.astuple (and .asdict) do deepcopy on all fields Nov 29, 2021
    @ericvsmith
    Copy link
    Member

    I think it's find to address both of these here.

    @ericvsmith ericvsmith added docs Documentation in the Doc dir 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes and removed stdlib Python modules in the Lib dir 3.7 (EOL) end of life 3.8 only security fixes labels Nov 29, 2021
    @ericvsmith ericvsmith added 3.11 only security fixes and removed 3.7 (EOL) end of life 3.8 only security fixes labels Nov 29, 2021
    @miss-islington
    Copy link
    Contributor

    New changeset c1f93f0 by andrei kulakov in branch 'main':
    bpo-43905: Expand dataclasses.astuple() and asdict() docs (GH-26154)
    c1f93f0

    @miss-islington
    Copy link
    Contributor

    New changeset 376b24e by Miss Islington (bot) in branch '3.9':
    bpo-43905: Expand dataclasses.astuple() and asdict() docs (GH-26154)
    376b24e

    @miss-islington
    Copy link
    Contributor

    New changeset 32f1491 by Miss Islington (bot) in branch '3.10':
    bpo-43905: Expand dataclasses.astuple() and asdict() docs (GH-26154)
    32f1491

    @ericvsmith
    Copy link
    Member

    Thanks, @andrei.avk!

    @ericvsmith ericvsmith added the type-bug An unexpected behavior, bug, or error label Nov 29, 2021
    @ericvsmith ericvsmith added the type-bug An unexpected behavior, bug, or error label Nov 29, 2021
    @akulakov
    Copy link
    Contributor

    Thank you for reviewing Eric!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    carljm pushed a commit that referenced this issue Mar 11, 2023
    Update dataclasses.astuple and dataclasses.asdict docstrings to reflect that they deep copy objects in the field values.
    iritkatriel pushed a commit to iritkatriel/cpython that referenced this issue Mar 12, 2023
    …ython#101806)
    
    Update dataclasses.astuple and dataclasses.asdict docstrings to reflect that they deep copy objects in the field values.
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes docs Documentation in the Doc dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants