Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

reference cycle affecting Pickler instances (Python3.8+) #83673

Closed
pierreglaser mannequin opened this issue Jan 29, 2020 · 4 comments
Closed

reference cycle affecting Pickler instances (Python3.8+) #83673

pierreglaser mannequin opened this issue Jan 29, 2020 · 4 comments
Labels
3.8 only security fixes 3.9 only security fixes performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@pierreglaser
Copy link
Mannequin

pierreglaser mannequin commented Jan 29, 2020

BPO 39492
Nosy @pitrou, @miss-islington, @pierreglaser
PRs
  • bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance #18266
  • [3.8] bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) #18316
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-02-02.20:23:21.907>
created_at = <Date 2020-01-29.23:39:27.963>
labels = ['3.8', 'library', '3.9', 'performance']
title = 'reference cycle affecting Pickler instances (Python3.8+)'
updated_at = <Date 2020-02-02.20:23:21.905>
user = 'https://github.com/pierreglaser'

    bugs.python.org fields:

    activity = <Date 2020-02-02.20:23:21.905>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2020-02-02.20:23:21.907>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2020-01-29.23:39:27.963>
creator = 'pierreglaser'
dependencies = []
files = []
hgrepos = []
issue_num = 39492
keywords = ['patch']
message_count = 4.0
messages = ['360995', '361252', '361254', '361255']
nosy_count = 3.0
nosy_names = ['pitrou', 'miss-islington', 'pierreglaser']
pr_nums = ['18266', '18316']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue39492'
versions = ['Python 3.8', 'Python 3.9']
    @pierreglaser
    Copy link
    Mannequin Author

    pierreglaser mannequin commented Jan 29, 2020

    The new Pickler reducer_override mechanism introduced in Python3.8 generates a reference cycle: for optimization purposes, a the pickler.reducer_override bound method is referenced into the reducer_override attribute of the Pickler's struct. Thus, until as a gc.collect call is performed, both the Pickler and all the elements it pickled (as they are part of its memo), wont be collected.

    We should break this cycle a the end of the dump() method.

    See reproducer below:

    import threading
import weakref
import pickle
import io


class MyClass:
    pass


my_object = MyClass()
collect = threading.Event()
_ = weakref.ref(my_object, lambda obj: collect.set())  # noqa


class MyPickler(pickle.Pickler):
    def reducer_override(self, obj):
        return NotImplemented


my_pickler = MyPickler(io.BytesIO())
my_pickler.dump(my_object)
del my_object
del my_pickler

# import gc
# gc.collect()

for i in range(5):
    collected = collect.wait(timeout=0.1)
    if collected:
        print('my_object was successfully collected')
        break

    @pierreglaser pierreglaser mannequin added 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir performance Performance or resource usage labels Jan 29, 2020
    @miss-islington
    Copy link
    Contributor

    New changeset 0f2f35e by Pierre Glaser in branch 'master':
    bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266)
    0f2f35e

    @pitrou
    Copy link
    Member

    pitrou commented Feb 2, 2020

    New changeset 1723687 by Antoine Pitrou in branch '3.8':
    [3.8] bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) (bpo-18316)
    1723687

    @pitrou
    Copy link
    Member

    pitrou commented Feb 2, 2020

    Thank you for the PR!

    @pitrou pitrou closed this as completed Feb 2, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes 3.9 only security fixes performance Performance or resource usage stdlib Python modules in the Lib dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @pitrou @miss-islington