Title: reference cycle affecting Pickler instances (Python3.8+)
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.9, Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, pierreglaser, pitrou
Priority: normal Keywords: patch

Created on 2020-01-29 23:39 by pierreglaser, last changed 2020-02-02 20:23 by pitrou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18266 merged pierreglaser, 2020-01-30 01:13
PR 18316 merged pitrou, 2020-02-02 19:47
Messages (4)
msg360995 - (view) Author: Pierre Glaser (pierreglaser) * Date: 2020-01-29 23:39
The new Pickler reducer_override mechanism introduced in `Python3.8` generates a reference cycle: for optimization purposes, a the pickler.reducer_override bound method is referenced into the reducer_override attribute of the Pickler's struct. Thus, until as a gc.collect call is performed, both the Pickler and all the elements it pickled (as they are part of its memo), wont be collected.

We should break this cycle a the end of the dump() method.

See reproducer below:
import threading
import weakref
import pickle
import io

class MyClass:

my_object = MyClass()
collect = threading.Event()
_ = weakref.ref(my_object, lambda obj: collect.set())  # noqa

class MyPickler(pickle.Pickler):
    def reducer_override(self, obj):
        return NotImplemented

my_pickler = MyPickler(io.BytesIO())
del my_object
del my_pickler

# import gc
# gc.collect()

for i in range(5):
    collected = collect.wait(timeout=0.1)
    if collected:
        print('my_object was successfully collected')
msg361252 - (view) Author: miss-islington (miss-islington) Date: 2020-02-02 18:55
New changeset 0f2f35e15f9fbee44ce042b724348419d8136bc5 by Pierre Glaser in branch 'master':
bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266)
msg361254 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2020-02-02 20:23
New changeset 17236873392533ce0c5d7bbf52cbd61bca171c59 by Antoine Pitrou in branch '3.8':
[3.8] bpo-39492: Fix a reference cycle between reducer_override and a Pickler instance (GH-18266) (#18316)
msg361255 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2020-02-02 20:23
Thank you for the PR!
Date User Action Args
2020-02-02 20:23:21pitrousetstatus: open -> closed
resolution: fixed
messages: + msg361255

stage: patch review -> resolved
2020-02-02 20:23:04pitrousetmessages: + msg361254
2020-02-02 19:47:53pitrousetpull_requests: + pull_request17692
2020-02-02 18:55:25miss-islingtonsetnosy: + miss-islington
messages: + msg361252
2020-01-30 01:13:13pierreglasersetkeywords: + patch
stage: patch review
pull_requests: + pull_request17643
2020-01-29 23:39:28pierreglasercreate