This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Custom pickler memory leak
Type: resource usage Stage:
Components: Library (Lib) Versions: Python 3.10
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: douglas-raillard-arm
Priority: normal Keywords:

Created on 2021-11-17 16:42 by douglas-raillard-arm, last changed 2022-04-11 14:59 by admin.

Messages (1)
msg406478 - (view) Author: Douglas Raillard (douglas-raillard-arm) Date: 2021-11-17 16:42
The following script exhibits the memory leak. It only happens if "dispatch_table" is set _before_ calling super().__init__, which is pretty unexpected.

    import pickle
    import io
    import gc
    import tracemalloc

    snap = tracemalloc.take_snapshot()

    class MyPickler(pickle.Pickler):
        def __init__(self, *args, **kwargs):
            # Swapping the next 2 lines "solves" the memory leak for some reason
            self.dispatch_table = dict()
            super().__init__(*args, **kwargs)

    for i in range(10000):
        if i % 1000 == 0:
            snap2 = tracemalloc.take_snapshot()
            stats=snap2.compare_to(snap, 'lineno')
            for s in stats[:10]:
            snap = snap2

        f = io.BytesIO()

The output of the last iteration is as follow. The leak of 562 kiB is apparent: size=562 KiB (+62.5 KiB), count=9000 (+1000), average=64 B
	/usr/lib/python3.10/ size=2376 B (-72 B), count=33 (-1), average=72 B
	/usr/lib/python3.10/ size=72 B (+72 B), count=1 (+1), average=72 B
	/usr/lib/python3.10/ size=252 B (+28 B), count=9 (+1), average=28 B
	/usr/lib/python3.10/ size=2104 B (+0 B), count=36 (+0), average=58 B
	/home/dourai01/Work/lisa/lisa/ size=1844 B (+0 B), count=9 (+0), average=205 B
	/usr/lib/python3.10/ size=1680 B (+0 B), count=35 (+0), average=48 B
	/usr/lib/python3.10/ size=1256 B (+0 B), count=3 (+0), average=419 B
	/usr/lib/python3.10/ size=832 B (+0 B), count=2 (+0), average=416 B
	/usr/lib/python3.10/ size=800 B (+0 B), count=2 (+0), average=400 B

If "dispatch_table" is set after calling super().__init__, there is no leak anymore:

	/usr/lib/python3.10/ size=740 B (+740 B), count=7 (+7), average=106 B
	/usr/lib/python3.10/ size=2088 B (-656 B), count=29 (-4), average=72 B
	/usr/lib/python3.10/ size=320 B (+320 B), count=1 (+1), average=320 B
	/usr/lib/python3.10/ size=0 B (-256 B), count=0 (-1)
	/usr/lib/python3.10/ size=72 B (+72 B), count=1 (+1), average=72 B
	/usr/lib/python3.10/ size=2008 B (+48 B), count=34 (+1), average=59 B
	/usr/lib/python3.10/ size=1584 B (+48 B), count=33 (+1), average=48 B
	/usr/lib/python3.10/ size=196 B (-28 B), count=7 (-1), average=28 B
	/usr/lib/python3.10/ size=84 B (+28 B), count=3 (+1), average=28 B
Date User Action Args
2022-04-11 14:59:52adminsetgithub: 89988
2021-11-17 16:42:28douglas-raillard-armcreate