➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Leak in pickle (?)
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, loewis, ncoghlan, pitrou, python-dev, skrah, vstinner
Priority: normal Keywords: patch

Created on 2013-12-13 10:14 by skrah, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
issue19972.patch skrah, 2013-12-13 10:40
Messages (10)
msg206044 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-13 10:14 
Hi Alexandre, the following leaks appear after 64c6d52793be. I'm
not sure yet if they're caused or just exposed by the changeset.


Code:
==============
import sys
import pickle
sys.exit(0)
==============


==8118== 864 (192 direct, 672 indirect) bytes in 3 blocks are definitely lost in loss record 2,198 of 2,365
==8118==    at 0x4C28B8C: malloc (vg_replace_malloc.c:270)
==8118==    by 0x41D12A: _PyMem_RawMalloc (obmalloc.c:60)
==8118==    by 0x41D5E9: PyObject_Malloc (obmalloc.c:351)
==8118==    by 0x533FC9: _PyObject_GC_Malloc (gcmodule.c:1726)
==8118==    by 0x534092: _PyObject_GC_New (gcmodule.c:1749)
==8118==    by 0x450CDF: new_dict (dictobject.c:391)
==8118==    by 0x4527BB: _PyDict_NewPresized (dictobject.c:1043)
==8118==    by 0x4DCAFD: PyEval_EvalFrameEx (ceval.c:2366)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4D3A59: PyEval_EvalCode (ceval.c:770)
==8118==    by 0x4CD6BE: builtin_exec (bltinmodule.c:876)
==8118==    by 0x5B11B3: PyCFunction_Call (methodobject.c:93)
==8118==    by 0x4E6946: ext_do_call (ceval.c:4546)
==8118==    by 0x4DFC60: PyEval_EvalFrameEx (ceval.c:2866)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4E5BF7: fast_function (ceval.c:4332)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x5B05F2: function_call (funcobject.c:632)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x42619E: _PyObject_CallMethodIdObjArgs (abstract.c:2359)
==8118==    by 0x502FBE: PyImport_ImportModuleLevelObject (import.c:1473)
==8118==    by 0x4CC3EC: builtin___import__ (bltinmodule.c:210)
==8118==    by 0x5B11D5: PyCFunction_Call (methodobject.c:99)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E4E01: PyEval_CallObjectWithKeywords (ceval.c:4101)
==8118==    by 0x4DD718: PyEval_EvalFrameEx (ceval.c:2466)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4D3A59: PyEval_EvalCode (ceval.c:770)
==8118==    by 0x4CD6BE: builtin_exec (bltinmodule.c:876)
==8118==    by 0x5B11B3: PyCFunction_Call (methodobject.c:93)
==8118==    by 0x4E6946: ext_do_call (ceval.c:4546)
==8118==    by 0x4DFC60: PyEval_EvalFrameEx (ceval.c:2866)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4E5BF7: fast_function (ceval.c:4332)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)


==8118== 2,112 (128 direct, 1,984 indirect) bytes in 2 blocks are definitely lost in loss record 2,301 of 2,365
==8118==    at 0x4C28B8C: malloc (vg_replace_malloc.c:270)
==8118==    by 0x41D12A: _PyMem_RawMalloc (obmalloc.c:60)
==8118==    by 0x41D5E9: PyObject_Malloc (obmalloc.c:351)
==8118==    by 0x533FC9: _PyObject_GC_Malloc (gcmodule.c:1726)
==8118==    by 0x4734D9: PyType_GenericAlloc (typeobject.c:784)
==8118==    by 0x456438: dict_new (dictobject.c:2604)
==8118==    by 0x473355: type_call (typeobject.c:751)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E6313: do_call (ceval.c:4454)
==8118==    by 0x4E57B5: call_function (ceval.c:4252)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4D3A59: PyEval_EvalCode (ceval.c:770)
==8118==    by 0x4CD6BE: builtin_exec (bltinmodule.c:876)
==8118==    by 0x5B11B3: PyCFunction_Call (methodobject.c:93)
==8118==    by 0x4E6946: ext_do_call (ceval.c:4546)
==8118==    by 0x4DFC60: PyEval_EvalFrameEx (ceval.c:2866)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4E5BF7: fast_function (ceval.c:4332)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x5B05F2: function_call (funcobject.c:632)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x42619E: _PyObject_CallMethodIdObjArgs (abstract.c:2359)
==8118==    by 0x502FBE: PyImport_ImportModuleLevelObject (import.c:1473)
==8118==    by 0x4CC3EC: builtin___import__ (bltinmodule.c:210)
==8118==    by 0x5B11D5: PyCFunction_Call (methodobject.c:99)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E4E01: PyEval_CallObjectWithKeywords (ceval.c:4101)
==8118==    by 0x4DD718: PyEval_EvalFrameEx (ceval.c:2466)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4D3A59: PyEval_EvalCode (ceval.c:770)
==8118==    by 0x4CD6BE: builtin_exec (bltinmodule.c:876)
==8118==    by 0x5B11B3: PyCFunction_Call (methodobject.c:93)
==8118==    by 0x4E6946: ext_do_call (ceval.c:4546)
==8118==    by 0x4DFC60: PyEval_EvalFrameEx (ceval.c:2866)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118== 


==8118== 2,402 (64 direct, 2,338 indirect) bytes in 1 blocks are definitely lost in loss record 2,304 of 2,365
==8118==    at 0x4C28B8C: malloc (vg_replace_malloc.c:270)
==8118==    by 0x41D12A: _PyMem_RawMalloc (obmalloc.c:60)
==8118==    by 0x41D5E9: PyObject_Malloc (obmalloc.c:351)
==8118==    by 0x533FC9: _PyObject_GC_Malloc (gcmodule.c:1726)
==8118==    by 0x534092: _PyObject_GC_New (gcmodule.c:1749)
==8118==    by 0x450CDF: new_dict (dictobject.c:391)
==8118==    by 0x450E4C: PyDict_New (dictobject.c:429)
==8118==    by 0x45F97D: module_init (moduleobject.c:372)
==8118==    by 0x47342B: type_call (typeobject.c:766)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E6313: do_call (ceval.c:4454)
==8118==    by 0x4E57B5: call_function (ceval.c:4252)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x5B05F2: function_call (funcobject.c:632)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x42619E: _PyObject_CallMethodIdObjArgs (abstract.c:2359)
==8118==    by 0x502FBE: PyImport_ImportModuleLevelObject (import.c:1473)
==8118==    by 0x4CC3EC: builtin___import__ (bltinmodule.c:210)
==8118==    by 0x5B11D5: PyCFunction_Call (methodobject.c:99)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E4E01: PyEval_CallObjectWithKeywords (ceval.c:4101)
==8118==    by 0x4DD718: PyEval_EvalFrameEx (ceval.c:2466)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4D3A59: PyEval_EvalCode (ceval.c:770)
==8118==    by 0x4CD6BE: builtin_exec (bltinmodule.c:876)
==8118==    by 0x5B11B3: PyCFunction_Call (methodobject.c:93)
==8118==    by 0x4E6946: ext_do_call (ceval.c:4546)
==8118==    by 0x4DFC60: PyEval_EvalFrameEx (ceval.c:2866)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4E5BF7: fast_function (ceval.c:4332)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E5ACA: fast_function (ceval.c:4322)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118== 

==8118== 4,919 (64 direct, 4,855 indirect) bytes in 1 blocks are definitely lost in loss record 2,347 of 2,365
==8118==    at 0x4C28B8C: malloc (vg_replace_malloc.c:270)
==8118==    by 0x41D12A: _PyMem_RawMalloc (obmalloc.c:60)
==8118==    by 0x41D5E9: PyObject_Malloc (obmalloc.c:351)
==8118==    by 0x533FC9: _PyObject_GC_Malloc (gcmodule.c:1726)
==8118==    by 0x534092: _PyObject_GC_New (gcmodule.c:1749)
==8118==    by 0x450CDF: new_dict (dictobject.c:391)
==8118==    by 0x450E15: new_dict_with_shared_keys (dictobject.c:420)
==8118==    by 0x458477: _PyObjectDict_SetItem (dictobject.c:3746)
==8118==    by 0x46215A: _PyObject_GenericSetAttrWithDict (object.c:1143)
==8118==    by 0x462394: PyObject_GenericSetAttr (object.c:1185)
==8118==    by 0x4618E4: PyObject_SetAttr (object.c:914)
==8118==    by 0x4DABEB: PyEval_EvalFrameEx (ceval.c:2111)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x5B05F2: function_call (funcobject.c:632)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x5A1348: method_call (classobject.c:347)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x482B2A: slot_tp_init (typeobject.c:5895)
==8118==    by 0x47342B: type_call (typeobject.c:766)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E6313: do_call (ceval.c:4454)
==8118==    by 0x4E57B5: call_function (ceval.c:4252)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4E5BF7: fast_function (ceval.c:4332)
==8118==    by 0x4E5796: call_function (ceval.c:4250)
==8118==    by 0x4DF8C7: PyEval_EvalFrameEx (ceval.c:2826)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x5B05F2: function_call (funcobject.c:632)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x5A1348: method_call (classobject.c:347)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x42632D: PyObject_CallFunctionObjArgs (abstract.c:2381)
==8118==    by 0x4DF288: PyEval_EvalFrameEx (ceval.c:2711)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x5B05F2: function_call (funcobject.c:632)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x42619E: _PyObject_CallMethodIdObjArgs (abstract.c:2359)
==8118==    by 0x502FBE: PyImport_ImportModuleLevelObject (import.c:1473)
==8118==    by 0x4CC3EC: builtin___import__ (bltinmodule.c:210)
==8118==    by 0x5B11D5: PyCFunction_Call (methodobject.c:99)
==8118==    by 0x4252E7: PyObject_Call (abstract.c:2087)
==8118==    by 0x4E4E01: PyEval_CallObjectWithKeywords (ceval.c:4101)
==8118==    by 0x4DD718: PyEval_EvalFrameEx (ceval.c:2466)
==8118==    by 0x4E3340: PyEval_EvalCodeEx (ceval.c:3576)
==8118==    by 0x4D3A59: PyEval_EvalCode (ceval.c:770)
==8118==    by 0x4CD6BE: builtin_exec (bltinmodule.c:876)
==8118==    by 0x5B11B3: PyCFunction_Call (methodobject.c:93)
==8118==    by 0x4E6946: ext_do_call (ceval.c:4546)
==8118==    by 0x4DFC60: PyEval_EvalFrameEx (ceval.c:2866)
==8118==
msg206051 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-13 10:40 
This patch fixes the leak, but needs review (I do not know the _pickle module
very well).
msg206053 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-12-13 10:58 
According to the example in the PEP 3121, issue19972.patch is not needed.

In practice, I don't see in PyImport_Cleanup() where tp_clear is called. This function calls:

   _PyModule_Clear(mod);
   Py_DECREF(mod);
msg206057 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-13 11:04 
See #11826 for more details where the freefunc is called.  Perhaps the.
docs or the PEP need an update.
msg206059 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-13 11:18 
The docs say:

inquiry m_clear

    A clear function to call during GC clearing of the module object, or NULL if not needed.

freefunc m_free

    A function to call during deallocation of the module object, or NULL if not needed.

So I assume that GC clearing is not happening if sys.exit() is called.  Note that
the leak only occurs with sys.exit().
msg206067 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-13 11:59 
Yeah, it sounds logical for m_free to do the clearing too (same as tp_dealloc vs. tp_clear).

Note that the PEP 3121 scheme turns out to be problematic for proper resource management: https://mail.python.org/pipermail/python-dev/2013-August/127862.html
msg206146 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-12-13 21:01 
The patch is good. I am not sure if you need the freefunc cast though.

The example in the PEP 3121 should updated if freefunc is actually required. I didn't define freefunc because of this example.
msg206153 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-13 22:21 
I just see that it should be:

static void
pickle_free(PyObject *m)
...


Even then, the cast is necessary, otherwise you get this warning:

/home/stefan/hg/cpython/Modules/_pickle.c:7450:1: warning: initialization from incompatible pointer type [enabled by default]
/home/stefan/hg/cpython/Modules/_pickle.c:7450:1: warning: (near initialization for '_picklemodule.m_free') [enabled by default]


I agree that it would be nice to update the PEP, but I guess that
would be Martin's call.
msg206179 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-12-14 12:43 
New changeset 8a78988fdb04 by Stefan Krah in branch 'default':
Issue #19972: Add rarely used freefunc.  This fixes a leak if sys.exit()
http://hg.python.org/cpython/rev/8a78988fdb04
msg206182 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-14 13:26 
Should be fixed. Thanks for the comments everyone.
History
Date User Action Args
2022-04-11 14:57:55adminsetgithub: 64171
2013-12-14 13:26:21skrahsetstatus: open -> closed
resolution: fixed
messages: + msg206182

stage: patch review -> resolved
2013-12-14 12:43:59python-devsetnosy: + python-dev
messages: + msg206179
2013-12-13 22:21:49skrahsetnosy: + loewis
messages: + msg206153
2013-12-13 21:01:49alexandre.vassalottisetmessages: + msg206146
2013-12-13 11:59:27pitrousetnosy: + ncoghlan, pitrou
messages: + msg206067
2013-12-13 11:18:17skrahsetmessages: + msg206059
2013-12-13 11:04:45skrahsetmessages: + msg206057
2013-12-13 10:58:17vstinnersetnosy: + vstinner
messages: + msg206053
2013-12-13 10:45:33skrahsetstage: patch review
2013-12-13 10:40:43skrahsetfiles: + issue19972.patch
keywords: + patch
messages: + msg206051
2013-12-13 10:14:34skrahcreate