classification
Title: ctypes: memory leak
Type: resource usage Stage: needs patch
Components: ctypes, Extension Modules Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: leak in ctypes.resize()
View: 16628
Assigned To: Nosy List: amaury.forgeotdarc, belopolsky, meador.inge, pitrou, skrah
Priority: normal Keywords:

Created on 2011-10-02 09:11 by skrah, last changed 2012-12-07 19:20 by pitrou. This issue is now closed.

Messages (5)
msg144765 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-10-02 09:11
Seen in test_multiprocessing:

==31662== 44 bytes in 1 blocks are definitely lost in loss record 687 of 10,548
==31662==    at 0x4C2154B: malloc (vg_replace_malloc.c:236)
==31662==    by 0x41CC27: PyMem_Malloc (object.c:1699)
==31662==    by 0x127D9F51: resize (callproc.c:1664)
==31662==    by 0x5759D8: PyCFunction_Call (methodobject.c:81)
==31662==    by 0x48E294: call_function (ceval.c:3980)
==31662==    by 0x4895E1: PyEval_EvalFrameEx (ceval.c:2605)
==31662==    by 0x48E67B: fast_function (ceval.c:4068)
==31662==    by 0x48E3C7: call_function (ceval.c:4001)
==31662==    by 0x4895E1: PyEval_EvalFrameEx (ceval.c:2605)
==31662==    by 0x48C54F: PyEval_EvalCodeEx (ceval.c:3355)
==31662==    by 0x48E786: fast_function (ceval.c:4078)
==31662==    by 0x48E3C7: call_function (ceval.c:4001)
msg144784 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-02 22:13
How did you obtain this? the resize() function is not called by test_multiprocessing.
And are you sure that it's not some kind of reference leak? (this pointer is tied to a CDataObject; its tp_alloc should free the memory)
msg144791 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-02 22:58
I can reproduce this with:

valgrind --tool=memcheck --log-file=leaks.txt --leak-check=full --suppressions=Misc/valgrind-python.supp ./python -m test test_ctypes

Where as:

valgrind --tool=memcheck --log-file=leaks.txt --leak-check=full --suppressions=Misc/valgrind-python.supp ./python -m test test_multiprocessing

turns up nothing in 'ctypes.resize'.
msg144797 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-03 00:22
> this pointer is tied to a CDataObject; its tp_alloc should free the 
> memory

The free in 'PyCData_clear' is conditional:

    if ((self->b_needsfree)
        && ((size_t)dict->size > sizeof(self->b_value)))
        PyMem_Free(self->b_ptr);

As written, 'PyCData_clear' has no way of knowing that memory has been 
{m,re}alloc'd in 'resize'.  So in some cases memory will leak.  Here is 
a small reproduction case extracted from 'test_varsize_struct.py'.

    from ctypes import *

    class X(Structure):
        _fields_ = [("item", c_int),
                    ("array", c_int * 1)]

    x = X()
    x.item = 42
    x.array[0] = 100
    new_size = sizeof(X) + sizeof(c_int) * 5
    resize(x, new_size)

One potential fix is:

diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -2440,7 +2440,7 @@ PyCData_clear(CDataObject *self)
     assert(dict); /* Cannot be NULL for CDataObject instances */
     Py_CLEAR(self->b_objects);
     if ((self->b_needsfree)
-        && ((size_t)dict->size > sizeof(self->b_value)))
+        && (self->b_ptr != (char *)&self->b_value))
         PyMem_Free(self->b_ptr);
     self->b_ptr = NULL;
     Py_CLEAR(self->b_base);

I need to think about that more, though.
msg177113 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-12-07 19:20
See patch in issue 16628.
History
Date User Action Args
2012-12-07 19:20:41pitrousetstatus: open -> closed

nosy: + pitrou
messages: + msg177113

superseder: leak in ctypes.resize()
resolution: duplicate
2011-10-03 00:22:15meador.ingesetmessages: + msg144797
versions: + Python 2.7, Python 3.2
2011-10-02 22:58:35meador.ingesetmessages: + msg144791
2011-10-02 22:13:54amaury.forgeotdarcsetmessages: + msg144784
2011-10-02 15:12:12meador.ingesetnosy: + amaury.forgeotdarc, belopolsky, meador.inge
components: + ctypes
2011-10-02 09:11:57skrahcreate