This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author scoder
Recipients scoder
Date 2015-06-19.06:31:17
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1434695478.8.0.352869011826.issue24469@psf.upfronthosting.co.za>
In-reply-to
Content
A Cython user noticed a memory leak when C-inheriting from "int".

http://thread.gmane.org/gmane.comp.python.cython.devel/15689

The Cython code to reproduce this is simply this:

    cdef class ExtendedInt(int): pass
 
    for j in xrange(10000000):
        ExtendedInt(j)

The problem is due to the free-list of the int type. It uses this code for deallocation:

"""
static void
int_dealloc(PyIntObject *v)
{
    if (PyInt_CheckExact(v)) {
        Py_TYPE(v) = (struct _typeobject *)free_list;
        free_list = v;
    }
    else
        Py_TYPE(v)->tp_free((PyObject *)v);
}

static void
int_free(PyIntObject *v)
{
    Py_TYPE(v) = (struct _typeobject *)free_list;
    free_list = v;
}
"""

Now, when C-inheriting from PyInt_Type without providing an own tp_free implementation, PyType_Ready() will inherit the supertype's tp_free slot, which means that int_dealloc() above will end up calling int_free() in all cases, not only for the exact-int case. Thus, whether or not it's exactly "int" or a subtype, the object will always be added to the free-list on deallocation.

However, in the subtype case, the free-list is actually ignored by int_new() and int_subtype_new(), so that as long as the user code only creates tons of int subtype instances and not plain int instances, the freelist will keep growing without bounds by pushing dead subtype objects onto it that are never reused.

There are two problems here:

1) int subtypes should not be added to the freelist, or at least not unless they have exactly the same struct size as PyIntObject (which the ExtendedInt type above does but other subtypes may not)

2) if a free-list is used, it should be used in all instantiation cases, not just in PyInt_FromLong().

Fixing 1) by adding a type check to int_free() would be enough to fix the overall problem. Here's a quickly hacked up change that seems to work for me:

"""
static void
int_free(PyIntObject *v)
{
    if (PyInt_CheckExact(v)) {
        Py_TYPE(v) = (struct _typeobject *)free_list;
        free_list = v;
    }
    else if (Py_TYPE(v)->tp_flags & Py_TPFLAGS_HAVE_GC)
        PyObject_GC_Del(v);  // untested by probably necessary
    else
        PyObject_Del(v);
}
"""
History
Date User Action Args
2015-06-19 06:31:18scodersetrecipients: + scoder
2015-06-19 06:31:18scodersetmessageid: <1434695478.8.0.352869011826.issue24469@psf.upfronthosting.co.za>
2015-06-19 06:31:18scoderlinkissue24469 messages
2015-06-19 06:31:17scodercreate