Author vstinner
Recipients Paul Monson, lukasz.langa, petr.viktorin, serge-sans-paille, steve.dower, vstinner
Date 2019-08-30.10:16:31
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1567160192.66.0.192510927226.issue37140@roundup.psfhosted.org>
In-reply-to
Content
Ok, it took me a while to understand the subtle ctypes internals. serge-sans-paille and me read the C code and run a debugger (gdb) on it to undertand how passing a structure by copy work in ctypes.

The root function is StructUnionType_paramfunc(): it copies the structure if it's larger than sizeof(void*) bytes.

StructUnionType_paramfunc() creates a new internal object which is stored into parg->obj. The only purpose of this object is to release the memory copy allocated by PyMem_Malloc().

This issue is a problem with the internal object: if the struture has a finalizer, the finalizer is called twice. First, it is called on the internal object which is more and less a full copy of the structure. Second, it is called on the structure (once the last reference to the structure is removed).

The code behaves as if the the finalizer is called twice. Even if it's two separated Python object, in fact the two objects contain the same structure values. For example, if the structure contains a pointer to memory block and the finalizer calls free(ptr): free(ptr) will be called twice with the same ptr value.

This surprising behavior comes from this code:

        void *new_ptr = PyMem_Malloc(self->b_size);
        if (new_ptr == NULL)
            return NULL;
        memcpy(new_ptr, self->b_ptr, self->b_size);
        copied_self = (CDataObject *)PyCData_AtAddress(
            (PyObject *)Py_TYPE(self), new_ptr);
        copied_self->b_needsfree = 1;

copied_self reuses the exact same type of the structure. If the structure has a finalizer defined in Python, it will be called.

copied_self finalized is called at the "cleanup:" label of  _ctypes_callproc():

    for (i = 0; i < argcount; ++i)
        Py_XDECREF(args[i].keep);

--

Trying to duplicate self isn't needed. All we need is to call PyMem_Free(ptr) at the _ctypes_callproc() exit.

My PR 15612 introduces a new internal StructParam_Type type which has exactly one purpose: call PyMem_Free(ptr) in its deallocator. The type has no finalizer.
History
Date User Action Args
2019-08-30 10:16:32vstinnersetrecipients: + vstinner, petr.viktorin, lukasz.langa, steve.dower, serge-sans-paille, Paul Monson
2019-08-30 10:16:32vstinnersetmessageid: <1567160192.66.0.192510927226.issue37140@roundup.psfhosted.org>
2019-08-30 10:16:32vstinnerlinkissue37140 messages
2019-08-30 10:16:31vstinnercreate