Author vstinner
Recipients Rhamphoryncus, barry, jlaurila, jszakmeister, ncoghlan, neilo, pitrou, pjmcnerney, rhettinger, tlesher, vstinner
Date 2013-03-06.12:32:29
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1362573152.17.0.808649419069.issue3329@psf.upfronthosting.co.za>
In-reply-to
Content
I attached a patch that I wrote for Wyplay: py_setallocators.patch. The patch adds two functions:

PyAPI_FUNC(int) Py_GetAllocators(
    char api,
    void* (**malloc_p) (size_t),
    void* (**realloc_p) (void*, size_t),
    void (**free_p) (void*)
    );

PyAPI_FUNC(int) Py_SetAllocators(
    char api,
    void* (*malloc) (size_t),
    void* (*realloc) (void*, size_t),
    void (*free) (void*)
    );

Where api is one of these values:

 - PY_ALLOC_SYSTEM_API: the system API (malloc, realloc, free)
 - PY_ALLOC_MEM_API: the PyMem_Malloc() API
 - PY_ALLOC_OBJECT_API: the PyObject_Malloc() API

These functions are used by the pytracemalloc project to hook PyMem_Malloc() and PyObject_Malloc() API. pytracemalloc traces all Python memory allocations to compute statistics per Python file.
https://pypi.python.org/pypi/pytracemalloc

Wyplay is also using Py_SetAllocators() internally to replace completly system allocators *before* Python is started. We have another private patch on Python adding a function. This function sets its own memory allocators, it is called before the start of Python thanks to an "__attribute__((constructor))" attribute.

--

If you use Py_SetAllocators() to replace completly a memory allocator (any memory allocation API), you have to do it before the first Python memory allocation (before Py_Main()) *or* your memory allocator must be able to recognize if a pointer was not allocated by him and pass the operation (realloc or free) to the previous memory allocator.

For example, PyObject_Free() is able to recognize that a pointer is part of its memory pool, or fallback to the system allocator (extract of the original code):

    if (Py_ADDRESS_IN_RANGE(p, pool)) {
        ...
        return;
    }
    free(p);

--

If you use Py_SetAllocators() to hook memory allocators (do something before and/or after calling the previous function, *without* touching the pointer nor the size), you can do it anytime.

--

I didn't run a benchmark yet to measure the overhead of the patch on Python performances.

New functions are not documented nor tested yet. If we want to test these new functions, we can write a simple hook tracing calls to the memory allocators and call the memory allocator.
History
Date User Action Args
2013-03-06 12:32:32vstinnersetrecipients: + vstinner, barry, rhettinger, ncoghlan, Rhamphoryncus, pitrou, jszakmeister, tlesher, jlaurila, neilo, pjmcnerney
2013-03-06 12:32:32vstinnersetmessageid: <1362573152.17.0.808649419069.issue3329@psf.upfronthosting.co.za>
2013-03-06 12:32:32vstinnerlinkissue3329 messages
2013-03-06 12:32:31vstinnercreate