Author kristjan.jonsson
Recipients Rhamphoryncus, amaury.forgeotdarc, barry, gregory.p.smith, jlaurila, jszakmeister, kristjan.jonsson, ncoghlan, neilo, pitrou, pjmcnerney, rhettinger, tlesher, vstinner
Date 2013-03-11.10:19:59
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1362997200.3.0.26943229459.issue3329@psf.upfronthosting.co.za>
In-reply-to
Content
At ccp we have something similar.  We are embedding python in the UnrealEngine on the PS3 and need to get everything through their allocators.  For the purpose of flexibility, we added an api similar to the OPs, but more flexible:

/* Support for custom allocators */
typedef void *(*PyCCP_Malloc_t)(size_t size, void *arg, const char *file, int line, const char *msg);
typedef void *(*PyCCP_Realloc_t)(void *ptr, size_t size, void *arg, const char *file, int line, const char *msg);
typedef void (*PyCCP_Free_t)(void *ptr, void *arg, const char *file, int line, const char *msg);
typedef size_t (*PyCCP_Msize_t)(void *ptr, void *arg);
typedef struct PyCCP_CustomAllocator_t
{
    PyCCP_Malloc_t  pMalloc;
    PyCCP_Realloc_t pRealloc;
    PyCCP_Free_t    pFree;
    PyCCP_Msize_t   pMsize;    /* can be NULL, or return -1 if no size info is avail. */
    void            *arg;      /* opaque argument for the functions */
} PyCCP_CustomAllocator_t;

/* To set an allocator!  use 0 for the regular allocator, 1 for the block allocator.
 * pass a null pointer to reset to internal default
 */
PyAPI_FUNC(void) PyCCP_SetAllocator(int which, const PyCCP_CustomAllocator_t *);

For a module to install itself as a "hook" at runtime, this approach can be extended by querying the current allocator, so that such a hook can the delegate the previous calls.

The "block" allocator here, is intended as the underlying allocator to be used by obmalloc.c.  Depending on platforms, this can then allocate aligned virtual memory directly, which is more efficient than layering that on-top of a malloc-like allocator.

There are areas in cPython that use malloc() directly.  Those are actually not needed in all cases, but to cope with them we change them all to new RAW api calls (using preprocessor macros).
Essentially, malloc() maps to PyCCP_RawMalloc() or PyMem_MALLOC_INNER() (both local additions) based on whether the particular site using malloc() requires truly gil free malloc or not.

For this reason, the custom allocators mentioned canot be assumed to be called with the GIL.  However, it is easily possible to extend the system above so that there is a GIL and non-GIL version for the 'regular' allocator.

I'll put details of the stuff we have done for EVE Online / Dust 514 on my blog.  It is this, but much much more too.

Hopefully we can arrive at a way to abstract memory allocation away from Python in a flexible and extendible manner :)
History
Date User Action Args
2013-03-11 10:20:00kristjan.jonssonsetrecipients: + kristjan.jonsson, barry, rhettinger, gregory.p.smith, amaury.forgeotdarc, ncoghlan, Rhamphoryncus, pitrou, vstinner, jszakmeister, tlesher, jlaurila, neilo, pjmcnerney
2013-03-11 10:20:00kristjan.jonssonsetmessageid: <1362997200.3.0.26943229459.issue3329@psf.upfronthosting.co.za>
2013-03-11 10:20:00kristjan.jonssonlinkissue3329 messages
2013-03-11 10:19:59kristjan.jonssoncreate