Message190937
py_setallocators-filename.patch: Here is a try to define an API providing the filename and line number of the C code. The Py_SetAllocators() API is unchanged:
PyAPI_FUNC(int) Py_SetAllocators(
char api,
void* (*malloc) (size_t size, void *user_data),
void* (*realloc) (void *ptr, size_t size, void *user_data),
void (*free) (void *ptr, void *user_data),
void *user_data
);
If Python is compiled with -DPYMEM_TRACE_MALLOC, user_data is not the last parameter passed to Py_SetAllocators() but a pointer to a _PyMem_Trace structure:
typedef struct {
void *data;
/* NULL and -1 when unknown */
const char *filename;
int lineno;
} _PyMem_Trace;
The problem is that the module using Py_SetAllocators() must be compiled differently depending on PYMEM_TRACE_MALLOC. Example from pytracemalloc, modified for this patch:
---
_PyMem_Trace *ctrace;
trace_api_t *api;
void *call_data;
void *ptr;
#ifdef PYMEM_TRACE_MALLOC
ctrace = (_PyMem_Trace *)data;
api = (trace_api_t *)ctrace->data;
ctrace->data = api->data;
call_data = data;
#else
ctrace = NULL;
api = (trace_api_t *)data;
call_data = api->data;
#endif
ptr = api->malloc(size, call_data);
...
---
I didn't like the "ctrace->data = api->data;" instruction: pytracemalloc modifies the input _PyMem_Trace structure.
pytracemalloc code is a little bit more complex, but "it works". pytracemalloc can reuse the filename and line number of the C module, or of the Python module. It can be configured at runtime. Example of output for the C module:
---
2013-06-11 00:36:30: Top 15 allocations per file and line (compared to 2013-06-11 00:36:25)
#1: Objects/dictobject.c:352: size=6 MiB (+4324 KiB), count=9818 (+7773), average=663 B
#2: Objects/unicodeobject.c:1085: size=6 MiB (+2987 KiB), count=61788 (+26197), average=111 B
#3: Objects/tupleobject.c:104: size=4054 KiB (+2176 KiB), count=44569 (+24316), average=93 B
#4: Objects/typeobject.c:770: size=2440 KiB (+1626 KiB), count=13906 (+10360), average=179 B
#5: Objects/bytesobject.c:107: size=2395 KiB (+1114 KiB), count=24846 (+11462), average=98 B
#6: Objects/funcobject.c:12: size=1709 KiB (+1103 KiB), count=11516 (+7431), average=152 B
#7: Objects/codeobject.c:117: size=1760 KiB (+871 KiB), count=11267 (+5578), average=160 B
#8: Objects/dictobject.c:399: size=784 KiB (+627 KiB), count=10040 (+8028), average=80 B
#9: Objects/listobject.c:159: size=420 KiB (+382 KiB), count=5386 (+4891), average=80 B
#10: Objects/frameobject.c:649: size=1705 KiB (+257 KiB), count=3374 (+505), average=517 B
#11: ???:?: size=388 KiB (+161 KiB), count=588 (+240), average=676 B
#12: Objects/weakrefobject.c:36: size=241 KiB (+138 KiB), count=2579 (+1482), average=96 B
#13: Objects/dictobject.c:420: size=135 KiB (+112 KiB), count=2031 (+1736), average=68 B
#14: Objects/classobject.c:59: size=109 KiB (+105 KiB), count=1400 (+1345), average=80 B
#15: Objects/unicodeobject.c:727: size=188 KiB (+86 KiB), count=1237 (+687), average=156 B
37 more: size=828 KiB (+315 KiB), count=8421 (+5281), average=100 B
Total Python memory: size=29 MiB (+16 MiB), count=212766 (+117312), average=145 B
Total process memory: size=68 MiB (+22 MiB) (ignore tracemalloc: 0 B)
---
I also had to modify the following GC functions to get more accurate information:
- _PyObject_GC_Malloc(size)
- _PyObject_GC_New(tp)
- _PyObject_GC_NewVar(tp, nitems)
- PyObject_GC_Del(op)
For example, PyTuple_New() calls PyObject_GC_NewVar() to allocate its memory. With my patch, you get "Objects/tupleobject.c:104" instead of a generic "Modules/gcmodule.c:1717". |
|
Date |
User |
Action |
Args |
2013-06-10 23:05:55 | vstinner | set | recipients:
+ vstinner, barry, rhettinger, gregory.p.smith, amaury.forgeotdarc, ncoghlan, Rhamphoryncus, pitrou, kristjan.jonsson, jszakmeister, tlesher, jlaurila, neilo, pjmcnerney |
2013-06-10 23:05:54 | vstinner | set | messageid: <1370905554.45.0.897236243062.issue3329@psf.upfronthosting.co.za> |
2013-06-10 23:05:54 | vstinner | link | issue3329 messages |
2013-06-10 23:05:53 | vstinner | create | |
|