Author njs
Recipients njs, pitrou, vstinner
Date 2016-03-12.01:28:50
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1457746131.59.0.210887032762.issue26530@psf.upfronthosting.co.za>
In-reply-to
Content
> Note: I don't understand what are domain, tag (address of the memory block?) or quantity (size of the memory block?).

Yes, the idea is basically the same as now, except with an extra const char * specifying the "domain", so that one could keep separate track of different kinds of allocations. So PyMem_Malloc would just call PyMem_RecordAlloc("heap", ptr, size) (or act equivalently to something that called that, etc.), but something like PyCuda might do PyMem_RecordAlloc("gpu", ptr, size) to track allocations in GPU memory. All the tracing stuff in tracemalloc would be awesome to have for GPU allocations, and it would hardly require any changes to enable it. Ditto for other leakable resources like file descriptors or shmem segments.

> I don't plan to store new information in tracemalloc. tracemalloc design is "simple": it stores a trace for each memory block. A memory block is identified by its address (void* pointer) and a trace is a tuple (size, traceback).
> Extending a trace would increase the memory footprint overhead of tracemalloc.

I think the extra footprint would be tiny? Logically, you'd index traces by (domain, pointer) instead of (pointer), but since we expect that the number of distinct domains is small (like... 1, in the common case) then the natural implementation strategy would be to replace the current {pointer: trace} mapping with a mapping {domain: {pointer: trace}}. So the total overhead would be basically the overhead of allocating a single additional dict.

That said, I think having the domain argument *in the public hookable API* is useful even if the tracemalloc hooks themselves decide to ignore it or just ignore all calls that don't refer to the "heap" domain -- because it's extremely cheap to put this in the API now while we're adding it, and it leaves the door open for someone else to come along later and make an awesome gpu heap tracker or whatever without having to first argue with python-dev and wait for the cpython release cycle.
History
Date User Action Args
2016-03-12 01:28:51njssetrecipients: + njs, pitrou, vstinner
2016-03-12 01:28:51njssetmessageid: <1457746131.59.0.210887032762.issue26530@psf.upfronthosting.co.za>
2016-03-12 01:28:51njslinkissue26530 messages
2016-03-12 01:28:50njscreate