Title: Enhance Python's Memory Instrumentation with COUNT_ALLOCS
Type: enhancement Stage: patch review
Components: Versions: Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eelizondo, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2018-03-12 19:31 by eelizondo, last changed 2018-03-12 22:05 by eelizondo.

Pull Requests
URL Status Linked Edit
PR 6091 closed eelizondo, 2018-03-12 22:05
Messages (4)
msg313684 - (view) Author: Eddie Elizondo (eelizondo) * Date: 2018-03-12 19:33
msg313685 - (view) Author: Eddie Elizondo (eelizondo) * Date: 2018-03-12 19:47
Currently, Python has very few instrumentation when it comes to the types of allocations that it's doing. For instance, we currently can't identify how many times an object uses free lists vs actual mallocs.

Currently, there's a special build which can be used by compiling with "-DCOUNT_ALLOCS". However, this build is not ABI compatible with extensions. Meaning you have to recompile all the modules that are used. Doing this on a large scale scenario (1000+ modules) is not feasible.

Thus, I propose the following enhancements:
* Make COUNT_ALLOCS ABI compatible

* Expand the counters to not only track allocs and frees but also distinguish between:
* GC Counts: Allocs/Frees coming from PyObject_GC_Malloc PyObject_GC_Del
* Memory Counts: Allocs/Frees coming from PyMem_Malloc/PyObject_Malloc PyObject_Free (modulo GC objects).
* Free Lists: "Allocs/Frees" coming from specific object's free_lists/caches.
* Misc: Extra kinds of "Allocs/Frees" such as code's zombie frame.
msg313686 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-12 19:56
Could tracemalloc help you?
msg313691 - (view) Author: Eddie Elizondo (eelizondo) * Date: 2018-03-12 20:13
@serhiy.storchaka tracemalloc can't distinguish between the usage of gc allocs, normal mallocs, and free list reuse.
Date User Action Args
2018-03-12 22:05:43eelizondosetkeywords: + patch
stage: patch review
pull_requests: + pull_request5852
2018-03-12 20:13:53eelizondosetmessages: + msg313691
2018-03-12 19:56:07serhiy.storchakasetnosy: + vstinner, serhiy.storchaka
messages: + msg313686
2018-03-12 19:47:08eelizondosetmessages: + msg313685
2018-03-12 19:33:55eelizondosetmessages: + msg313684
title: [WIP] Enhance Python's Memory Instrumentation with COUNT_ALLCOS -> Enhance Python's Memory Instrumentation with COUNT_ALLOCS
2018-03-12 19:33:31eelizondosettitle: [WIP] Enhancing Python's COUNT_ALLOCS -> [WIP] Enhance Python's Memory Instrumentation with COUNT_ALLCOS
2018-03-12 19:31:45eelizondosettitle: Enhancing Python -> [WIP] Enhancing Python's COUNT_ALLOCS
2018-03-12 19:31:15eelizondocreate