classification
Title: Remove sys._debugmallocstats() function and PYTHONMALLOCSTATS environment variable
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.10
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, pablogsal, pitrou, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2020-05-26 14:48 by vstinner, last changed 2020-05-26 16:43 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20430 closed vstinner, 2020-05-26 14:53
Messages (7)
msg369987 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 14:48
I never used sys._debugmallocstats() function or PYTHONMALLOCSTATS environment variable, whereas I spend significant time on optimizing memory allocators and free lists. The output is written into stderr which is very convenient to process these data.

I only "re"-discovered recently this debug feature when writing PR 20247 "Make tuple free list per-interpreter": I had to update _PyTuple_DebugMallocStats().

Removing this debug feature reduces the maintenance burden.

Attached PR removes all code related to the sys._debugmallocstats() function and the PYTHONMALLOCSTATS environment variable.

If someone wants to hack Python memory allocators or free lists, I suggest to add temporary code to dump some stats. But I don't think that it's worth it to keep the feature for all uses in Python.

People who worry about the memory usage of their application don't have to go into this low level of details, but care more about memory peak, where the memory was allocated (ex: tracemalloc module), RSS memory, etc.
msg369989 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 14:54
INADA-san, Antoine, Serhiy, Pablo: did any of you used this feature recently? Is it useful for your hacks?
msg369995 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 15:01
"Recent" changes in builtin Python debug tools.

* In Python 3.9, I removed the "COUNT_ALLOCS" special build: bpo-39489.

* Debug build of Python 3.8 is now ABI compatible with release build: I disabled Py_TRACE_REFS macro by default in --with-pydebug build. There is a new opt-in --with-trace-refs option for configure. We have a dedicated buildbot to ensure that the feature continue to work.

* In Python 3.8, I modified the debug hooks on Python memory allocators to omit the serial number by default. It reduces the memory footprint when these hooks are used (1 size_t per memory block). Extract of Objects/obmalloc.c:

/* Uncomment this define to add the "serialno" field */
/* #define PYMEM_DEBUG_SERIALNO */
msg370007 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2020-05-26 15:56
I use it often when I investigate memory usage.
It provides some useful information even in release Python build. For example:

* Which size class is most allocated?
* How many block are allocated?
* Which size class have most free blocks? (e.g. Inner fragmentation)
* Combined with tracing application to stderr, which part of the application increase memory usage?
msg370008 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 15:57
> I use it often when I investigate memory usage.

Oh. If you use it, we should keep the feature :-)
msg370013 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-26 16:33
Please, don't remove these as I use them often to track down memory usage in a similar way as Inada-san.
msg370016 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 16:43
> Please, don't remove these as I use them often to track down memory usage in a similar way as Inada-san.

Ok, sure. I close my issue and I closed my PR.

Note: Maybe your advanced usage of statistics on memory allocators should be documented somewhere in https://devguide.python.org/.
History
Date User Action Args
2020-05-26 16:43:38vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg370016

stage: patch review -> resolved
2020-05-26 16:33:35pablogsalsetmessages: + msg370013
2020-05-26 15:57:58vstinnersetmessages: + msg370008
2020-05-26 15:56:46inada.naokisetmessages: + msg370007
2020-05-26 15:01:57vstinnersetmessages: + msg369995
2020-05-26 14:54:16vstinnersetnosy: + pitrou, inada.naoki, serhiy.storchaka, pablogsal
messages: + msg369989
2020-05-26 14:53:12vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19687
2020-05-26 14:48:52vstinnercreate