classification
Title: Add sys._debugmallocstats()
Type: enhancement Stage: resolved
Components: Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: dmalcolm, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2012-05-11 22:40 by dmalcolm, last changed 2012-06-22 21:02 by dmalcolm. This issue is now closed.

Files
File name Uploaded Description Edit
add-debug-malloc-stats.patch dmalcolm, 2012-05-11 22:40 review
add-debug-malloc-stats-v2.patch dmalcolm, 2012-05-14 16:15 review
add-debug-malloc-stats-v3.patch dmalcolm, 2012-05-14 18:45 review
Messages (6)
msg160459 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2012-05-11 22:40
I'm attaching a patch which generalizes the at-exit PYTHONMALLOCSTATS memory usage report, so that it's available in a regular build and can be triggered from Python, by calling:
   sys._debugmallocstats()

This can be useful when debugging memory usage issues: a script can call the debug hook before and after certain activities, and the before/after amounts can be compared.

The patch moves this arena-accouting code when a new arena is allocated:
     ++ntimes_arena_allocated;
     if (narenas_currently_allocated > narenas_highwater)
         narenas_highwater = narenas_currently_allocated;
from out of the #ifdef PYMALLOC_DEBUG guard and into all PYMALLOC configurations, so that this data is available within the dump.  Given how much activity happens when a new arena is allocated, I believe this doesn't impact performance.

It changes _PyObject_DebugMallocStats() to take an arbitrary FILE*, rather than assuming stderr (which was handy for my original use-case of debugging a web server).  This function is already marked with PyAPI_FUNC() but not documented (albeit only present in a debug build).

Tested with --with-pymalloc, --without-pymalloc, and --with-pydebug

FWIW, Red Hat has been using a version of this patch in RHEL 5 as of RHEL 5.6 (http://rhn.redhat.com/errata/RHSA-2011-0027.html), and also in Fedora since September 2011 with python-2.7.2-15 and python3-3.2.2-6 (for the forthcoming Fedora 17).
msg160480 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-12 16:45
Nice idea. I don't see any obvious problem with the patch, except that the test should reuse test.script_helper.assert_python_ok().
msg160635 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2012-05-14 16:15
Updated version of the patch, using test.script_helper.assert_python_ok() and adding a NEWS entry
msg160640 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-14 17:10
One other nit: the C API functions shouldn't be included in the limited API.
msg160654 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2012-05-14 18:45
Thanks.  I'm attaching an updated version of the patch, wrapping all new C entrypoints within a #ifndef Py_LIMITED_API

I also moved the existing _PyObject_DebugMallocStats() entrypoint to within a #ifndef Py_LIMITED_API.  As noted above, it is not documented (and these patches change it from accepting void to a FILE*).
msg163459 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-22 18:56
New changeset d63a80abfbec by David Malcolm in branch 'default':
Issue #14785: Add sys._debugmallocstats() to help debug low-level memory allocation issues
http://hg.python.org/cpython/rev/d63a80abfbec
History
Date User Action Args
2012-06-22 21:02:57dmalcolmsetkeywords: - needs review
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2012-06-22 18:56:57python-devsetnosy: + python-dev
messages: + msg163459
2012-05-14 18:45:55dmalcolmsetfiles: + add-debug-malloc-stats-v3.patch

messages: + msg160654
2012-05-14 17:10:53pitrousetmessages: + msg160640
2012-05-14 16:15:26dmalcolmsetfiles: + add-debug-malloc-stats-v2.patch

messages: + msg160635
2012-05-12 16:45:52pitrousetnosy: + pitrou
messages: + msg160480
2012-05-11 22:40:30dmalcolmcreate