This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients vstinner
Date 2013-08-28.23:53:10
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1377733991.41.0.484218634069.issue18874@psf.upfronthosting.co.za>
In-reply-to
Content
Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number.

I implemented the module as a third party module for Python 2.5-3.4, but it requires to patch and recompile Python:
https://pypi.python.org/pypi/pytracemalloc


My proposed implementation is different:

* use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project) instead of depending on the glib library ; I simplified and adapted the implementation for my usage

* no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable

* traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table

I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization.


TODO:

* The (high-level) API should be reviewed and discussed

* The documention should be improved

* PyMem_Raw API is not hooked yet because the code is not thread-safe (it relies on the Python GIL)

* Filenames should not be encoded: the hash table should directly use Unicode strings (PyObject*), encoding a filename allocates memory and may call the garbage collector

* Memory is not released at exit


For the documentation, see the following page:
https://pypi.python.org/pypi/pytracemalloc


See also the https://bitbucket.org/haypo/pyfailmalloc project and the issue #18408.
History
Date User Action Args
2013-08-28 23:53:11vstinnersetrecipients: + vstinner
2013-08-28 23:53:11vstinnersetmessageid: <1377733991.41.0.484218634069.issue18874@psf.upfronthosting.co.za>
2013-08-28 23:53:11vstinnerlinkissue18874 messages
2013-08-28 23:53:10vstinnercreate