classification
Title: Test leaks of memory not managed by Python allocator
Type: enhancement Stage:
Components: Tests Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: erlendaasland, serhiy.storchaka, vstinner, xiang.zhang
Priority: normal Keywords:

Created on 2018-10-23 19:36 by serhiy.storchaka, last changed 2021-02-23 09:27 by erlendaasland.

Messages (7)
msg328336 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-23 19:36
Would be nice to add a possibility to test memory leaks if memory is allocated not by Python allocators, but inside external libraries. This would allow to catch leaks on the bridge between Python and external libraries. See for example issue34794.
msg328346 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-23 21:42
You can try to use Valgrind for that?
msg328520 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-26 08:09
Can it be used on buildbots and in CI tests?
msg328530 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-26 10:18
> Can it be used on buildbots and in CI tests?

These kind of tool produces a lot of false alarms :-( They are also hard to use.
msg328531 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-26 10:20
Before seeing how to automate it, first someone should try to detect the bpo-34794 memory leak manually ;-)

By the way, when I implemented the PEP 445, I tried to configure OpenSSL to use Python memory allocators (to benefit of tracemalloc), but the OpenSSL API for that didn't work at all: bpo-18227.
msg328532 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-26 10:27
About automation, regrtest -R checks memory leaks using sys.getallocatedblocks(). But this function only tracks PyMem_Malloc() and PyObject_Malloc() (which is now technically the same memory allocator, since Python 3.6: https://docs.python.org/dev/c-api/memory.html#default-memory-allocators )

I tried to track PyMem_RawMalloc() using regrtest but... the raw memory allocator is not really "deterministic", it's hard to get reliable behavior. See: bpo-26850.

Tracking memory leaks is an hard topic :-) I'm happy that I finally fixed tracemalloc to track properly objects in free lists: bpo-35053!
msg328533 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-26 10:39
It seems like the easiest thing to do thta would directly benefit (to tracemalloc users) is to continue the implementation of bpo-18227:

* _sqlite: call sqlite3_config(SQLITE_CONFIG_MALLOC, pMem) to use PyMem_RawMalloc()
* _ssl: try again CRYPTO_set_mem_functions()

Python modules already using Python memory allocators:

* zlib: "zst.zalloc = PyZlib_Malloc" which calls PyMem_RawMalloc
* _decimal: mpd_mallocfunc = PyMem_Malloc
* _lzma: "self->alloc.alloc = PyLzma_Malloc" which calls PyMem_RawMalloc
* pyexpat: XML_ParserCreate_MM(encoding, &ExpatMemoryHandler,...) with ExpatMemoryHandler = {PyObject_Malloc, PyObject_Realloc, PyObject_Free}
* _bz2: "bzalloc = BZ2_Malloc" which calls PyMem_RawMalloc()

Using Python memory allocators gives access to Python builtin "memory debugger", even in release mode using PYTHONMALLOC=debug or -X dev.
History
Date User Action Args
2021-02-23 09:27:18erlendaaslandsetnosy: + erlendaasland
2018-10-26 10:39:25vstinnersetmessages: + msg328533
2018-10-26 10:27:03vstinnersetmessages: + msg328532
2018-10-26 10:20:41vstinnersetmessages: + msg328531
2018-10-26 10:18:55vstinnersetmessages: + msg328530
2018-10-26 08:09:22serhiy.storchakasetmessages: + msg328520
2018-10-24 15:44:09xiang.zhangsetnosy: + xiang.zhang
2018-10-23 21:42:48vstinnersetmessages: + msg328346
2018-10-23 19:36:59serhiy.storchakacreate