classification
Title: PyMem_RawMalloc(): update also sys.getallocatedblocks() in debug mode
Type: enhancement Stage:
Components: Versions: Python 3.6
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: haypo, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-04-26 11:35 by haypo, last changed 2016-05-20 09:46 by haypo. This issue is now closed.

Files
File name Uploaded Description Edit
pymem_rawmalloc_blocks.patch haypo, 2016-04-26 11:35 review
Messages (3)
msg264250 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-04-26 11:35
I modified PyMem_Malloc() to use the pymalloc allocator in the issue #26249. This change helped to find a memory leak in test_format that I introduced in Python 3.6:
http://bugs.python.org/issue26249#msg264174

This memory leak gave me an idea: PyMem_RawMalloc() should also update sys.getallocatedblocks() (number of currently allocated blocks). It would help to find memory leaks using "python -m test -R 3:3" in extension modules using PyMem_RawMalloc() (and not PyMem_Malloc() or PyObject_Malloc()).

Attached patch uses an atomic variable _Py_AllocatedBlocks, but only in debug mode. I chose to only change the variable in debug mode to:

* not impact performances
* I don't know if atomic variables are well supported (especially the "var++" operation)
* I don't know yet the impact of this change (how sys.getallocatedblocks() is used).

(The patch would be simpler if the release mode would also be impacted.)

The patch changes _PyObject_Alloc() and _PyObject_Free() in debug mode to only account allocations directly made by pymalloc, to let PyMem_RawMalloc() and PyMem_RawFree() update the _Py_AllocatedBlocks variable. In release mode, _PyObject_Alloc() and _PyObject_Free() are responsible to update the _Py_AllocatedBlocks variable for allocations delegated to PyMem_RawMalloc() and PyMem_RawFree().
msg264268 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-04-26 12:23
The expected side effect of this change is that hunting memory leaks in regrtest (python -m test) may get more noise.

I ran the Python test suite using "./python -m test -R 3:3": test_nntplib, test_tools and test_unittest failed.

Run alone, test_tools still fails:
-------------
$ ./python -m test -R 3:3 test_tools
0:00:00 [1/1] test_tools
beginning 6 repetitions
123456
......
test_tools leaked [0, 5, 20] references, sum=25
test_tools leaked [0, 1, 4] memory blocks, sum=5
test_tools took 44 sec
1 test failed:
    test_tools
Total duration: 0:00:45
-------------

But if I run test_tools one more time, it doesn't fail anymore...

Example of two sequential runs using two processes:
-------------
$ ./python -m test -R 3:3 test_tools
Run tests sequentially
0:00:00 [1/1] test_tools
beginning 6 repetitions
123456
......
test_tools took 44 sec
1 test OK.
Total duration: 0:00:44

$ ./python -m test -R 3:3 test_tools
Run tests sequentially
0:00:00 [1/1] test_tools
beginning 6 repetitions
123456
......
test_tools leaked [2, 0, 10] references, sum=12
test_tools leaked [0, 0, 3] memory blocks, sum=3
test_tools took 43 sec
1 test failed:
    test_tools
Total duration: 0:00:44
-------------

test_nntplib doesn't fail anymore when run alone.

Oh, and test_unittest failure doesn't seem related to my change: "./python -m test -R 3:3 test_unittest" already fails without my change.
msg265927 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-05-20 09:46
I'm not convinced myself by this change, it may make detection of memory leaks much more tricky for a little gain. I prefer to reject my own issue ;-)
History
Date User Action Args
2016-05-20 09:46:46hayposetstatus: open -> closed
resolution: rejected
messages: + msg265927
2016-04-26 12:23:32hayposetmessages: + msg264268
2016-04-26 11:35:23haypocreate