Issue26850
Created on 2016-04-26 11:35 by vstinner, last changed 2016-05-20 09:46 by vstinner. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
pymem_rawmalloc_blocks.patch | vstinner, 2016-04-26 11:35 | review |
Messages (3) | |||
---|---|---|---|
msg264250 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2016-04-26 11:35 | |
I modified PyMem_Malloc() to use the pymalloc allocator in the issue #26249. This change helped to find a memory leak in test_format that I introduced in Python 3.6: http://bugs.python.org/issue26249#msg264174 This memory leak gave me an idea: PyMem_RawMalloc() should also update sys.getallocatedblocks() (number of currently allocated blocks). It would help to find memory leaks using "python -m test -R 3:3" in extension modules using PyMem_RawMalloc() (and not PyMem_Malloc() or PyObject_Malloc()). Attached patch uses an atomic variable _Py_AllocatedBlocks, but only in debug mode. I chose to only change the variable in debug mode to: * not impact performances * I don't know if atomic variables are well supported (especially the "var++" operation) * I don't know yet the impact of this change (how sys.getallocatedblocks() is used). (The patch would be simpler if the release mode would also be impacted.) The patch changes _PyObject_Alloc() and _PyObject_Free() in debug mode to only account allocations directly made by pymalloc, to let PyMem_RawMalloc() and PyMem_RawFree() update the _Py_AllocatedBlocks variable. In release mode, _PyObject_Alloc() and _PyObject_Free() are responsible to update the _Py_AllocatedBlocks variable for allocations delegated to PyMem_RawMalloc() and PyMem_RawFree(). |
|||
msg264268 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2016-04-26 12:23 | |
The expected side effect of this change is that hunting memory leaks in regrtest (python -m test) may get more noise. I ran the Python test suite using "./python -m test -R 3:3": test_nntplib, test_tools and test_unittest failed. Run alone, test_tools still fails: ------------- $ ./python -m test -R 3:3 test_tools 0:00:00 [1/1] test_tools beginning 6 repetitions 123456 ...... test_tools leaked [0, 5, 20] references, sum=25 test_tools leaked [0, 1, 4] memory blocks, sum=5 test_tools took 44 sec 1 test failed: test_tools Total duration: 0:00:45 ------------- But if I run test_tools one more time, it doesn't fail anymore... Example of two sequential runs using two processes: ------------- $ ./python -m test -R 3:3 test_tools Run tests sequentially 0:00:00 [1/1] test_tools beginning 6 repetitions 123456 ...... test_tools took 44 sec 1 test OK. Total duration: 0:00:44 $ ./python -m test -R 3:3 test_tools Run tests sequentially 0:00:00 [1/1] test_tools beginning 6 repetitions 123456 ...... test_tools leaked [2, 0, 10] references, sum=12 test_tools leaked [0, 0, 3] memory blocks, sum=3 test_tools took 43 sec 1 test failed: test_tools Total duration: 0:00:44 ------------- test_nntplib doesn't fail anymore when run alone. Oh, and test_unittest failure doesn't seem related to my change: "./python -m test -R 3:3 test_unittest" already fails without my change. |
|||
msg265927 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2016-05-20 09:46 | |
I'm not convinced myself by this change, it may make detection of memory leaks much more tricky for a little gain. I prefer to reject my own issue ;-) |
History | |||
---|---|---|---|
Date | User | Action | Args |
2016-05-20 09:46:46 | vstinner | set | status: open -> closed resolution: rejected messages: + msg265927 |
2016-04-26 12:23:32 | vstinner | set | messages: + msg264268 |
2016-04-26 11:35:23 | vstinner | create |