classification
Title: Malloc debug hooks: display memory block traceback on error
Type: enhancement Stage:
Components: Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo, martin.panter, python-dev
Priority: normal Keywords: patch

Created on 2016-03-15 00:39 by haypo, last changed 2016-03-18 10:04 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
pymem_tb.patch haypo, 2016-03-15 00:39 review
Messages (12)
msg261793 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-03-15 00:39
Python implements debug hooks on Python memory allocators:
https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks

Problem: buffer understand and buffer overflow are only detected when a memory block is released, which may occur far from the code responsible to create the buffer.

Attached patch dumps the traceback where a memory block was allocated on error in malloc debug hooks. The feature requires to enable tracemalloc to record tracebacks.


test.py used in below example:
----
import _testcapi

def f():
    _testcapi.pymem_buffer_overflow()

f()
----


Example:
---
haypo@selma$ ./python -X tracemalloc=5 test.py 
Debug memory block at address p=0x22e8be0: API 'm'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x22e8bf0 are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x78 *** OUCH
        ...
    The block was made by call #37240 to debug malloc/realloc.
    Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb

Memory block traceback (most recent call first):
  File "test.py", line 4
  File "test.py", line 6

Fatal Python error: bad trailing pad byte

Current thread 0x00007f4a93d9c700 (most recent call first):
  File "test.py", line 4 in f
  File "test.py", line 6 in <module>
Abandon (core dumped)
---

The patch adds the "Memory block traceback" section.
msg261813 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-03-15 13:17
See also the issue #26567: "Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted".
msg261816 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-03-15 14:34
I reviewed my own patch on Rietveld :-)
msg261827 - (view) Author: Roundup Robot (python-dev) Date: 2016-03-15 20:58
New changeset 18a19e62bac5 by Victor Stinner in branch 'default':
Enhance and rewrite traceback dump C functions
https://hg.python.org/cpython/rev/18a19e62bac5

New changeset fea3c6e9a38e by Victor Stinner in branch '3.5':
_tracemalloc: store lineno as unsigned int
https://hg.python.org/cpython/rev/fea3c6e9a38e
msg261828 - (view) Author: Roundup Robot (python-dev) Date: 2016-03-15 21:42
New changeset cef6a32d805f by Victor Stinner in branch 'default':
On memory error, dump the memory block traceback
https://hg.python.org/cpython/rev/cef6a32d805f
msg261829 - (view) Author: Roundup Robot (python-dev) Date: 2016-03-15 21:50
New changeset 8215dae7ec3c by Victor Stinner in branch 'default':
Oops, revert unwanted change used to create an example
https://hg.python.org/cpython/rev/8215dae7ec3c
msg261830 - (view) Author: Roundup Robot (python-dev) Date: 2016-03-15 22:47
New changeset 769dfcb701ee by Victor Stinner in branch 'default':
Issue #26564: Fix test_capi
https://hg.python.org/cpython/rev/769dfcb701ee
msg261841 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-16 07:18
FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

..\Python\traceback.c(513): error C2057: expected constant expression [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2466: cannot allocate an array of constant size 0 [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2133: 'buffer': unknown size [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
msg261845 - (view) Author: Roundup Robot (python-dev) Date: 2016-03-16 08:46
New changeset 5f2284ecf9c6 by Victor Stinner in branch 'default':
Fix compilation error of traceback.c on Windows
https://hg.python.org/cpython/rev/5f2284ecf9c6
msg261846 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-03-16 08:51
Martin Panter added the comment:
> FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

Thank you, sometimes I miss buildbot failures.

I pushed changes which had 2 tests, so I knew that all buildbots were
red, and I planned to come back the day after to check.

For Windows, it's a pity, the expression is constant. Stupid compiler
:-) VS2015 doesn't support C99 yet? Anyway, it's trivial to fix the
code. I just fixed it.

I will check again if buildbots are now happy with the current code ;-)
msg261849 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-03-16 10:55
Ok, buildbots are green again. I close the issue.
msg261956 - (view) Author: Roundup Robot (python-dev) Date: 2016-03-18 10:04
New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774
History
Date User Action Args
2016-03-18 10:04:45python-devsetmessages: + msg261956
2016-03-16 10:55:09hayposetstatus: open -> closed
resolution: fixed
messages: + msg261849
2016-03-16 08:51:31hayposetmessages: + msg261846
2016-03-16 08:46:06python-devsetmessages: + msg261845
2016-03-16 07:18:12martin.pantersetnosy: + martin.panter
messages: + msg261841
2016-03-15 22:47:30python-devsetmessages: + msg261830
2016-03-15 21:50:11python-devsetmessages: + msg261829
2016-03-15 21:42:24python-devsetmessages: + msg261828
2016-03-15 20:58:15python-devsetnosy: + python-dev
messages: + msg261827
2016-03-15 14:34:54hayposetmessages: + msg261816
2016-03-15 13:17:41hayposetmessages: + msg261813
2016-03-15 00:39:55haypocreate