Python implements debug hooks on Python memory allocators:
https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks

Problem: buffer understand and buffer overflow are only detected when a memory block is released, which may occur far from the code responsible to create the buffer.

Attached patch dumps the traceback where a memory block was allocated on error in malloc debug hooks. The feature requires to enable tracemalloc to record tracebacks.

test.py used in below example:
----

import _testcapi

def f():
    _testcapi.pymem_buffer_overflow()

f()

Example:
---
haypo@selma$ ./python -X tracemalloc=5 test.py
Debug memory block at address p=0x22e8be0: API 'm'
16 bytes originally requested
The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
The 8 pad bytes at tail=0x22e8bf0 are not all FORBIDDENBYTE (0xfb):
at tail+0: 0x78 *** OUCH
...
The block was made by call bpo-37240 to debug malloc/realloc.
Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb

Memory block traceback (most recent call first):
File "test.py", line 4
File "test.py", line 6

Fatal Python error: bad trailing pad byte

Current thread 0x00007f4a93d9c700 (most recent call first):
File "test.py", line 4 in f
File "test.py", line 6 in <module>
Abandon (core dumped)
---

The patch adds the "Memory block traceback" section.

See also the issue bpo-26567: "Use tracemalloc to display the traceback where an object was allocated when a ResourceWarning is emitted".

I reviewed my own patch on Rietveld :-)

New changeset 18a19e62bac5 by Victor Stinner in branch 'default':
Enhance and rewrite traceback dump C functions
https://hg.python.org/cpython/rev/18a19e62bac5

New changeset fea3c6e9a38e by Victor Stinner in branch '3.5':
_tracemalloc: store lineno as unsigned int
https://hg.python.org/cpython/rev/fea3c6e9a38e

New changeset cef6a32d805f by Victor Stinner in branch 'default':
On memory error, dump the memory block traceback
https://hg.python.org/cpython/rev/cef6a32d805f

New changeset 8215dae7ec3c by Victor Stinner in branch 'default':
Oops, revert unwanted change used to create an example
https://hg.python.org/cpython/rev/8215dae7ec3c

New changeset 769dfcb701ee by Victor Stinner in branch 'default':
Issue bpo-26564: Fix test_capi
https://hg.python.org/cpython/rev/769dfcb701ee

FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

..\Python\traceback.c(513): error C2057: expected constant expression [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2466: cannot allocate an array of constant size 0 [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]
..\Python\traceback.c(513): error C2133: 'buffer': unknown size [D:\buildarea\3.x.bolen-windows10\build\PCbuild\pythoncore.vcxproj]

New changeset 5f2284ecf9c6 by Victor Stinner in branch 'default':
Fix compilation error of traceback.c on Windows
https://hg.python.org/cpython/rev/5f2284ecf9c6

Martin Panter added the comment:

FYI Victor if you haven’t already noticed, the Windows buildbots are choking on a variable-length array:

Thank you, sometimes I miss buildbot failures.

I pushed changes which had 2 tests, so I knew that all buildbots were
red, and I planned to come back the day after to check.

For Windows, it's a pity, the expression is constant. Stupid compiler
:-) VS2015 doesn't support C99 yet? Anyway, it's trivial to fix the
code. I just fixed it.

I will check again if buildbots are now happy with the current code ;-)

Ok, buildbots are green again. I close the issue.

New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774

I'm reopening this to commit a fix for a comment which became obsolete following this change: https://hg.python.org/cpython/rev/18a19e62bac5.

New changeset 40125ab by Irit Katriel in branch 'master':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
40125ab

New changeset cecbaa3 by Miss Islington (bot) in branch '3.8':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
cecbaa3

New changeset 1c70d40 by Miss Islington (bot) in branch '3.9':
bpo-26564: fix obsolete comment in traceback.c (GH-23819)
1c70d40

I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.

I do not propose to change anything. It is just curiosity.

I am wondering why 53/22? 5/2 is as good for 8-bytes long -- both gives 21-bytes buffer.

53/22 is closer to math.log(256)/math.log(10) than 5/2, and 8*53 should not overflow :-)