Attached patch:

- Add PYTHONMALLOC env var which accepts 4 values:

  * pymalloc: use pymalloc for PyObject_Malloc(), malloc for PyMem_Malloc() and PyMem_RawMalloc()
  * pymalloc_debug: pymalloc + debug hooks
  * malloc: use malloc for PyObject_Malloc(), PyMem_Malloc() and PyMem_RawMalloc()
  * malloc_debug: malloc + debug hooks

- Add support for debug hooks in release mode
- Add unit test for debug hooks in test_capi.py
- Add unit on misuse of memory allocators in test_capi.py

PYTHONMALLOC is used even if -E command line option is used, I prefer to keep the code simple. PYTHONMALLOC must be checked before the first call to any Python memory allocator, so it must occur very earlier. Well, it's simply the first instruction of main()...

My main use case is to be able to use debug hooks to detect:

* misuse of python memory allocators like object allocated with PyObject_Malloc() and freed with PyMem_Free()
* buffer overflow
* buffer underlow

The output of sys._debugmallocstats() contains different info:

* Stats of type free lists
* pymalloc stats
* number of calls to memory allocators

The available info depends on:

* pymalloc disabled at compilation with ./configure --without-pymalloc
* pymalloc disabled at runtime with PYTHONMALLOC=malloc or PYTHONMALLOC=malloc_debug
* debug hooks adds the "times object malloc called" counter: by default if Python is compiled with ./configure --with-pydebug, or enabled at runtime using PYTHONMALLOC=pymalloc_debug or PYTHONMALLOC=malloc_debug

Example with Python compiled in release mode.

By default, a buffer overflow is not detected. It may crash later, in a random place...

$ ./python -c 'import _testcapi; _testcapi.pymem_buffer_overflow()'


Enabling debug hooks detects the buffer overflow immediatly:


$ PYTHONMALLOC=pymalloc_debug ./python -c 'import _testcapi; _testcapi.pymem_buffer_overflow()'
Debug memory block at address p=0x1a7f490: API 'm'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x1a7f4a0 are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x78 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #35014 to debug malloc/realloc.
    Data at p: cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb cb
Fatal Python error: bad trailing pad byte

Current thread 0x00007fca30572700 (most recent call first):
  File "<string>", line 1 in <module>
Abandon (core dumped)

The motivation to support PYTHONMALLOC=malloc (always use malloc) and not just PYTHONMALLOC=debug (enable debug hooks) is to allow to use external memory debugger like Valgrind.

Valgrind doesn't like pymalloc (pymalloc raises false alarms), we had to a configuration option for it: ./configure --with-valgrind.

Using PYTHONMALLOC=malloc allows to use Valgrind on the system Python, it's easier to use. No need to recompile Python.

New changeset af2fc10f703b by Victor Stinner in branch '3.5':
Issue #26516: Enhance Python mem allocators doc
https://hg.python.org/cpython/rev/af2fc10f703b

More complete patch (version 2):

* Document PYTHONMALLOC environment variable
* Add PYTHONMALLOC=debug to install debug hooks without forcing a specific memory allocator (keep the default memory allocator)
* Fix sys._debugmallocstats() for PYTHONMALLOC=malloc: don't display pymalloc stats
* _testcapi.pymem_api_misuse(): more realistic code, use PyMem_Mallloc() + PyMem_RawFree(). This code works in release mode (since PyMem and PyMem_Raw use the same allocator: malloc/free), but fail with a fatal error with debug hooks (API misused).

When I wrote the PEP 445, I already proposed to add a new environment variable to install debug hooks on a Python compiled in release mode:

https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable

But the proposition was rejected because of the PEP 432, but 3 years later, this PEP is still stuck at the draft state. So I propose to move on.

Anyway, PYTHONMALLOC is really a corner case since it must be read *very early* (first instruction of main()).


Oh by the way, I forgot to explain the initial motivation for this change! It's the issue #26249: "Change PyMem_Malloc to use pymalloc allocator". With this change, applications misuing PyMem_Malloc API (ex: alloc memory with PyMem_Malloc(), free memory with free()) will start to crash. PYTHONMALLOC=debug should help these applications to detect bugs easier and get an obvious error message.

Patch 3:

- Ooops, I updated pymem_api_misuse(), but I forgot to update the related unit test. It's now fixed.

Oh, I forgot to update wmain() for Windows.

Patch 4 handles wmain() too. I added a few comments and reformatted some parts of the code.

New changeset aa280432e9c7 by Victor Stinner in branch 'default':
Add PYTHONMALLOC env var
https://hg.python.org/cpython/rev/aa280432e9c7

I reworked my patch before pushing it:

* it now respects -E and -I command line options
* I documented changes in Doc/, Misc/NEWS, What's New in Python 3.6, Misc/README.valgrind, etc.
* Debug hooks are now also installed when Python is compiled in debug mode without pymalloc
* PYTHONMALLOCSTATS is now ignored if the allocator if pymalloc is not used
* PYTHONMALLOC is ignored if it's an empty string

Note: I just checked PYTHONMALLOCSTATS=1 with -X tracemalloc. It looks like it works as expected. Tracemalloc hooks are unregistered before stats are displayed at exit, _Pymem_PymallocEnabled() returns 1 as expected.

test_capi fails on Windows:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7348/steps/test/logs/stdio

Probably a issue with newline characters.

New changeset 3c3df33f2655 by Victor Stinner in branch 'default':
Issue #26516: Fix test_capi on Windows
https://hg.python.org/cpython/rev/3c3df33f2655

New changeset d8d6ec1333e6 by Victor Stinner in branch 'default':
Issue #26516: Fix test_capi on 32-bit system
https://hg.python.org/cpython/rev/d8d6ec1333e6

New changeset 7534eb7bd57e by Victor Stinner in branch 'default':
Issue #26516: Fix test_capi on AIX
https://hg.python.org/cpython/rev/7534eb7bd57e

Buildbots are happy, I close the issue.

New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774

New changeset c4c14e34e528 by Victor Stinner in branch 'default':
Don't define _PyMem_PymallocEnabled() if pymalloc is disabled
https://hg.python.org/cpython/rev/c4c14e34e528