Hi,

When trying to build CPython "tip" with AddressSanitizer enabled, I get the following crash,

make Parser/pgen
make[1]: Entering directory `/scratch/repos/cpython'
make[1]: `Parser/pgen' is up to date.
make[1]: Leaving directory `/scratch/repos/cpython'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
=================================================================
==1375== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60380000f020 at pc 0x40b260 bp 0x7fff6e6cbbf0 sp 0x7fff6e6cbbe8
READ of size 4 at 0x60380000f020 thread T0
    #0 0x40b25f in _PyObject_Realloc /scratch/repos/cpython/Objects/obmalloc.c:1551
    #1 0x403048 in PyNode_AddChild /scratch/repos/cpython/Parser/node.c:98
    #2 0x4037fe in push /scratch/repos/cpython/Parser/parser.c:126
    #3 0x4121bd in parsetok /scratch/repos/cpython/Parser/parsetok.c:211
    #4 0x413344 in PyParser_ParseFileFlags /scratch/repos/cpython/Parser/parsetok.c:86
    #5 0x413422 in getgrammar /scratch/repos/cpython/Parser/pgenmain.c:92
    #6 0x401860 in main /scratch/repos/cpython/Parser/pgenmain.c:55
    #7 0x3cb2221b74 in ?? ??:0
    #8 0x401994 in _start ??:?
0x60380000f020 is located 96 bytes to the left of 0-byte region [0x60380000f080,0x60380000f080)
==1375== AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_allocator2.cc:216 "((id)) != (0)" (0x0, 0x0)
    #0 0x3ab681237d in ?? ??:0
    #1 0x3ab68191b3 in ?? ??:0
    #2 0x3ab680563b in ?? ??:0
    #3 0x3ab6816e8f in ?? ??:0
    #4 0x3ab6817ff1 in ?? ??:0
    #5 0x3ab6812763 in ?? ??:0
    #6 0x40b25f in _PyObject_Realloc /scratch/repos/cpython/Objects/obmalloc.c:1551
    #7 0x403048 in PyNode_AddChild /scratch/repos/cpython/Parser/node.c:98
    #8 0x4037fe in push /scratch/repos/cpython/Parser/parser.c:126
    #9 0x4121bd in parsetok /scratch/repos/cpython/Parser/parsetok.c:211
    #10 0x413344 in PyParser_ParseFileFlags /scratch/repos/cpython/Parser/parsetok.c:86
    #11 0x413422 in getgrammar /scratch/repos/cpython/Parser/pgenmain.c:92
    #12 0x401860 in main /scratch/repos/cpython/Parser/pgenmain.c:55
    #13 0x3cb2221b74 in ?? ??:0
    #14 0x401994 in _start ??:?
make: *** [Include/graminit.h] Error 1

...

AddressSanitizer is a fast memory error detector. See http://clang.llvm.org/docs/AddressSanitizer.html for details.


Build options used
==================

export CFLAGS="-fsanitize=address -g"

export LDFLAGS="-fsanitize=address"

./configure --prefix=/opt/python --with-valgrind


Software Versions
==================

$ hg log

changeset:   84902:6e1dd1ce95b8
branch:      2.7
tag:         tip
parent:      84893:1dbcb0299088
user:        Terry Jan Reedy <tjreedy@udel.edu>
date:        Tue Jul 30 01:37:36 2013 -0400
summary:     Make all idle test case names end with 'Test'.

$ gcc --version
gcc (GCC) 4.8.1 20130603 (Red Hat 4.8.1-1)

$ cat /etc/redhat-release 
Fedora release 19 (Schrödinger’s Cat)

Fix / Work-around
=================

See http://code.google.com/p/address-sanitizer/wiki/AddressSanitizer#Turning_off_instrumentation

Turning off the instrumentation for _PyObject_Free and  _PyObject_Realloc is required.

Advantages
==========

AddressSanitizer can *now* be used to test and debug CPython.

AddressSanitizer has helped me find a large number of bugs in other softwares.

This patch should only be applied if we are sure that the allocator is fine and doing such things by design.

The warning is due to the Py_ADDRESS_IN_RANGE() macro: it's a know limitation, we have the same problem with valgrind.

This would be a nice neature. It would IMO be even nicer to have an ASAN-enabled buildbot.

Using,

# define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS __attribute__((no_address_safety_analysis)) __attribute__ ((noinline))

instead of,

# define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS __attribute__((no_address_safety_analysis))

seems to be a more future-proof (against inlining) solution.

I am attaching the latest tested patch against tip.

New changeset f6792f734fcc by Nick Coghlan in branch 'default':
Close #18596: Support address sanity checking in clang/GCC
http://hg.python.org/cpython/rev/f6792f734fcc

Incorporated, but as Charles-François noted, a buildbot running with it enabled would be nice.