classification
Title: enable usage of AddressSanitizer in CPython [PATCH]
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, halfie, haypo, ncoghlan, neologix, python-dev
Priority: normal Keywords: patch

Created on 2013-07-30 08:52 by halfie, last changed 2013-09-28 14:31 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
enable-ASAN-in-CPython.diff halfie, 2013-07-30 08:52 patch to enable ASAN in CPython review
ASAN-compat-35da5d848ffd-v3.patch halfie, 2013-09-25 12:51 review
Messages (7)
msg193917 - (view) Author: (halfie) Date: 2013-07-30 08:52
Hi,

When trying to build CPython "tip" with AddressSanitizer enabled, I get the following crash,

make Parser/pgen
make[1]: Entering directory `/scratch/repos/cpython'
make[1]: `Parser/pgen' is up to date.
make[1]: Leaving directory `/scratch/repos/cpython'
Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
=================================================================
==1375== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60380000f020 at pc 0x40b260 bp 0x7fff6e6cbbf0 sp 0x7fff6e6cbbe8
READ of size 4 at 0x60380000f020 thread T0
    #0 0x40b25f in _PyObject_Realloc /scratch/repos/cpython/Objects/obmalloc.c:1551
    #1 0x403048 in PyNode_AddChild /scratch/repos/cpython/Parser/node.c:98
    #2 0x4037fe in push /scratch/repos/cpython/Parser/parser.c:126
    #3 0x4121bd in parsetok /scratch/repos/cpython/Parser/parsetok.c:211
    #4 0x413344 in PyParser_ParseFileFlags /scratch/repos/cpython/Parser/parsetok.c:86
    #5 0x413422 in getgrammar /scratch/repos/cpython/Parser/pgenmain.c:92
    #6 0x401860 in main /scratch/repos/cpython/Parser/pgenmain.c:55
    #7 0x3cb2221b74 in ?? ??:0
    #8 0x401994 in _start ??:?
0x60380000f020 is located 96 bytes to the left of 0-byte region [0x60380000f080,0x60380000f080)
==1375== AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_allocator2.cc:216 "((id)) != (0)" (0x0, 0x0)
    #0 0x3ab681237d in ?? ??:0
    #1 0x3ab68191b3 in ?? ??:0
    #2 0x3ab680563b in ?? ??:0
    #3 0x3ab6816e8f in ?? ??:0
    #4 0x3ab6817ff1 in ?? ??:0
    #5 0x3ab6812763 in ?? ??:0
    #6 0x40b25f in _PyObject_Realloc /scratch/repos/cpython/Objects/obmalloc.c:1551
    #7 0x403048 in PyNode_AddChild /scratch/repos/cpython/Parser/node.c:98
    #8 0x4037fe in push /scratch/repos/cpython/Parser/parser.c:126
    #9 0x4121bd in parsetok /scratch/repos/cpython/Parser/parsetok.c:211
    #10 0x413344 in PyParser_ParseFileFlags /scratch/repos/cpython/Parser/parsetok.c:86
    #11 0x413422 in getgrammar /scratch/repos/cpython/Parser/pgenmain.c:92
    #12 0x401860 in main /scratch/repos/cpython/Parser/pgenmain.c:55
    #13 0x3cb2221b74 in ?? ??:0
    #14 0x401994 in _start ??:?
make: *** [Include/graminit.h] Error 1

...

AddressSanitizer is a fast memory error detector. See http://clang.llvm.org/docs/AddressSanitizer.html for details.


Build options used
==================

export CFLAGS="-fsanitize=address -g"

export LDFLAGS="-fsanitize=address"

./configure --prefix=/opt/python --with-valgrind


Software Versions
==================

$ hg log

changeset:   84902:6e1dd1ce95b8
branch:      2.7
tag:         tip
parent:      84893:1dbcb0299088
user:        Terry Jan Reedy <tjreedy@udel.edu>
date:        Tue Jul 30 01:37:36 2013 -0400
summary:     Make all idle test case names end with 'Test'.

$ gcc --version
gcc (GCC) 4.8.1 20130603 (Red Hat 4.8.1-1)

$ cat /etc/redhat-release 
Fedora release 19 (Schrödinger’s Cat)

Fix / Work-around
=================

See http://code.google.com/p/address-sanitizer/wiki/AddressSanitizer#Turning_off_instrumentation

Turning off the instrumentation for _PyObject_Free and  _PyObject_Realloc is required.

Advantages
==========

AddressSanitizer can *now* be used to test and debug CPython.

AddressSanitizer has helped me find a large number of bugs in other softwares.
msg193935 - (view) Author: (halfie) Date: 2013-07-30 14:19
This patch should only be applied if we are sure that the allocator is fine and doing such things by design.
msg193973 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-07-31 05:24
The warning is due to the Py_ADDRESS_IN_RANGE() macro: it's a know limitation, we have the same problem with valgrind.

This would be a nice neature. It would IMO be even nicer to have an ASAN-enabled buildbot.
msg193974 - (view) Author: (halfie) Date: 2013-07-31 06:00
Using,

# define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS __attribute__((no_address_safety_analysis)) __attribute__ ((noinline))

instead of,

# define ATTRIBUTE_NO_ADDRESS_SAFETY_ANALYSIS __attribute__((no_address_safety_analysis))

seems to be a more future-proof (against inlining) solution.
msg198382 - (view) Author: (halfie) Date: 2013-09-25 12:51
I am attaching the latest tested patch against tip.
msg198517 - (view) Author: Roundup Robot (python-dev) Date: 2013-09-28 14:29
New changeset f6792f734fcc by Nick Coghlan in branch 'default':
Close #18596: Support address sanity checking in clang/GCC
http://hg.python.org/cpython/rev/f6792f734fcc
msg198519 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-28 14:31
Incorporated, but as Charles-François noted, a buildbot running with it enabled would be nice.
History
Date User Action Args
2014-03-23 13:19:07neologixlinkissue20953 superseder
2013-09-28 14:31:00ncoghlansetmessages: + msg198519
2013-09-28 14:29:15python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg198517

resolution: fixed
stage: patch review -> resolved
2013-09-25 12:51:44halfiesetfiles: + ASAN-compat-35da5d848ffd-v3.patch

messages: + msg198382
2013-09-25 05:47:56ncoghlansetnosy: + ncoghlan
2013-07-31 06:00:59halfiesetmessages: + msg193974
2013-07-31 05:24:10neologixsetnosy: + neologix
messages: + msg193973
2013-07-30 14:31:38christian.heimessettype: crash -> enhancement
2013-07-30 14:19:37halfiesettype: enhancement -> crash
messages: + msg193935
2013-07-30 13:30:30pitrousetnosy: + haypo, christian.heimes

type: crash -> enhancement
stage: patch review
2013-07-30 08:52:39halfiecreate