classification
Title: memory leaks in pgen build step abort build with address sanitizer enabled
Type: Stage: resolved
Components: Build Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, benjamin.peterson, geeknik, ned.deily, xiang.zhang
Priority: low Keywords: patch

Created on 2016-08-16 22:18 by geeknik, last changed 2018-06-26 03:40 by benjamin.peterson. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7869 merged benjamin.peterson, 2018-06-23 03:34
PR 7915 merged zach.ware, 2018-06-25 21:48
Messages (9)
msg272889 - (view) Author: geeknik (geeknik) Date: 2016-08-16 22:18
CC=afl-clang-fast CXX=afl-clang-fast++ AFL_USE_ASAN=1 ./configure --disable-ipv6

Passes fine.

AFL_USE_ASAN=1 make

Ends with the following:

Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c

=================================================================
==29392==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 12416 byte(s) in 8 object(s) allocated from:
    #0 0x4a1f2e in realloc (/root/cpython/Parser/pgen+0x4a1f2e)
    #1 0x4cfcca in PyMem_RawRealloc /root/cpython/Objects/obmalloc.c:414:12

Indirect leak of 2080 byte(s) in 3 object(s) allocated from:
    #0 0x4a1f2e in realloc (/root/cpython/Parser/pgen+0x4a1f2e)
    #1 0x4cfcca in PyMem_RawRealloc /root/cpython/Objects/obmalloc.c:414:12

Indirect leak of 898 byte(s) in 86 object(s) allocated from:
    #0 0x4a1c3b in __interceptor_malloc (/root/cpython/Parser/pgen+0x4a1c3b)
    #1 0x7fbd7abe3989 in __strdup /build/glibc-uPj9cH/glibc-2.19/string/strdup.c:42

Indirect leak of 520 byte(s) in 1 object(s) allocated from:
    #0 0x4a1c3b in __interceptor_malloc (/root/cpython/Parser/pgen+0x4a1c3b)
    #1 0x4d3fa0 in PyMem_RawMalloc /root/cpython/Objects/obmalloc.c:396:12
    #2 0x4d3fa0 in _PyObject_Alloc /root/cpython/Objects/obmalloc.c:1467

Indirect leak of 178 byte(s) in 33 object(s) allocated from:
    #0 0x4a1c3b in __interceptor_malloc (/root/cpython/Parser/pgen+0x4a1c3b)
    #1 0x4c6756 in translabel /root/cpython/Parser/grammar.c:197:28
    #2 0x4c6756 in _Py_translatelabels /root/cpython/Parser/grammar.c:141

SUMMARY: AddressSanitizer: 16092 byte(s) leaked in 131 allocation(s).
Makefile:804: recipe for target 'Include/graminit.h' failed
make: *** [Include/graminit.h] Error 23
msg272996 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-17 22:33
Thanks for the report but, AFAIK, pgen is only used during the build of Python and pgen is not installed (by "make install").  This doesn't seem like it is worth worrying about.  Or am I missing something?  Feel free to reopen if so or if someone comes up with a patch.
msg272998 - (view) Author: geeknik (geeknik) Date: 2016-08-17 22:37
FYI, I was only able to build Python with ASAN by passing ASAN_OPTIONS=detect_leaks=0 along with the make command, otherwise ASAN wanted to stop the build process as soon as it detected this leak.
msg273000 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-08-17 22:52
OK, that's not unreasonable and I see there have been earlier issues opened and addressed for similar problems (e.g. Issue18695).  Perhaps someone will want to dive in.
msg273268 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016-08-21 05:48
Valgrind can also complain:

valgrind --leak-check=yes --suppressions=Misc/valgrind-python.supp Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
==6836== Memcheck, a memory error detector
==6836== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==6836== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==6836== Command: Parser/pgen ./Grammar/Grammar Include/graminit.h Python/graminit.c
==6836== 
Translating labels ...
==6836== 
==6836== HEAP SUMMARY:
==6836==     in use at exit: 18,236 bytes in 133 blocks
==6836==   total heap usage: 848 allocs, 715 frees, 781,340 bytes allocated
==6836== 
==6836== 6 bytes in 1 blocks are possibly lost in loss record 1 of 11
==6836==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x50E1379: strdup (strdup.c:42)
==6836==    by 0x403405: _Py_addlabel (grammar.c:104)
==6836==    by 0x403D40: newnfagrammar (pgen.c:116)
==6836==    by 0x403E21: metacompile (pgen.c:158)
==6836==    by 0x4056F1: _Py_pgen (pgen.c:668)
==6836==    by 0x40CBE3: getgrammar (pgenmain.c:125)
==6836==    by 0x40C932: main (pgenmain.c:64)
==6836== 
==6836== 178 bytes in 33 blocks are possibly lost in loss record 2 of 11
==6836==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x403838: translabel (grammar.c:203)
==6836==    by 0x40356A: _Py_translatelabels (grammar.c:147)
==6836==    by 0x405711: _Py_pgen (pgen.c:670)
==6836==    by 0x40CBE3: getgrammar (pgenmain.c:125)
==6836==    by 0x40C932: main (pgenmain.c:64)
==6836== 
==6836== 584 bytes in 1 blocks are possibly lost in loss record 3 of 11
==6836==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x405774: _PyMem_RawMalloc (obmalloc.c:83)
==6836==    by 0x4076DE: _PyMem_DebugRawAlloc (obmalloc.c:1903)
==6836==    by 0x4077CC: _PyMem_DebugRawMalloc (obmalloc.c:1926)
==6836==    by 0x405EDF: PyMem_RawMalloc (obmalloc.c:396)
==6836==    by 0x406D16: _PyObject_Alloc (obmalloc.c:1466)
==6836==    by 0x407515: _PyObject_Realloc (obmalloc.c:1756)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x4030C6: _Py_addstate (grammar.c:55)
==6836==    by 0x405497: convert (pgen.c:611)
==6836== 
==6836== 640 bytes in 1 blocks are possibly lost in loss record 4 of 11
==6836==    at 0x4C2DD9F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x4057F3: _PyMem_RawRealloc (obmalloc.c:105)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40756A: _PyObject_Realloc (obmalloc.c:1777)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x403B2A: addnfastate (pgen.c:52)
==6836==    by 0x4047E5: compile_atom (pgen.c:300)
==6836==    by 0x4045E9: compile_item (pgen.c:268)
==6836==    by 0x40438D: compile_alt (pgen.c:237)
==6836== 
==6836== 744 bytes in 1 blocks are definitely lost in loss record 5 of 11
==6836==    at 0x4C2DD9F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x4057F3: _PyMem_RawRealloc (obmalloc.c:105)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40756A: _PyObject_Realloc (obmalloc.c:1777)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x403D8C: addnfa (pgen.c:126)
==6836==    by 0x403F4E: compile_rule (pgen.c:178)
==6836==    by 0x403E86: metacompile (pgen.c:164)
==6836==    by 0x4056F1: _Py_pgen (pgen.c:668)
==6836== 
==6836== 800 bytes in 1 blocks are possibly lost in loss record 6 of 11
==6836==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x405774: _PyMem_RawMalloc (obmalloc.c:83)
==6836==    by 0x4076DE: _PyMem_DebugRawAlloc (obmalloc.c:1903)
==6836==    by 0x407914: _PyMem_DebugRawRealloc (obmalloc.c:1971)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40631C: new_arena (obmalloc.c:1064)
==6836==    by 0x4067FC: _PyObject_Alloc (obmalloc.c:1339)
==6836==    by 0x406D66: _PyObject_Malloc (obmalloc.c:1476)
==6836==    by 0x4076DE: _PyMem_DebugRawAlloc (obmalloc.c:1903)
==6836==    by 0x4077CC: _PyMem_DebugRawMalloc (obmalloc.c:1926)
==6836==    by 0x407B23: _PyMem_DebugMalloc (obmalloc.c:2028)
==6836==    by 0x406179: PyObject_Malloc (obmalloc.c:489)
==6836== 
==6836== 892 bytes in 85 blocks are possibly lost in loss record 7 of 11
==6836==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x50E1379: strdup (strdup.c:42)
==6836==    by 0x40304A: _Py_adddfa (grammar.c:42)
==6836==    by 0x405691: maketables (pgen.c:655)
==6836==    by 0x405701: _Py_pgen (pgen.c:669)
==6836==    by 0x40CBE3: getgrammar (pgenmain.c:125)
==6836==    by 0x40C932: main (pgenmain.c:64)
==6836== 
==6836== 2,272 bytes in 3 blocks are possibly lost in loss record 8 of 11
==6836==    at 0x4C2DD9F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x4057F3: _PyMem_RawRealloc (obmalloc.c:105)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40756A: _PyObject_Realloc (obmalloc.c:1777)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x4030C6: _Py_addstate (grammar.c:55)
==6836==    by 0x405497: convert (pgen.c:611)
==6836==    by 0x404F7B: makedfa (pgen.c:489)
==6836==    by 0x4056C0: maketables (pgen.c:656)
==6836== 
==6836== 2,864 bytes in 1 blocks are possibly lost in loss record 9 of 11
==6836==    at 0x4C2DD9F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x4057F3: _PyMem_RawRealloc (obmalloc.c:105)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40756A: _PyObject_Realloc (obmalloc.c:1777)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x4033AA: _Py_addlabel (grammar.c:98)
==6836==    by 0x40480D: compile_atom (pgen.c:301)
==6836==    by 0x4045E9: compile_item (pgen.c:268)
==6836==    by 0x40433D: compile_alt (pgen.c:232)
==6836== 
==6836== 3,464 bytes in 1 blocks are possibly lost in loss record 10 of 11
==6836==    at 0x4C2DD9F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x4057F3: _PyMem_RawRealloc (obmalloc.c:105)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40756A: _PyObject_Realloc (obmalloc.c:1777)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x402FE4: _Py_adddfa (grammar.c:36)
==6836==    by 0x405691: maketables (pgen.c:655)
==6836==    by 0x405701: _Py_pgen (pgen.c:669)
==6836==    by 0x40CBE3: getgrammar (pgenmain.c:125)
==6836== 
==6836== 5,792 bytes in 5 blocks are possibly lost in loss record 11 of 11
==6836==    at 0x4C2DD9F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6836==    by 0x4057F3: _PyMem_RawRealloc (obmalloc.c:105)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x405F70: PyMem_RawRealloc (obmalloc.c:414)
==6836==    by 0x40756A: _PyObject_Realloc (obmalloc.c:1777)
==6836==    by 0x407999: _PyMem_DebugRawRealloc (obmalloc.c:1986)
==6836==    by 0x407BB2: _PyMem_DebugRealloc (obmalloc.c:2049)
==6836==    by 0x40620A: PyObject_Realloc (obmalloc.c:507)
==6836==    by 0x403B2A: addnfastate (pgen.c:52)
==6836==    by 0x4047E5: compile_atom (pgen.c:300)
==6836==    by 0x4045E9: compile_item (pgen.c:268)
==6836==    by 0x40433D: compile_alt (pgen.c:232)
==6836== 
==6836== LEAK SUMMARY:
==6836==    definitely lost: 744 bytes in 1 blocks
==6836==    indirectly lost: 0 bytes in 0 blocks
==6836==      possibly lost: 17,492 bytes in 132 blocks
==6836==    still reachable: 0 bytes in 0 blocks
==6836==         suppressed: 0 bytes in 0 blocks
==6836== 
==6836== For counts of detected and suppressed errors, rerun with: -v
==6836== ERROR SUMMARY: 11 errors from 11 contexts (suppressed: 484 from 2)
msg320228 - (view) Author: Zackery Spytz (ZackerySpytz) * (Python triager) Date: 2018-06-22 12:16
These memory leaks were fixed in commit 9ac11a752a19c3b8607582a3d5ccb615c467124b, though the fix wasn't applied to 2.7.
msg320273 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-22 21:16
Thanks, Zackery.  @Benjamin, any opinion about a need for a 2.7 backport?  Otherwise, it sounds like this can be closed.
msg320296 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-06-23 04:04
New changeset 482fac74999028af4764645a1f28dd4289b2f255 by Benjamin Peterson in branch '2.7':
[2.7] properly free memory in pgen. (GH-7869) (closes bpo-27780
https://github.com/python/cpython/commit/482fac74999028af4764645a1f28dd4289b2f255
msg320467 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-06-26 03:40
New changeset 16137fe22f3f9bbbd50fac2a729172976c553718 by Benjamin Peterson (Zachary Ware) in branch '2.7':
bpo-27780: Make pgen.c C89 compliant (GH-7915)
https://github.com/python/cpython/commit/16137fe22f3f9bbbd50fac2a729172976c553718
History
Date User Action Args
2018-06-26 03:40:22benjamin.petersonsetmessages: + msg320467
2018-06-25 21:48:49zach.waresetpull_requests: + pull_request7518
2018-06-23 04:04:24benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg320296

stage: patch review -> resolved
2018-06-23 03:34:21benjamin.petersonsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request7477
2018-06-22 21:16:20ned.deilysetnosy: + benjamin.peterson
messages: + msg320273
2018-06-22 12:16:36ZackerySpytzsetnosy: + ZackerySpytz
messages: + msg320228
2016-11-05 14:56:14r.david.murraylinkissue28620 superseder
2016-08-21 05:48:09xiang.zhangsetnosy: + xiang.zhang
messages: + msg273268
2016-08-17 22:52:41ned.deilysetstatus: closed -> open
versions: + Python 3.6
title: Memory leak during Python build (from git c3ff7e7) on Debian 8.5 x64 -> memory leaks in pgen build step abort build with address sanitizer enabled
messages: + msg273000

resolution: wont fix -> (no value)
stage: resolved -> needs patch
2016-08-17 22:37:58geekniksetmessages: + msg272998
2016-08-17 22:33:52ned.deilysetstatus: open -> closed
priority: normal -> low


nosy: + ned.deily
messages: + msg272996
resolution: wont fix
stage: resolved
2016-08-16 22:18:56geeknikcreate