classification
Title: Failure to build with address sanitizer
Type: compile error Stage: resolved
Components: Build Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: btharper, matrixise, miss-islington, ned.deily, vstinner
Priority: normal Keywords: patch

Created on 2019-03-19 03:47 by btharper, last changed 2019-10-08 13:03 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12425 merged btharper, 2019-03-19 03:53
PR 12431 merged vstinner, 2019-03-19 11:14
PR 12432 merged vstinner, 2019-03-19 11:17
PR 12435 merged vstinner, 2019-03-19 13:31
PR 12436 merged vstinner, 2019-03-19 13:40
PR 12444 closed btharper, 2019-03-19 17:15
PR 12453 merged vstinner, 2019-03-20 00:00
PR 12454 merged vstinner, 2019-03-20 00:10
PR 12968 merged vstinner, 2019-04-26 10:14
PR 13006 merged vstinner, 2019-04-29 10:09
PR 16598 merged btharper, 2019-10-05 17:44
PR 16622 merged miss-islington, 2019-10-07 16:20
Messages (28)
msg338315 - (view) Author: Ben Harper (btharper) * Date: 2019-03-19 03:47
Trying to run make after './configure --with-address-sanitizer --with-pydebug' fails with leak of locale string
msg338326 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-19 08:14
just for me, btharper, which system do you use? because I have a fedora 29 and when I try to compile with these flags, getaddrinfo is not found.
msg338329 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-19 08:32
ok, found

I have to disable ipv6 and install libasan

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi

=================================================================
==5408==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 34 byte(s) in 1 object(s) allocated from:
    #0 0x7fb085e03c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
    #1 0x4d99a2 in _PyMem_RawMalloc Objects/obmalloc.c:99
    #2 0x4d9b4a in _PyMem_DebugRawAlloc Objects/obmalloc.c:2020
    #3 0x4d9ba5 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2049
    #4 0x4dc354 in PyMem_RawMalloc Objects/obmalloc.c:527
    #5 0x4dc510 in _PyMem_RawStrdup Objects/obmalloc.c:613
    #6 0x652f5f in get_ctype_locale Python/preconfig.c:483
    #7 0x6560a6 in _PyPreConfig_Read Python/preconfig.c:506
    #8 0x616cde in _PyCoreConfig_ReadPreConfig Python/coreconfig.c:1342
    #9 0x620564 in _PyCoreConfig_Read Python/coreconfig.c:1378
    #10 0x65ef06 in pyinit_coreconfig Python/pylifecycle.c:740
    #11 0x65fa05 in _Py_InitializeCore Python/pylifecycle.c:786
    #12 0x42abba in pymain_init Modules/main.c:370
    #13 0x42addc in pymain_main Modules/main.c:889
    #14 0x42b2bb in _Py_UnixMain Modules/main.c:940
    #15 0x4258ee in main Programs/python.c:16
    #16 0x7fb085983412 in __libc_start_main ../csu/libc-start.c:308

Direct leak of 34 byte(s) in 1 object(s) allocated from:
    #0 0x7fb085e03c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
    #1 0x4d99a2 in _PyMem_RawMalloc Objects/obmalloc.c:99
    #2 0x4d9b4a in _PyMem_DebugRawAlloc Objects/obmalloc.c:2020
    #3 0x4d9ba5 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2049
    #4 0x4dc354 in PyMem_RawMalloc Objects/obmalloc.c:527
    #5 0x4dc510 in _PyMem_RawStrdup Objects/obmalloc.c:613
    #6 0x652f5f in get_ctype_locale Python/preconfig.c:483
    #7 0x6560a6 in _PyPreConfig_Read Python/preconfig.c:506
    #8 0x6585f8 in pyinit_preconfig Python/pylifecycle.c:723
    #9 0x65f8b0 in _Py_InitializeCore Python/pylifecycle.c:781
    #10 0x42abba in pymain_init Modules/main.c:370
    #11 0x42addc in pymain_main Modules/main.c:889
    #12 0x42b2bb in _Py_UnixMain Modules/main.c:940
    #13 0x4258ee in main Programs/python.c:16
    #14 0x7fb085983412 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: 68 byte(s) leaked in 2 allocation(s).
generate-posix-vars failed
make: *** [Makefile:586: pybuilddir.txt] Error 1
msg338338 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-19 10:33
your PR seems to be fine.

but I continue to get two refleaks with valgrind but don't worry, @vstinner is working on these refleaks.

==5440== 64 bytes in 1 blocks are possibly lost in loss record 1 of 2
==5440==    at 0x483880B: malloc (vg_replace_malloc.c:309)
==5440==    by 0x46BFAB: _PyMem_RawMalloc (obmalloc.c:99)
==5440==    by 0x46BEA3: _PyMem_DebugRawAlloc (obmalloc.c:2020)
==5440==    by 0x46BF33: _PyMem_DebugRawMalloc (obmalloc.c:2049)
==5440==    by 0x46D304: PyMem_RawMalloc (obmalloc.c:527)
==5440==    by 0x52C1EF: PyThread_allocate_lock (thread_pthread.h:330)
==5440==    by 0x5154F0: _PyRuntimeState_Init_impl (pystate.c:52)
==5440==    by 0x515A76: _PyRuntimeState_Init (pystate.c:77)
==5440==    by 0x514114: _PyRuntime_Initialize (pylifecycle.c:88)
==5440==    by 0x4236FE: pymain_init (main.c:339)
==5440==    by 0x423980: pymain_main (main.c:889)
==5440==    by 0x423A6A: _Py_UnixMain (main.c:940)
==5440== 
==5440== 64 bytes in 1 blocks are possibly lost in loss record 2 of 2
==5440==    at 0x483880B: malloc (vg_replace_malloc.c:309)
==5440==    by 0x46BFAB: _PyMem_RawMalloc (obmalloc.c:99)
==5440==    by 0x46BEA3: _PyMem_DebugRawAlloc (obmalloc.c:2020)
==5440==    by 0x46BF33: _PyMem_DebugRawMalloc (obmalloc.c:2049)
==5440==    by 0x46D304: PyMem_RawMalloc (obmalloc.c:527)
==5440==    by 0x52C1EF: PyThread_allocate_lock (thread_pthread.h:330)
==5440==    by 0x515506: _PyRuntimeState_Init_impl (pystate.c:58)
==5440==    by 0x515A76: _PyRuntimeState_Init (pystate.c:77)
==5440==    by 0x514114: _PyRuntime_Initialize (pylifecycle.c:88)
==5440==    by 0x4236FE: pymain_init (main.c:339)
==5440==    by 0x423980: pymain_main (main.c:889)
==5440==    by 0x423A6A: _Py_UnixMain (main.c:940)
msg338341 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-19 10:50
New changeset e130a07eb20c4b655d182d5d10d778c7584efe55 by Victor Stinner (btharper) in branch 'master':
bpo-36356: Fix memory leak in _PyPreConfig_Read() (GH-12425)
https://github.com/python/cpython/commit/e130a07eb20c4b655d182d5d10d778c7584efe55
msg338352 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-19 13:19
New changeset a712679a2bffffefaacdc05f788d6ea50f72a561 by Victor Stinner in branch 'master':
bpo-36333, bpo-36356: Fix _PyEval_FiniThreads() (GH-12432)
https://github.com/python/cpython/commit/a712679a2bffffefaacdc05f788d6ea50f72a561
msg338353 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-19 13:20
New changeset fecc4f2b474f16062514e95a67e66080fd626e14 by Victor Stinner in branch 'master':
bpo-36356: Release Unicode interned strings on Valgrind (#12431)
https://github.com/python/cpython/commit/fecc4f2b474f16062514e95a67e66080fd626e14
msg338357 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-19 13:54
New changeset f5f336a819a3d881bb217bf8f9b5cacba03a4e45 by Victor Stinner in branch 'master':
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12435)
https://github.com/python/cpython/commit/f5f336a819a3d881bb217bf8f9b5cacba03a4e45
msg338358 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-19 14:08
New changeset 935250d6f3ac7ba91e1ea8e6ca63aaf7f605e291 by Victor Stinner in branch '3.7':
bpo-36356: pymain_free() calls _PyRuntime_Finalize() (GH-12436)
https://github.com/python/cpython/commit/935250d6f3ac7ba91e1ea8e6ca63aaf7f605e291
msg338364 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-19 15:15
At least, "./python -V" no longer leaks at commit dcf617152e1d4c4a5e7965733928858a9c0936ca.

$ ./configure --with-valgrind
$ make
$ valgrind ./python -V
==9553== Memcheck, a memory error detector
==9553== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==9553== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==9553== Command: ./python -V
==9553== 
Python 3.8.0a2+
==9553== 
==9553== HEAP SUMMARY:
==9553==     in use at exit: 0 bytes in 0 blocks
==9553==   total heap usage: 33 allocs, 33 frees, 5,933 bytes allocated
==9553== 
==9553== All heap blocks were freed -- no leaks are possible
==9553== 
==9553== For counts of detected and suppressed errors, rerun with: -v
==9553== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
msg338378 - (view) Author: Ben Harper (btharper) * Date: 2019-03-19 16:32
I'm on Ubuntu 18.10/amd64 compiling with Ubuntu's GCC 8.2.0, I know there's some libraries that are missing dependencies (including bz2 and sqlite) so I may have missed the ipv6 dependencies as well.

My eventual goal was to be able to build a pgo optimized build of cpython with the address sanitizer turned on. Currently on my build of f5f336a819 I see 12 failing tests and a heap use after free as part of the ctypes test suite. 

The ctypes use after free has already been submitted as bpo-36253 since it was the only thing asan caught during tests with the leak sanitizer turned off.
msg338431 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-20 02:11
New changeset 4a1468e593c4b67d8c78b78070fff9e18ec5d790 by Victor Stinner in branch 'master':
bpo-36356: Fix _PyCoreConfig_Read() (GH-12454)
https://github.com/python/cpython/commit/4a1468e593c4b67d8c78b78070fff9e18ec5d790
msg338774 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-25 00:37
For what it's worth, with current HEAD of master (commit 62be33870e2f8517314bf9c7275548e799296f7e which includes previously merged PRs from this issue), current macOS clang with address sanitizer and pydebug enabled gets an assertion failure in parsetok.c.  Current HEAD of 3.7 does not.

$ ./configure --with-address-sanitizer --prefix=/tmp/d --with-pydebu
$ make -j3
[...]
./python -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup -fsanitize=address    ' OPT='-g -O0 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	./python -E ./setup.py  build
Assertion failed: ((intptr_t)(int)(a - line_start) == (a - line_start)), function parsetok, file Parser/parsetok.c, line 308.
/bin/sh: line 1: 95059 Abort trap: 6           CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup -fsanitize=address    ' OPT='-g -O0 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py $quiet build
make: *** [sharedmods] Error 134
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
# same result with explicit CC=clang
msg338776 - (view) Author: Ben Harper (btharper) * Date: 2019-03-25 01:52
I'm not sure about the change from 3.7 to master, but that assertion may be more likely to happen while instrumented with ASAN due to the extra space reserved between heap objects. As far as I can tell it's just expecting that the offset of two pointers will fit within an int instead of a intptr_t (4 bytes versus 8 bytes on my system).

For me running the test_pydoc from the test suite fails reliably with the parsetok.c assertion failure, but can be made to pass with a smaller ASAN redzone. The redzone must be a power of 2 and at least 16, default of 2048.

Fails:
ASAN_OPTIONS="max_redzone=256" ./python Tools/scripts/run_tests.py test_pydoc

Passes:
ASAN_OPTIONS="max_redzone=128" ./python Tools/scripts/run_tests.py test_pydoc

Values of 16, 32, and 64 also pass.
msg338907 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2019-03-26 17:13
@Ned, I have found a solution on Linux, will try on this evening with my
macOS laptop. 
I hope to come with a solution on this evening or tomorrow, but need to
check my proto.
msg340907 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-26 11:05
New changeset 4cb525a1f0cba152d1da61e531f2ad5db633120a by Victor Stinner in branch 'master':
bpo-36356: pymain_exit_error() only call pymain_free() for exit (GH-12968)
https://github.com/python/cpython/commit/4cb525a1f0cba152d1da61e531f2ad5db633120a
msg340912 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-26 11:20
See also bpo-36724 "Clear _PyRuntime at exit".
msg341060 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 09:16
New changeset b36e5d627d4232a01850707eb78a5067f3fd77f4 by Victor Stinner in branch 'master':
bpo-36356: Destroy the GIL at exit (GH-12453)
https://github.com/python/cpython/commit/b36e5d627d4232a01850707eb78a5067f3fd77f4
msg341061 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 09:19
The initial issue has been fixed, so I close the issue. I prefer to get more specific issues like bpo-36724 "Clear _PyRuntime at exit" to following changes.

Thanks Ben Harper for the bug report and the fix.

Note: Ben, I like your songs! Good to know that you chose to contribute to Python in your free time! ;-)
msg341064 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 09:47
Oh. test_daemon_threads_shutdown_stdout_deadlock of test_io fails randomly on buildbots.

I ran "./python -m test test_io -m '*daemon*' -F -j4" for 5 minutes on my Fedora 29 before merging my PR, but I didn't get any failure :-(

======================================================================
FAIL: test_daemon_threads_shutdown_stdout_deadlock (test.test_io.CMiscIOTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_io.py", line 4186, in test_daemon_threads_shutdown_stdout_deadlock
    self.check_daemon_threads_shutdown_deadlock('stdout')
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_io.py", line 4177, in check_daemon_threads_shutdown_deadlock
    self.assertIn("Fatal Python error: could not acquire lock "
AssertionError: "Fatal Python error: could not acquire lock for <_io.BufferedWriter name='<stdout>'> at interpreter shutdown, possibly due to daemon threads" not found in 'Fatal Python error: PyCOND_FINI(_PyRuntime.ceval.gil.cond) failed'

Failures:

AMD64 Fedora Rawhide Clang Installed 3.x: fail then pass
https://buildbot.python.org/all/#/builders/188/builds/93

s390x Debian 3.x: fail then pass
https://buildbot.python.org/all/#/builders/13/builds/2799

s390x RHEL 3.x: fail twice
https://buildbot.python.org/all/#/builders/21/builds/2780

AMD64 Debian root 3.x: fail twice
https://buildbot.python.org/all/#/builders/27/builds/2759
msg341065 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 11:04
New changeset 99fcc616d400cd31af0733c3f8cc93bcc1d32a44 by Victor Stinner in branch 'master':
Revert "bpo-36356: Destroy the GIL at exit (GH-12453)" (GH613006)
https://github.com/python/cpython/commit/99fcc616d400cd31af0733c3f8cc93bcc1d32a44
msg341066 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-29 11:04
I just reverted the change. I'm unable to reproduce and I am not available this week to investigate anyway.
msg354107 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-10-07 16:20
New changeset 321def805abc5b7c92c7e90ca90cb2434fdab855 by Yury Selivanov (Ben Harper) in branch 'master':
bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)
https://github.com/python/cpython/commit/321def805abc5b7c92c7e90ca90cb2434fdab855
msg354116 - (view) Author: miss-islington (miss-islington) Date: 2019-10-07 16:38
New changeset 13915a3100608f011b29da2f3716c990f523b631 by Miss Islington (bot) in branch '3.8':
bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)
https://github.com/python/cpython/commit/13915a3100608f011b29da2f3716c990f523b631
msg354119 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-07 16:46
Can we close this issue, or is there a remaining known bug?
msg354123 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-10-07 17:23
FWIW, the assertion error on macOS reported above in msg338774 does not seem to occur anymore with current HEAD of master and with recent system clang (Apple clang version 11.0.0 (clang-1100.0.33.8)) on macOS 10.14.6.
msg354200 - (view) Author: Ben Harper (btharper) * Date: 2019-10-08 12:55
The default build of python with ASAN builds successfully. Test suite and PGO build are currently blocked, but that can be separated if it makes more sense to handle them individually.

Would it make sense to add an ASAN build to the CI pipeline to make detection automatic going forward?
msg354201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-08 13:03
> The default build of python with ASAN builds successfully. Test suite and PGO build are currently blocked, but that can be separated if it makes more sense to handle them individually.

Yeah, please open one issue per ASAN issue (try to group similar warnings in the same Python issue?).


> Would it make sense to add an ASAN build to the CI pipeline to make detection automatic going forward?

Gregory maintains a clang UBSan buildbot:
https://buildbot.python.org/all/#/builders/135

Usually I prefer to setup a buildbot when most issues are fixed.

I close this issue.
History
Date User Action Args
2019-10-08 13:03:16vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg354201

stage: patch review -> resolved
2019-10-08 12:56:32yselivanovsetnosy: - yselivanov
2019-10-08 12:55:35btharpersetmessages: + msg354200
2019-10-07 17:23:48ned.deilysetmessages: + msg354123
2019-10-07 16:46:05vstinnersetmessages: + msg354119
2019-10-07 16:38:03miss-islingtonsetnosy: + miss-islington
messages: + msg354116
2019-10-07 16:20:12miss-islingtonsetpull_requests: + pull_request16209
2019-10-07 16:20:01yselivanovsetnosy: + yselivanov
messages: + msg354107
2019-10-05 17:44:59btharpersetpull_requests: + pull_request16186
2019-04-29 11:04:51vstinnersetmessages: + msg341066
2019-04-29 11:04:12vstinnersetmessages: + msg341065
2019-04-29 10:09:37vstinnersetstage: resolved -> patch review
pull_requests: + pull_request12927
2019-04-29 09:47:21vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg341064
2019-04-29 09:19:57vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg341061

stage: patch review -> resolved
2019-04-29 09:16:01vstinnersetmessages: + msg341060
2019-04-26 11:20:15vstinnersetmessages: + msg340912
2019-04-26 11:05:54vstinnersetmessages: + msg340907
2019-04-26 10:14:21vstinnersetpull_requests: + pull_request12895
2019-03-26 17:13:04matrixisesetmessages: + msg338907
2019-03-25 01:52:42btharpersetmessages: + msg338776
2019-03-25 00:37:51ned.deilysetnosy: + ned.deily
messages: + msg338774
2019-03-20 02:11:42vstinnersetmessages: + msg338431
2019-03-20 00:10:25vstinnersetpull_requests: + pull_request12407
2019-03-20 00:00:51vstinnersetpull_requests: + pull_request12406
2019-03-19 17:15:33btharpersetpull_requests: + pull_request12399
2019-03-19 16:32:31btharpersetmessages: + msg338378
2019-03-19 15:15:04vstinnersetmessages: + msg338364
2019-03-19 14:08:22vstinnersetmessages: + msg338358
2019-03-19 13:54:01vstinnersetmessages: + msg338357
2019-03-19 13:40:57vstinnersetpull_requests: + pull_request12391
2019-03-19 13:31:20vstinnersetpull_requests: + pull_request12390
2019-03-19 13:20:32vstinnersetmessages: + msg338353
2019-03-19 13:19:43vstinnersetmessages: + msg338352
2019-03-19 11:17:46vstinnersetpull_requests: + pull_request12387
2019-03-19 11:14:36vstinnersetpull_requests: + pull_request12385
2019-03-19 10:50:29vstinnersetmessages: + msg338341
2019-03-19 10:33:13matrixisesetmessages: + msg338338
2019-03-19 08:32:10matrixisesetmessages: + msg338329
2019-03-19 08:14:53matrixisesetnosy: + matrixise
messages: + msg338326
2019-03-19 03:54:55xtreaksetnosy: + vstinner
2019-03-19 03:53:08btharpersetkeywords: + patch
stage: patch review
pull_requests: + pull_request12380
2019-03-19 03:47:57btharpercreate