This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: AddressSanitizer: Skip tests directly in Python, not with external config
Type: Stage: resolved
Components: Tests Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2022-02-04 10:09 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 31632 merged vstinner, 2022-03-01 09:28
PR 31634 merged vstinner, 2022-03-01 15:04
PR 31644 merged vstinner, 2022-03-02 16:16
Messages (11)
msg412499 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-04 10:09
It seems like bpo-45200: "Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)" is not fixed yet.

In the GitHub Action job, test_multiprocessing_fork is skipped because it's too slow, so the job doesn't hang.

Yesterday, I modified the ASAN buildbot to double its timeout from 20 to 40 minutes:
https://github.com/python/buildmaster-config/commit/5a37411e75c9475d48eabaac18102a3c9fc2d467

But it's useful, when it hangs, it hangs forever. Exmaple on the AMD64 Arch Linux Asan Debug 3.9 buildbot (with the new config):

---
(test.test_multiprocessing_fork.WithProcessesTestPicklingConnections) ... ok
Timeout (0:35:00)!
---
https://buildbot.python.org/all/#/builders/588/builds/332

Tests are tuned for ASAN, but the configuration is copied and inconsistent between the GitHub Actions job and the buildbot configuration.

I propose to move this configuration directly into Python.

test_decimal.py checks for "-fsanitize=address" in CFLAGS and skip some tests if it's present.
msg412501 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-04 10:19
Notes:

* test.support has check_sanitizer() function. Tests using it:

  * test_crypt
  * test_idle
  * test_tix
  * test_tk
  * test_ttk_guionly


* test_decimal also checks for memory sanitizer and skip some tests if it's present:

MEMORY_SANITIZER = (
    '-fsanitize=memory' in _cflags or
    '--with-memory-sanitizer' in _config_args
)   

* test_faulthandler suppress crash report, similir to support.SuppressCrashReport, directly in the C code used by tests: faulthandler_suppress_crash_report() function.

* Objects/obmalloc.c checks "#if __has_feature(address_sanitizer)": is ASAN enabled?

* GitHub Action config: .github/workflows/build.yml. Skipped tests:

  * test___all__
  * test_concurrent_futures
  * test_multiprocessing_fork
  * test_multiprocessing_forkserver
  * test_multiprocessing_spawn
  * test_peg_generator
  * test_tools

Comment:
---
# Skip test_tools test_peg_generator test_concurrent_futures because
# there are too slow: between 5 and 20 minutes on this CI.
#
# Skip multiprocessing and concurrent.futures tests which are affected by
# bpo-45200 bug: libasan dead lock in pthread_create().
#
# test___all__ is skipped because importing some modules directly can trigger
# known problems with ASAN (like tk or crypt).
---

* Buildbot configuration: UnixAsanBuild class of master/custom/factories.py. Options:

  ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0'

With the comment:
---
# See https://bugs.python.org/issue42985 for more context on why
# SIGSEGV is ignored on purpose.
---

  Skipped tests:
  
  * test_ctypes
  * test_capi
  * test_crypt
  * test_decimal
  * test_faulthandler
  * test_interpreters

With the comment:
---
# These tests are currently raising false positives or are interfering with the ASAN mechanism,
# so we need to skip them unfortunately.
---
msg412506 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-02-04 13:00
I have recently added some decorators in test.
support to deactivate tests if running under the sanitizers.
msg414247 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-01 09:25
I created a PR to no longer skip tests in buildbots:
https://github.com/python/buildmaster-config/pull/314
msg414248 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-01 09:35
Status with the 2 pending PRs:

* GH-31632
* https://github.com/python/buildmaster-config/pull/314

Tests only skipped on ASAN:

* _test_multiprocessing

Skip on ASAN and MSAN:

* test___all__
* test_concurrent_futures
* test_crypt
* test_decimal.test_maxcontext_exact_arith()
* test_idle
* test_peg_generator
* test_tix
* test_tk
* test_tools
* test_ttk_guionly

No longer skipped:

* test_capi
* test_ctypes
* test_faulthandler
* test_interpreters
msg414250 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-01 09:38
I created bpo-46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'.
msg414262 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-01 14:00
"""
No longer skipped:

* test_capi
* test_ctypes
* test_faulthandler
* test_interpreters
"""

I built Python manually with:

  ./configure --with-pydebug CC=clang LD=clang --with-address-sanitizer 

These tests no longer with ASAN_OPTIONS used on our CI:

ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0' ./python -m test (...)
msg414266 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-01 14:44
New changeset 9204bb72a2da5885facc747e63d2bd2d654606fe by Victor Stinner in branch 'main':
bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632)
https://github.com/python/cpython/commit/9204bb72a2da5885facc747e63d2bd2d654606fe
msg414370 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-02 16:05
New changeset 93264452d952d9ba604bacf2947c2df5dd477931 by Victor Stinner in branch '3.10':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634)
https://github.com/python/cpython/commit/93264452d952d9ba604bacf2947c2df5dd477931
msg414379 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-02 17:12
New changeset 359bc392ba2b8f2acca223426c8210bb74f724c6 by Victor Stinner in branch '3.9':
[3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) (GH-31644)
https://github.com/python/cpython/commit/359bc392ba2b8f2acca223426c8210bb74f724c6
msg414380 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-02 17:13
> https://github.com/python/buildmaster-config/pull/314

I merged this PR as well. The new buildbot configuration will be deployed soon.
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90791
2022-03-02 17:13:40vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg414380

stage: patch review -> resolved
2022-03-02 17:12:54vstinnersetmessages: + msg414379
2022-03-02 16:16:12vstinnersetpull_requests: + pull_request29764
2022-03-02 16:05:28vstinnersetmessages: + msg414370
2022-03-01 15:04:34vstinnersetpull_requests: + pull_request29756
2022-03-01 14:44:17vstinnersetmessages: + msg414266
2022-03-01 14:00:40vstinnersetmessages: + msg414262
2022-03-01 09:38:41vstinnersetmessages: + msg414250
2022-03-01 09:35:44vstinnersetmessages: + msg414248
2022-03-01 09:28:07vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request29754
2022-03-01 09:25:57vstinnersetmessages: + msg414247
2022-02-04 13:00:45pablogsalsetmessages: + msg412506
2022-02-04 10:19:13vstinnersetmessages: + msg412501
2022-02-04 10:09:11vstinnercreate