I recently changed how tests are skipped in ASAN, MSAN and UBSAN CIs (buildbot workers and GitHub Action jobs):

* bpo-46633
* https://github.com/python/buildmaster-config/commit/0fd1e3e49e4b688d5767501484cccea5fa35d3fc

3 tests are now failing on "AMD64 Arch Linux Usan 3.x":

* test_ctypes
* test_faulthandler
* test_hashlib

First failed build:
https://buildbot.python.org/all/#/builders/719/builds/632

test_ctypes: test_shorts() of ctypes.test.test_bitfields.C_Test is failing with:

---
test_shorts (ctypes.test.test_bitfields.C_Test) ... /home/vstinner/python/main/Modules/_ctypes/cfield.c:554:5: runtime error: shift exponent 18446744073709551614 is too large for 16-bit type 'short'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/vstinner/python/main/Modules/_ctypes/cfield.c:554:5 in 
---

It's a test on the "h" format code:
----
#define LOW_BIT(x)  ((x) & 0xFFFF)
#define NUM_BITS(x) ((x) >> 16)

#define GET_BITFIELD(v, size)                                           \
    if (NUM_BITS(size)) {                                               \
        v <<= (sizeof(v)*8 - LOW_BIT(size) - NUM_BITS(size));           \
        v >>= (sizeof(v)*8 - NUM_BITS(size));                           \

static PyObject *
h_get(void *ptr, Py_ssize_t size)
{
    short val;
    memcpy(&val, ptr, sizeof(val));
    GET_BITFIELD(val, size); // <==== HERE
    return PyLong_FromLong((long)val);
}

static struct fielddesc formattable[] = {
    ...
    { 'h', h_set, h_get, NULL, h_set_sw, h_get_sw},
    ...
};
----

test_hashlib: test_gil() fails with:
---
test_gil (test.test_hashlib.HashLibTestCase) ... /home/vstinner/python/main/Modules/_sha3/kcp/KeccakP-1600-opt64.c:467:9: runtime error: load of misaligned address 0x000002daafd7 for type 'UINT64' (aka 'unsigned long'), which requires 8 byte alignment
0x000002daafd7: note: pointer points here
 23 23 23 23 23  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  23 23 23
             ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/vstinner/python/main/Modules/_sha3/kcp/KeccakP-1600-opt64.c:467:9 in 
---

test_faulthandler: test_sigfpe() fails with:

======================================================================
FAIL: test_sigfpe (test.test_faulthandler.FaultHandlerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_faulthandler.py", line 228, in test_sigfpe
    self.check_fatal_error("""
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/test/test_faulthandler.py", line 129, in check_fatal_error
    self.check_error(code, line_number, fatal_error, **kw)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/python/main/Lib/test/test_faulthandler.py", line 122, in check_error
    self.assertRegex(output, regex)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: '(?m)^Fatal Python error: Floating point exception\n\nCurrent thread 0x[0-9a-f]+ \\(most recent call first\\):\n  File "<string>", line 3 in <module>' not found in 'Modules/faulthandler.c:1112:11: runtime error: division by zero\nSUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/faulthandler.c:1112:11 in '

New changeset 4173d677a1d7c72bb32d292fbff1b4cf073d615c by Victor Stinner in branch 'main':
bpo-46913: Fix test_faulthandler.test_sigfpe() on UBSAN (GH-31662)
https://github.com/python/cpython/commit/4173d677a1d7c72bb32d292fbff1b4cf073d615c

Reproducer of test_ctypes undefined behavior:
---
from ctypes import *

class BITS(Structure):
    _fields_ = [("A", c_int, 1),
                ("B", c_int, 2),
                ("C", c_int, 3),
                ("D", c_int, 4),
                ("E", c_int, 5),
                ("F", c_int, 6),
                ("G", c_int, 7),
                ("H", c_int, 8),
                ("I", c_int, 9),

                ("M", c_short, 1),
                ("N", c_short, 2),
                ("O", c_short, 3),
                ("P", c_short, 4),
                ("Q", c_short, 5),
                ("R", c_short, 6),
                ("S", c_short, 7)]

b = BITS()
a = getattr(b, "M")
---

>     GET_BITFIELD(val, size); // <==== HERE

I expanded the macro:
---
    if (NUM_BITS(size)) {
        size_t s = sizeof(val)*8;
        Py_ssize_t low = LOW_BIT(size);
        Py_ssize_t nbits = NUM_BITS(size);
        // val=0, size=65553 = (1 << 16) + 17
        // s=16, low=17, nbits=1
        // s - low - nbits = (size_t)-2
        val <<= (s - low - nbits);
        val >>= (s - nbits);
    }
---

The problem is that (s - low - nbits) is negative (-2), but becomes a very large number since it's unsigned (s is unsigned: "sizeof(v)*8" in the macro).

C code:
---
#include <stdio.h>

int main()
{
    short s = 0x7fff;
    size_t z = (size_t)-2;
    s <<= z;
    printf("s = %hi\n", s);
    return 0;
}
---

GCC and clang disagree :-D
---
$ gcc x.c -o x -O3 -Wall -Wextra -Wconversion && ./x
s = 0

$ clang x.c -o x -O3 -Wall -Wextra -Wconversion && ./x
s = -5784
---

Moreover, runtime the binary built by clang produces a different result at each run...
---
$ ./x
s = -4824
$ ./x
s = 4120
$ ./x
s = -22344
$ ./x
s = -26744
$ ./x
s = -18184
---

> https://github.com/python/buildmaster-config/commit/0fd1e3e49e4b688d5767501484cccea5fa35d3fc

Previously, 4 tests were skipped on the UBSAN buildbot

* test_faulthandler
* test_hashlib
* test_concurrent_futures
* test_ctypes

It's not a regression, it's just that I made the buildbot stricter. I'm planning to attempt to fix the 3 failing tests, or at least skip them in Python, rather than skipping them in the buildbot configuration.

New changeset 65b92ccdec2ee4a99e54aaf7ae2d9bbc2ebfe549 by Victor Stinner in branch 'main':
bpo-46913: Fix test_faulthandler.test_read_null() on UBSan (GH31672)
https://github.com/python/cpython/commit/65b92ccdec2ee4a99e54aaf7ae2d9bbc2ebfe549

See also bpo-46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'.

New changeset 6d0d7d2b8c1e04fd51c6cb29cc09a41b60b97b7b by Victor Stinner in branch 'main':
bpo-46913: test_hashlib skips _sha3 tests on UBSan (GH-31673)
https://github.com/python/cpython/commit/6d0d7d2b8c1e04fd51c6cb29cc09a41b60b97b7b

New changeset ad1b04451d3aca2c6fa6dbe2891676a4e0baac49 by Victor Stinner in branch 'main':
bpo-46913: Skip test_ctypes.test_shorts() on UBSan (GH-31674)
https://github.com/python/cpython/commit/ad1b04451d3aca2c6fa6dbe2891676a4e0baac49

I pushed changes just to turn back the buildbot back to green, but undefined behaviors in test_ctypes.test_shorts() and _sha3 (tested by test_hashlib) must be fixed.

Previously, test_ctypes, test_hashlib and test_faulthandler were fully skipped on UBSan. Now only 1 test_ctypes test and a few test_hashlib tests are skipped on the UBSan buildbot (test_faulthandler now pass on UBSan).

New changeset 7b5b429adab4fe0fe81858fe3831f06adc2e2141 by Victor Stinner in branch '3.10':
[3.10] bpo-46913: Fix test_ctypes, test_hashlib, test_faulthandler on UBSan (GH-31675)
https://github.com/python/cpython/commit/7b5b429adab4fe0fe81858fe3831f06adc2e2141

New changeset 6a14330318c9c7aedf3e9841c3dfea337064d8e6 by Victor Stinner in branch '3.9':
bpo-46913: Fix test_ctypes, test_hashlib, test_faulthandler on UBSan (GH-31675) (GH-31676)
https://github.com/python/cpython/commit/6a14330318c9c7aedf3e9841c3dfea337064d8e6

I don't agree with GH-31673. Did you try defining NO_MISALIGNED_ACCESSES instead?

> I don't agree with GH-31673. Did you try defining NO_MISALIGNED_ACCESSES instead?

Did you read the commit message? The change is not about skipping the test, but fixing the CI. Previously, test_hashlib was not run at all on the UBSan buildbot, now most test_hashlib tests are run. The intent is to make sure that we don't add *new* undefined behavior.

As I wrote in the commit, the UD must be fixed in _sha3.

No, I didn't try NO_MISALIGNED_ACCESSES, I don't know this macro. If you have an idea on how _sha3 can be fixed on the UBSan buildbot, please go head!

The ctypes overflow is probably the same as described in Issue 28169 and Issue 15119

Current status of tests skipped on ASAN, MSAN and UBSAN.

Only ASAN (1):

* _test_multiprocessing.py:76:if support.check_sanitizer(address=True):

ASAN and MSAN (10):

* test___all__.py:14:if support.check_sanitizer(address=True, memory=True):
* test_concurrent_futures.py:35:if support.check_sanitizer(address=True, memory=True):
* test_crypt.py:7:    if check_sanitizer(address=True, memory=True):
* test_decimal.py:5510:    @unittest.skipIf(check_sanitizer(address=True, memory=True),
* test_idle.py:5:if check_sanitizer(address=True, memory=True):
* test_peg_generator/__init__.py:7:if support.check_sanitizer(address=True, memory=True):
* test_tix.py:7:if check_sanitizer(address=True, memory=True):
* test_tk.py:6:if check_sanitizer(address=True, memory=True):
* test_tools/__init__.py:10:if support.check_sanitizer(address=True, memory=True):
* test_ttk_guionly.py:6:if check_sanitizer(address=True, memory=True):

Only UB (1):

* test_hashlib.py:68:SKIP_SHA3 = support.check_sanitizer(ub=True)

I enabled the test on ASAN on test_crypt and I confirm that I get a crash on calling a NULL function.

The Python crypt.crypt() function calls the C function crypt_r() which is intercepted by libasan, but the libasan implementation calls a NULL function. I don't know why.

$ ./configure --with-address-sanitizer --with-pydebug
$ make clean
$ ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0" make
$ ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0" gdb -args ./python -m test -v test_crypt 
(gdb) run
(...)
0:00:00 load avg: 0.53 Run tests sequentially
0:00:00 load avg: 0.53 [1/1] test_crypt

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

(gdb) where
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff761189f in __interceptor_crypt_r.part.0 () from /lib64/libasan.so.6
#2  0x00007fffe6a40821 in crypt_crypt_impl (module=<module at remote 0x6080004ae7c0>, word=0xfcb050 <importlib.bootstrap_external_toplevel_consts_7+48> "", salt=0x6080004bc660 "$6$d8Imx7a5WbE12iK4")
    at /home/vstinner/python/main/Modules/_cryptmodule.c:44
#3  0x00007fffe6a40695 in crypt_crypt (module=<module at remote 0x6080004ae7c0>, args=0x629000001368, nargs=2) at /home/vstinner/python/main/Modules/clinic/_cryptmodule.c.h:58
(...)

$ ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0" ./python -m test -v test_crypt 
(...)
0:00:00 load avg: 0.56 Run tests sequentially
0:00:00 load avg: 0.56 [1/1] test_crypt
Fatal Python error: Segmentation fault

Current thread 0x00007f367c6c77c0 (most recent call first):
  File "/home/vstinner/python/main/Lib/crypt.py", line 82 in crypt
  File "/home/vstinner/python/main/Lib/crypt.py", line 94 in _add_method
  File "/home/vstinner/python/main/Lib/crypt.py", line 105 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 931 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1149 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1178 in _find_and_load
  File "/home/vstinner/python/main/Lib/test/test_crypt.py", line 6 in <module>
  (...)

The crypt_r() interceptor issue was reported in January 2021 to libasan:
https://github.com/google/sanitizers/issues/1365

> I enabled the test on ASAN on test_crypt and I confirm that I get a crash on calling a NULL function.

Note: I built Python with GCC ASAN (-fsanitize=address).