This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: memoryview rely on _Bool undefined behavior (test_buffer fails on Python built with GCC 11)
Type: Stage: resolved
Components: Tests Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords:

Created on 2020-12-07 15:00 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (10)
msg382643 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-07 15:00
PPC64LE Fedora Rawhide LTO 3.x:
https://buildbot.python.org/all/#/builders/448/builds/433

FAIL: test_memoryview_cast (test.test_buffer.TestBufferProtocol)
FAIL: test_memoryview_cast_1D_ND (test.test_buffer.TestBufferProtocol)
FAIL: test_memoryview_compare_random_formats (test.test_buffer.TestBufferProtocol)
FAIL: test_ndarray_format_shape (test.test_buffer.TestBufferProtocol)
FAIL: test_ndarray_format_strides (test.test_buffer.TestBufferProtocol)
FAIL: test_ndarray_getbuf (test.test_buffer.TestBufferProtocol)
FAIL: test_ndarray_index_getitem_single (test.test_buffer.TestBufferProtocol)
FAIL: test_ndarray_random (test.test_buffer.TestBufferProtocol)
FAIL: test_ndarray_slice_assign_single (test.test_buffer.TestBufferProtocol)

Example:

FAIL: test_ndarray_slice_assign_single (test.test_buffer.TestBufferProtocol)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-ppc64le.lto/build/Lib/test/test_buffer.py", line 1836, in test_ndarray_slice_assign_single
    self.assertEqual(mv, nd)
AssertionError: <memory at 0x7fff80e51dc0> != <ndarray object at 0x7fff80c35030>
msg382644 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-07 15:05
On Python 3.6/x86-64, Python does crash in libregrtest while loading test_code:

0:01:13 load avg: 2.23 [ 56/406] test_code

Fatal Python error: Segmentation fault
Current thread 0x00007f5a6864d740 (most recent call first):
  File "/builddir/build/BUILD/Python-3.6.12/Lib/test/test_code.py", line 288 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678 in exec_module
  File "<frozen importlib._bootstrap>", line 665 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971 in _find_and_load
  File "<frozen importlib._bootstrap>", line 994 in _gcd_import
  File "/builddir/build/BUILD/Python-3.6.12/Lib/importlib/__init__.py", line 126 in import_module
  (...)

On Python 3.5/x86-64, test_ctypes.test_callbacks() crashs with a SIGSEGV:

test_callbacks (ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase) ...

Fatal Python error: Segmentation fault
Current thread 0x00007f5de8322740 (most recent call first):
  File "/builddir/build/BUILD/Python-3.5.10/Lib/ctypes/test/test_as_parameter.py", line 90 in test_callbacks

The strange part is that the test suite pass on some platforms depending on the Python version.

It might be a GCC 11 regression.
msg382645 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-07 15:08
Copy of Miro Hrončok's email to Fedora python-devel list:

https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/ADKCB6HXGZHJPYX5LSJ7422FITYSOAL3/

----
Hello.

I've been notified that many of the Python interpreters started to fail with new 
GNU Toolchain (gcc 11, glibc 2.33).

https://koschei.fedoraproject.org/package/pythonX.Y (replace X.Y with version)

Python 3.5
  - test_buffer fails on aarch64
  - test_buffer+test_dbm_gnu fails on ppc64le
  - %check segafults on x86_64

Python 3.6
  - %check segafults on x86_64
  - test_buffer fails on aarch64

Python 3.7
  - test_buffer fails on aarch64+ppc64le

Python 3.8
  - seems to build fine

Python 3.9
  - test_buffer fails on aarch64+ppc64le

Python 3.10
  - seems to build fine

PyPys have not yet been collected by Koschei, Python 2.7 seems fine.
----


Python is built with LTO+PGO on Fedora.
msg382646 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-07 15:12
test_buffer also fails with GCC 11 on Python 3.8 built without LTO nor PGO, just -Og (./configure --enable-pydebug).

PPC64LE Fedora Rawhide Refleaks 3.8:
https://buildbot.python.org/all/#/builders/398/builds/101

test.pythoninfo:

sysconfig[CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall

sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall

sysconfig[PY_CFLAGS_NODIST]: -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include/internal

sysconfig[PY_STDMODULE_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include

Note: I'm not sure why test.pythoninfo didn't log LDFLAGS? It's supposed to log them.
msg382693 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-07 23:00
> FAIL: test_memoryview_cast (test.test_buffer.TestBufferProtocol)

It might be related to _Bool. See also bpo-39689.
msg382701 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-08 00:24
> FAIL: test_memoryview_cast (test.test_buffer.TestBufferProtocol)

On AArch64, GCC 11 miscompiles unpack_cmp() of Objects/memoryview.c for the '?' case:

    /* boolean */
    case '?': CMP_SINGLE(p, q, _Bool); return equal;

I reported the bug to GCC upstream: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98190
msg382719 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-08 09:49
See also: Behavior of struct “?” format (native _Bool)
https://discuss.python.org/t/behavior-of-struct-format-native-bool/3774
msg382973 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-14 10:18
> I reported the bug to GCC upstream: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98190

GCC got a fix:
https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=3e60ddeb8220ed388819bb3f14e8caa9309fd3c2
msg384322 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-04 11:55
The Fedora package gcc-11.0.0-0.11 includes https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=3e60ddeb8220ed388819bb3f14e8caa9309fd3c2 fix for the https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98190 bug.
msg384323 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-04 11:58
"PPC64LE Fedora Rawhide LTO 3.x" buildbot is back to green. test.pythoninfo now says:

CC.version: gcc (GCC) 11.0.0 20201223 (Red Hat 11.0.0-0)
History
Date User Action Args
2022-04-11 14:59:39adminsetgithub: 86753
2021-01-04 11:58:06vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg384323

stage: resolved
2021-01-04 11:55:51vstinnersetmessages: + msg384322
2020-12-14 10:18:49vstinnersetmessages: + msg382973
2020-12-08 09:49:58vstinnersettitle: test_buffer fails on Python built with GCC 11 -> memoryview rely on _Bool undefined behavior (test_buffer fails on Python built with GCC 11)
2020-12-08 09:49:30vstinnersetmessages: + msg382719
2020-12-08 00:24:19vstinnersetmessages: + msg382701
2020-12-07 23:00:01vstinnersetmessages: + msg382693
2020-12-07 15:12:39vstinnersetmessages: + msg382646
2020-12-07 15:08:16vstinnersetmessages: + msg382645
2020-12-07 15:05:07vstinnersetmessages: + msg382644
2020-12-07 15:00:21vstinnercreate