classification
Title: test-complex of test_numeric_tower.test_complex() crashes intermittently on Ubuntu buildbots
Type: crash Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, mark.dickinson, ned.deily, pitrou, skrah, vstinner
Priority: normal Keywords:

Created on 2018-05-23 23:30 by vstinner, last changed 2018-07-05 16:26 by vstinner. This issue is now closed.

Messages (14)
msg317474 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-23 23:30
http://buildbot.python.org/all/#/builders/88/builds/1044

0:07:59 load avg: 1.58 [107/416/1] test_numeric_tower crashed (Exit code -11)
Fatal Python error: Segmentation fault

Current thread 0x405c4dc0 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_numeric_tower.py", line 184 in test_complex
(...)
Re-running test 'test_numeric_tower' in verbose mode
test_complex (test.test_numeric_tower.ComparisonTest) ... ok
test_mixed_comparisons (test.test_numeric_tower.ComparisonTest) ... ok
test_binary_floats (test.test_numeric_tower.HashTest) ... ok
test_bools (test.test_numeric_tower.HashTest) ... ok
test_complex (test.test_numeric_tower.HashTest) ... ok
test_decimals (test.test_numeric_tower.HashTest) ... ok
test_fractions (test.test_numeric_tower.HashTest) ... ok
test_hash_normalization (test.test_numeric_tower.HashTest) ... ok
test_integers (test.test_numeric_tower.HashTest) ... ok

----------------------------------------------------------------------
Ran 9 tests in 0.573s

OK
msg317521 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 03:35
The same intermittent failure was seen on the ARMv7 Ubuntu 3.7 builder:

http://buildbot.python.org/all/#/builders/117/builds/314/steps/4/logs/stdio

0:00:36 load avg: 5.14 [ 48/415/1] test_numeric_tower crashed (Exit code -11) -- running: test_unicodedata (36 sec), test_tokenize (36 sec), test_lib2to3 (36 sec)
Fatal Python error: Segmentation fault
Current thread 0x400f64d0 (most recent call first):
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_numeric_tower.py", line 184 in test_complex
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/case.py", line 615 in run
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/case.py", line 663 in __call__
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 122 in run
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/suite.py", line 84 in __call__
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/unittest/runner.py", line 176 in run
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 1882 in _run_suite
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 1972 in run_unittest
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 176 in runtest_inner
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 130 in runtest
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest_mp.py", line 67 in run_tests_slave
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 517 in _main
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 510 in main
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 585 in main
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 46 in _main
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 50 in <module>
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 85 in _run_code
  File "/ssd/buildbot/buildarea/3.7.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 193 in _run_module_as_main
msg317544 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-05-24 08:19
Those two builders are using Ubuntu 14.04 specifically (with different CPU architectures).  I couldn't reproduce on my 16.04 machine.  Perhaps something odd on 14.04?

Do we have builds with Adress Sanatizer or Undefined Behavior Sanitizer?
msg317579 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-24 14:34
> Do we have builds with Adress Sanatizer or Undefined Behavior Sanitizer?

Gregory recently added new USBan builders. Example:
http://buildbot.python.org/all/#/builders/135

It's something very new, there are millions of logged warnings :-)
msg317962 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 00:14
> http://buildbot.python.org/all/#/builders/88/builds/1044

This is x86 Ubuntu Shared 3.x, and the test_complex() does still crash sometimes on this buildbot, and only this buildbot :-(

http://buildbot.python.org/all/#/builders/88/builds/1067

0:08:11 load avg: 1.62 [170/416/1] test_numeric_tower crashed (Exit code -11)
Fatal Python error: Segmentation fault

Current thread 0x405c5dc0 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_numeric_tower.py", line 184 in test_complex
msg317965 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 00:24
x86 Ubuntu Shared 3.x is a 32-bit build:

* Ubuntu 14.04.4 LTS (Linode 2048 KVM VPS, Xeon E5-2680 v3/2.50GHz host, 1 core)
* os.cpu_count: 1
* os.uname: posix.uname_result(sysname='Linux', nodename='buildbot-ubuntu', release='4.5.5-x86-linode88', version='#2 SMP Fri May 20 15:30:53 EDT 2016', machine='i686')
* platform.architecture: 32bit ELF
* platform.platform: Linux-4.5.5-x86-linode88-i686-with-glibc2.0
* sysconfig[CC]: gcc -pthread
* sysconfig[CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -Wstrict-prototypes
* sysconfig[CONFIG_ARGS]: '--with-pydebug' '--enable-shared'
* sysconfig[OPT]: -g -Og -Wall -Wstrict-prototypes
* sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -Wstrict-prototypes
msg317967 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 00:46
ARMv7 Ubuntu 3.7 builder is an ARM Cortex A15:

* Ubuntu 14.04 ARM - Quad 2.3Ghz Cortex A15 2GiB (Nvidia Jetson Tegra K1) armv7l
* os.cpu_count: 1
* os.uname: posix.uname_result(sysname='Linux', nodename='tegra-ubuntu', release='3.10.40-gc017b03', version='#1 SMP PREEMPT Mon Feb 2 17:50:48 PST 2015', machine='armv7l')
* platform.architecture: 32bit ELF
* platform.platform: Linux-3.10.40-gc017b03-armv7l-with-debian-jessie-sid
* sysconfig[CC]: gcc -pthread
* sysconfig[CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -Wstrict-prototypes
* sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -Wstrict-prototypes
* sysconfig[OPT]: -g -Og -Wall -Wstrict-prototypes
msg317968 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-05-29 01:03
FYI - the "ARMv7 Ubuntu 14.04" builder is in serious need of an update, I plan to turn it into a modern Debian system after 3.7.0 is out the door.

A specific test crashing intermittently seems odd though, this buildbot has been running the same (out of date) software for many years.  That it happens on someone's 32-bit x86 14.04 bot as well as my 32-bit arm one is interesting.

tegra-ubuntu:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/4.8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 4.8.4 (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3)
msg318217 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-30 22:34
See also bpo-33701.
msg318303 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-31 13:15
> Lib/test/test_numeric_tower.py:184 in test_complex

The crash occurs at:

    def test_complex(self):
        # comparisons with complex are special:  equality and inequality
        # comparisons should always succeed, but order comparisons should
        # raise TypeError.
        z = 1.0 + 0j
        w = -3.14 + 2.7j

        for v in 1, 1.0, F(1), D(1), complex(1):  # <~~ HERE
            self.assertEqual(z, v)
            self.assertEqual(v, z)

where F = fractions.Fraction and D = decimal.Decimal
msg318304 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-31 13:16
I bet that the bug comes from _decimal. Fraction is implemented in pure Python, and complex type is supposed to be simpler than the _decimal.Decimal type.

The whole mystery is why the bug only occurs in test_numeric_tower.test_complex() and why not in test_complex, test_fractions or test_decimal!? And why the bug is random.
msg318305 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2018-05-31 13:23
Good luck finding it then, Victor.
msg318406 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 12:33
The test also crashed on x86 Gentoo Non-Debug with X 3.7:

http://buildbot.python.org/all/#/builders/115/builds/308

0:03:11 load avg: 4.82 [ 77/415/1] test_numeric_tower crashed (Exit code -11)
Fatal Python error: Segmentation fault

Current thread 0xb74e9700 (most recent call first):
  File "/buildbot/buildarea/3.7.ware-gentoo-x86.nondebug/build/Lib/test/test_numeric_tower.py", line 184 in test_complex
msg321113 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-05 16:26
I didn't see the bug since at least one month. I close the issue.
History
Date User Action Args
2018-07-05 16:26:49vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg321113

stage: resolved
2018-06-01 12:33:47vstinnersetmessages: + msg318406
2018-05-31 13:23:34skrahsetnosy: + skrah
messages: + msg318305
2018-05-31 13:16:52vstinnersetmessages: + msg318304
2018-05-31 13:15:22vstinnersetmessages: + msg318303
2018-05-30 22:34:32vstinnersetmessages: + msg318217
2018-05-29 01:03:04gregory.p.smithsetmessages: + msg317968
2018-05-29 00:46:11vstinnersetmessages: + msg317967
2018-05-29 00:25:00vstinnersetmessages: + msg317965
2018-05-29 00:14:30vstinnersetmessages: + msg317962
2018-05-24 14:34:37vstinnersetmessages: + msg317579
2018-05-24 08:19:16pitrousetnosy: + gregory.p.smith, pitrou
messages: + msg317544
2018-05-24 03:35:40ned.deilysetnosy: + mark.dickinson, ned.deily
title: test_numeric_tower.test_complex() crashed on x86 Ubuntu Shared 3.x -> test-complex of test_numeric_tower.test_complex() crashes intermittently on Ubuntu buildbots
messages: + msg317521

versions: + Python 3.7
2018-05-23 23:31:12vstinnersettitle: test_numeric_tower.test_complex() crashed on -> test_numeric_tower.test_complex() crashed on x86 Ubuntu Shared 3.x
type: crash
components: + Tests
versions: + Python 3.8
2018-05-23 23:30:49vstinnercreate