classification
Title: Modules/gcmodule.c:434: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Type: crash Stage: resolved
Components: Interpreter Core, Tests Versions: Python 3.9
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, vstinner
Priority: normal Keywords:

Created on 2020-04-07 21:09 by vstinner, last changed 2020-04-08 02:35 by gregory.p.smith. This issue is now closed.

Messages (5)
msg365944 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-07 21:08
ARMv7 Debian buster 3.x:
https://buildbot.python.org/all/#/builders/168/builds/688

0:12:12 load avg: 9.96 [332/420] test_multiprocessing_spawn passed (8 min) -- running: test_zipfile (1 min 11 sec), test_largefile (1 min 25 sec), test_weakref (1 min 17 sec), test_asyncio (1 min 49 sec), test_concurrent_futures (4 min 16 sec)
./Include/object.h:492: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback

object address  : 0xb5ef05a0
object refcount : -1
object type     : 0x6f9c6c
object type name: method_descriptor
object repr     : <refcnt -1 at 0xb5ef05a0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x1e11c50)

Current thread 0xb6f44010 (most recent call first):
<no Python frame>
/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/psm_eec4c151': [Errno 2] No such file or directory: '/psm_eec4c151'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

(...)

0:13:03 load avg: 8.51 [355/420/1] test_concurrent_futures failed (5 min 5 sec) -- running: test_asyncio (2 min 40 sec), test_compileall (33.7 sec)
Modules/gcmodule.c:434: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0xb63581c0
object refcount : 0
object type     : 0x69d1d0
object type name: tuple
object repr     : <refcnt 0 at 0xb63581c0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0xb6fb3010 (most recent call first):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/typing.py", line 762 in __setattr__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/typing.py", line 658 in __init__
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/typing.py", line 1549 in _alias
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/typing.py", line 1607 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/asyncio/staggered.py", line 6 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/asyncio/base_events.py", line 45 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/asyncio/__init__.py", line 8 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/support/__init__.py", line 6 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/cmdline.py", line 4 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/__init__.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 11 in <module>
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 87 in _run_code
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 97 in _run_module_code
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 210 in run_module
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/spawn.py", line 258 in _fixup_main_from_name
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/spawn.py", line 234 in prepare
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/spawn.py", line 125 in _main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/forkserver.py", line 313 in _serve_one
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/forkserver.py", line 274 in main
  File "<string>", line 1 in <module>
test_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_cancelled (test.test_concurrent_futures.FutureTests) ... ok
test_done (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_cancelled (test.test_concurrent_futures.FutureTests) ... ok

(...)
test_ressources_gced_in_workers (test.test_concurrent_futures.ProcessPoolForkserverProcessPoolExecutorTest) ... ERROR
(...)

======================================================================
ERROR: test_ressources_gced_in_workers (test.test_concurrent_futures.ProcessPoolForkserverProcessPoolExecutorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_concurrent_futures.py", line 952, in test_ressources_gced_in_workers
    mgr = get_context(self.ctx).Manager()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/managers.py", line 556, in start
    self._address = reader.recv()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/connection.py", line 255, in recv
    buf = self._recv_bytes()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/multiprocessing/connection.py", line 388, in _recv
    raise EOFError
EOFError

Stdout:
2.10s
msg365946 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-07 21:25
Other errors in the same build:

0:14:10 load avg: 7.70 [392/420/2] test_http_cookiejar crashed (Exit code -6) -- running: test_buffer (1 min 9 sec), test_pickle (57.5 sec), test_asyncio (3 min 47 sec), test_tarfile (36.8 sec), test_unparse (46.4 sec)
./Include/object.h:492: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback

object address  : 0xb53dddc0
object refcount : -1
object type     : 0x772950
object type name: str
object repr     : <refcnt -1 at 0xb53dddc0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0xb6f1b010 (most recent call first):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest.py", line 193 in runtest
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/runtest_mp.py", line 80 in run_tests_worker
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 654 in _main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 634 in main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/libregrtest/main.py", line 712 in main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 43 in _main
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/regrtest.py", line 47 in <module>
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 87 in _run_code
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/runpy.py", line 197 in _run_module_as_main

(...)

0:14:26 load avg: 7.84 [402/420/3] test_unicode crashed (Exit code -6) -- running: test_buffer (1 min 25 sec), test_pickle (1 min 13 sec), test_asyncio (4 min 3 sec), test_unparse (1 min 2 sec)
Objects/unicodeobject.c:559: _PyUnicode_CheckConsistency: Assertion failed: compact->wstr_length == ascii->length
Enable tracemalloc to get the memory block allocation traceback

object address  : 0xb12af060
object refcount : 1
object type     : 0x76d950
object type name: str
object repr     : Objects/unicodeobject.c:559: _PyUnicode_CheckConsistency: Assertion failed: compact->wstr_length == ascii->length
Enable tracemalloc to get the memory block allocation traceback

object address  : 0xb12af060
object refcount : 1
object type     : 0x76d950
object type name: str
object repr     : Objects/unicodeobject.c:559: _PyUnicode_CheckConsistency: Assertion failed: compact->wstr_length == ascii->length
Enable tracemalloc to get the memory block allocation traceback

(... same error dozens of times ...)
msg365950 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-07 21:40
Gregory: is this worker sick? Does it always rebuild from scratch?
msg365967 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-08 00:39
While I did a general apt update on that worker, I suspect I may have knocked the heatsink loose over the weekend while dealing with an upcoming bot sitting behind it on the shelf.  That'd explain why it is alternately crashing and making mystery results.  I'll figure it out.
msg365969 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-08 02:35
bad buildbot hardware.  i've taken that bot offline.
History
Date User Action Args
2020-04-08 02:35:57gregory.p.smithsetstatus: open -> closed
type: crash
messages: + msg365969

resolution: not a bug
stage: resolved
2020-04-08 00:39:03gregory.p.smithsetmessages: + msg365967
2020-04-07 21:40:00vstinnersetnosy: + gregory.p.smith
messages: + msg365950
2020-04-07 21:25:18vstinnersetmessages: + msg365946
2020-04-07 21:09:00vstinnercreate