classification
Title: test__xxsubinterpreters: AMD64 Fedora Stable 3.x buildbot aborts at test_still_running
Type: crash Stage: resolved
Components: Subinterpreters, Tests Versions: Python 3.11
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Mark.Shannon Nosy List: Mark.Shannon, erlendaasland, pablogsal, vstinner
Priority: normal Keywords:

Created on 2021-06-26 09:23 by erlendaasland, last changed 2021-07-14 21:10 by erlendaasland. This issue is now closed.

Files
File name Uploaded Description Edit
test-stdio.txt erlendaasland, 2021-06-26 09:23
Messages (9)
msg396546 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-06-26 09:23
Builder: https://buildbot.python.org/all/#/builders/543

The buildbot has failed since 0982ded179f280176868c1c4eccf77bf70687816:
bpo-44032: Move pointer to code object from frame-object to frame specials array. (GH-26771)

Test stdio log attached.
msg396547 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-06-26 09:23
Fatal Python error: Py_EndInterpreter: thread still has a frame
Python runtime state: initialized

Thread 0x00007ff625ee4740 (most recent call first):
  File "<string>", line 4 in <module>
make: *** [Makefile:1256: buildbottest] Aborted (core dumped)
program finished with exit code 2
elapsedTime=382.700268
test_still_running (test.test__xxsubinterpreters.DestroyTests) ...
msg396614 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-28 08:02
test-stdio.txt:

======================================================================
FAIL: test_from_subinterpreter (test.test_interpreters.TestInterpreterIsRunning)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64/build/Lib/test/test_interpreters.py", line 265, in test_from_subinterpreter
    self.assertEqual(out.strip(), 'True')
AssertionError: 'False' != 'True'
- False
+ True

======================================================================
FAIL: test_main (test.test_interpreters.TestInterpreterIsRunning)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64/build/Lib/test/test_interpreters.py", line 245, in test_main
    self.assertTrue(main.is_running())
AssertionError: False is not true
msg396619 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-06-28 08:43
And today's price for uninformative error message goes to...
"AssertionError: 'False' != 'True'"

;)


I'm looking into it.
msg396623 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-06-28 09:40
I cannot reproduce this on ubuntu

Ubuntu 20.04.2 LTS
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

I've tried these configs:
./configure --with-pydebug
./configure
./configure --enable-optimizations
./configure --enable-optimizations --with-lto


I also tried turning the timeout down to 2 seconds, but the failure doesn't look anything like the reported error.
msg396624 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-06-28 09:40
Can someone with fedora reproduce this failure?
msg396666 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-28 22:08
On my laptop with 4 physical CPUs (8 threads), I can reproduce test_still_running() random failure with:

$ ./python -m test test__xxsubinterpreters -j50 -v -F
(...)
0:00:37 load avg: 24.31 [ 19/1] test__xxsubinterpreters failed (36.2 sec) (...)
(...)
======================================================================
FAIL: test_still_running (test.test__xxsubinterpreters.DestroyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test__xxsubinterpreters.py", line 753, in test_still_running
    self.assertTrue(interpreters.is_running(interp),
AssertionError: False is not true : Interp 48 should be running before destruction.

FAILED (failures=1, skipped=6)

Warning -- Uncaught thread exception: RuntimeError
Exception in thread Thread-7 (run):
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/home/vstinner/python/main/Lib/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vstinner/python/main/Lib/test/test__xxsubinterpreters.py", line 46, in run
    interpreters.run_string(interp, dedent(f"""
RuntimeError: unrecognized interpreter ID 48

test test__xxsubinterpreters failed
msg397251 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-07-10 22:35
FYI, this buildbot has been green after Victor's git clean adjustment in the buildbot repo.

I've been trying to reproduce the test_still_running failure on the buildbot, but have not been able to do so deterministically.
msg397508 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-07-14 21:10
The AMD64 Fedora Stable 3.x buildbot has not been red for one week. Closing this now. Victor/Pablo, please reopen if you disagree.
History
Date User Action Args
2021-07-14 21:10:59erlendaaslandsetstatus: pending -> closed
resolution: not a bug
messages: + msg397508

stage: resolved
2021-07-10 22:35:36erlendaaslandsetstatus: open -> pending
2021-07-10 22:35:23erlendaaslandsetmessages: + msg397251
2021-06-28 22:08:17vstinnersetmessages: + msg396666
2021-06-28 09:40:46Mark.Shannonsetmessages: + msg396624
2021-06-28 09:40:14Mark.Shannonsetmessages: + msg396623
2021-06-28 08:43:06Mark.Shannonsetassignee: Mark.Shannon
messages: + msg396619
2021-06-28 08:02:33vstinnersetmessages: + msg396614
2021-06-26 09:23:54erlendaaslandsetmessages: + msg396547
2021-06-26 09:23:40erlendaaslandcreate