classification
Title: Importing asyncio after deleting a coroutine object and before cleaning it up leads to crashing on Python3.11
Type: crash Stage: resolved
Components: asyncio Versions: Python 3.11
process
Status: open Resolution: fixed
Dependencies: Superseder:
Assigned To: Mark.Shannon Nosy List: Dennis Sweeney, Mark.Shannon, asvetlov, terry.reedy, xxm, yselivanov
Priority: release blocker Keywords: 3.11regression, patch

Created on 2021-11-16 08:44 by xxm, last changed 2021-11-30 00:47 by Dennis Sweeney.

Pull Requests
URL Status Linked Edit
PR 29700 merged Mark.Shannon, 2021-11-22 11:31
Messages (11)
msg406388 - (view) Author: Xinmeng Xia (xxm) Date: 2021-11-16 08:44
The following crashing can only reproduce on Python3.11.  In this case, we import "asyncio" after deleting a coroutine object and before cleaning it up, leading to crashing. 


test.py
=======================
async def f():
    pass
f = f()
frame = f.cr_frame
del f

import asyncio

frame.clear()
======================

>>>Python3.11 -Werror test.py
Exception ignored in: <coroutine object f at 0x7f860ad9cdd0>
Traceback (most recent call last):
  File "python311/Lib/warnings.py", line 506, in _warn_unawaited_coroutine
    warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeWarning: coroutine 'f' was never awaited
Segmentation fault (core dumped)



Version: Python 3.11.0a2+ on Ubuntu 16.04
msg406636 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-11-20 01:03
No crash on Windows running in CommandPrompt (using -i to ensure that python survives running the test code):

C:\Users\Terry>py -3.10 -i f:/dev/tem/tem.py
f:\dev\tem\tem.py:5: RuntimeWarning: coroutine 'f' was never awaited
  del f
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
>>> quit()

C:\Users\Terry>py -3.11 -i f:/dev/tem/tem.py
f:\dev\tem\tem.py:5: RuntimeWarning: coroutine 'f' was never awaited
  del f
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
>>> quit()

Running in IDLE, I only see the first warning.  (IDLE bug?  python behavior difference?  Don't know yet.)  On 3.11, the remote process crashes and IDLE's Shell does an unrequested restart.
msg406637 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-11-20 01:05
On 3.11, both of the last two lines are required for the crash and restart.
msg406650 - (view) Author: Xinmeng Xia (xxm) Date: 2021-11-20 10:22
Thanks for testing it. Maybe it only crashes on Unix-like operating systems. I also try it on MacOS 11.6.1. with -i to ensure that python survives running the test code. On MacOS, it reports the following crashing information: 

------------------- 
xxm$ Python-3.11.0a2/python.exe -i test.py 
test.py:5: RuntimeWarning: coroutine 'f' was never awaited
  del f
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Segmentation fault: 11
--------------------
msg406686 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-11-20 18:55
The crash in IDLE could be in tcl or any of the extra modules imported by IDLE, such as tkinter and socket.  It should be retested on Windows once this appears fixed otherwise.
msg406735 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-11-21 19:38
I got a crash on Windows in Objects/genobject.c:

void
_PyGen_Finalize(PyObject *self)
{
    PyGenObject *gen = (PyGenObject *)self;
    PyObject *res = NULL;
    PyObject *error_type, *error_value, *error_traceback;

    if (gen->gi_xframe == NULL ||  _PyFrameHasCompleted(gen->gi_xframe)) { <------- Crash is here
        /* Generator isn't paused, so no need to close */
        return;
    }
    ...


It looks like gen->gi_xframe is a junk-but-not-NULL pointer that is getting dereferenced by _PyFrameHasCompleted.

Maybe related to bpo-44590.
msg406736 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-11-21 19:43
I think the import is irrelevant (luckily). This still crashes:

async def f():
    pass
frame = f().cr_frame
frame.clear()
msg406737 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-11-21 19:56
Even without garbage-collecting the coroutine, we get a failed assertion in debug mode (but no crash with the assertion removed):

Python 3.11.0a2+ (heads/main:c8c21bdd19, Nov 21 2021, 13:58:01) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> async def f():
...     pass
...
>>> coro = f()
>>> frame = coro.cr_frame
>>> frame.clear()
<stdin>:1: RuntimeWarning: coroutine 'f' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Assertion failed: f->f_frame->generator == NULL, file C:\Users\sween\Source\Repos\cpython2\cpython\Objects\frameobject.c, line 705
msg406762 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-11-22 10:53
I'm seeing a similar failure on a debug build of 3.10 as well.

./python 
Python 3.10.0+ (heads/3.10:9e7a2e4920, Nov 22 2021, 10:51:32) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> async def f():
...     pass
... 
>>> coro = f()
>>> frame = coro.cr_frame
>>> frame.clear()
<stdin>:1: RuntimeWarning: coroutine 'f' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
python: Objects/frameobject.c:695: frame_clear: Assertion `f->f_gen == NULL' failed.
Aborted (core dumped)


I don't know why that assertion is there, it doesn't seem to be valid.
msg406780 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-11-22 14:01
New changeset 7fd92a8b7ee5bed28c2681fa38e0a1e76200dd8e by Mark Shannon in branch 'main':
bpo-45813: Make sure that frame->generator is NULLed when generator is deallocated. (GH-29700)
https://github.com/python/cpython/commit/7fd92a8b7ee5bed28c2681fa38e0a1e76200dd8e
msg407345 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-11-30 00:47
I think the PR fixed one case, but the other case (when coro is kept around) still fails an assertion:

Python 3.11.0a2+ (heads/main:734ed35383, Nov 29 2021, 19:29:25) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> async def f(): pass
...
>>> coro = f()
>>> frame = coro.cr_frame
>>> frame.clear()
<stdin>:1: RuntimeWarning: coroutine 'f' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Assertion failed: f->f_frame->generator == NULL, file C:\Users\sween\Source\Repos\cpython2\multiply\Objects\frameobject.c, line 689

Perhaps there should be some sort of backport as well?
History
Date User Action Args
2021-11-30 00:47:45Dennis Sweeneysetstatus: closed -> open

messages: + msg407345
2021-11-29 12:04:15asvetlovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-11-22 14:01:31Mark.Shannonsetmessages: + msg406780
2021-11-22 11:31:05Mark.Shannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request27936
2021-11-22 11:22:28Mark.Shannonsetkeywords: + 3.11regression
assignee: Mark.Shannon
priority: normal -> release blocker
2021-11-22 10:53:59Mark.Shannonsetmessages: + msg406762
2021-11-21 19:56:23Dennis Sweeneysetmessages: + msg406737
2021-11-21 19:43:37Dennis Sweeneysetmessages: + msg406736
2021-11-21 19:38:02Dennis Sweeneysetnosy: + Dennis Sweeney, Mark.Shannon
messages: + msg406735
2021-11-20 18:55:09terry.reedysetmessages: + msg406686
2021-11-20 10:22:56xxmsetmessages: + msg406650
2021-11-20 01:05:10terry.reedysetmessages: + msg406637
2021-11-20 01:03:57terry.reedysetnosy: + terry.reedy
messages: + msg406636
2021-11-16 08:44:12xxmcreate