Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing asyncio after deleting a coroutine object and before cleaning it up leads to crashing on Python3.11 #89971

Closed
xxm mannequin opened this issue Nov 16, 2021 · 13 comments
Assignees
Labels
3.11 only security fixes release-blocker topic-asyncio type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@xxm
Copy link
Mannequin

xxm mannequin commented Nov 16, 2021

BPO 45813
Nosy @terryjreedy, @asvetlov, @markshannon, @1st1, @sweeneyde
PRs
  • bpo-45813: Make sure that frame->generator is NULLed when generator is deallocated. #29700
  • bpo-45813: Drop redundant assertion from frame.clear() #29990
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/markshannon'
    closed_at = <Date 2021-12-08.16:08:25.057>
    created_at = <Date 2021-11-16.08:44:12.273>
    labels = ['3.11', 'release-blocker', 'type-crash', 'expert-asyncio']
    title = 'Importing asyncio after deleting a coroutine object and before cleaning it up leads to crashing on Python3.11'
    updated_at = <Date 2021-12-08.16:08:25.056>
    user = 'https://bugs.python.org/xxm'

    bugs.python.org fields:

    activity = <Date 2021-12-08.16:08:25.056>
    actor = 'asvetlov'
    assignee = 'Mark.Shannon'
    closed = True
    closed_date = <Date 2021-12-08.16:08:25.057>
    closer = 'asvetlov'
    components = ['asyncio']
    creation = <Date 2021-11-16.08:44:12.273>
    creator = 'xxm'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45813
    keywords = ['patch', '3.11regression']
    message_count = 13.0
    messages = ['406388', '406636', '406637', '406650', '406686', '406735', '406736', '406737', '406762', '406780', '407345', '408017', '408032']
    nosy_count = 6.0
    nosy_names = ['terry.reedy', 'asvetlov', 'Mark.Shannon', 'yselivanov', 'Dennis Sweeney', 'xxm']
    pr_nums = ['29700', '29990']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'crash'
    url = 'https://bugs.python.org/issue45813'
    versions = ['Python 3.11']

    @xxm
    Copy link
    Mannequin Author

    xxm mannequin commented Nov 16, 2021

    The following crashing can only reproduce on Python3.11. In this case, we import "asyncio" after deleting a coroutine object and before cleaning it up, leading to crashing.

    test.py
    =======================

    async def f():
        pass
    f = f()
    frame = f.cr_frame
    del f
    
    import asyncio
    
    frame.clear()

    ======================

    >>>Python3.11 -Werror test.py
    Exception ignored in: <coroutine object f at 0x7f860ad9cdd0>
    Traceback (most recent call last):
      File "python311/Lib/warnings.py", line 506, in _warn_unawaited_coroutine
        warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    RuntimeWarning: coroutine 'f' was never awaited
    Segmentation fault (core dumped)

    Version: Python 3.11.0a2+ on Ubuntu 16.04

    @xxm xxm mannequin added 3.11 only security fixes topic-asyncio type-crash A hard crash of the interpreter, possibly with a core dump labels Nov 16, 2021
    @terryjreedy
    Copy link
    Member

    No crash on Windows running in CommandPrompt (using -i to ensure that python survives running the test code):

    C:\Users\Terry>py -3.10 -i f:/dev/tem/tem.py
    f:\dev\tem\tem.py:5: RuntimeWarning: coroutine 'f' was never awaited
      del f
    RuntimeWarning: Enable tracemalloc to get the object allocation traceback
    >>> quit()
    
    C:\Users\Terry>py -3.11 -i f:/dev/tem/tem.py
    f:\dev\tem\tem.py:5: RuntimeWarning: coroutine 'f' was never awaited
      del f
    RuntimeWarning: Enable tracemalloc to get the object allocation traceback
    >>> quit()

    Running in IDLE, I only see the first warning. (IDLE bug? python behavior difference? Don't know yet.) On 3.11, the remote process crashes and IDLE's Shell does an unrequested restart.

    @terryjreedy
    Copy link
    Member

    On 3.11, both of the last two lines are required for the crash and restart.

    @xxm
    Copy link
    Mannequin Author

    xxm mannequin commented Nov 20, 2021

    Thanks for testing it. Maybe it only crashes on Unix-like operating systems. I also try it on MacOS 11.6.1. with -i to ensure that python survives running the test code. On MacOS, it reports the following crashing information:

    -------------------
    xxm$ Python-3.11.0a2/python.exe -i test.py
    test.py:5: RuntimeWarning: coroutine 'f' was never awaited
    del f
    RuntimeWarning: Enable tracemalloc to get the object allocation traceback
    Segmentation fault: 11
    --------------------

    @terryjreedy
    Copy link
    Member

    The crash in IDLE could be in tcl or any of the extra modules imported by IDLE, such as tkinter and socket. It should be retested on Windows once this appears fixed otherwise.

    @sweeneyde
    Copy link
    Member

    I got a crash on Windows in Objects/genobject.c:

    void
    _PyGen_Finalize(PyObject *self)
    {
        PyGenObject *gen = (PyGenObject *)self;
        PyObject *res = NULL;
        PyObject *error_type, *error_value, *error_traceback;
    
        if (gen->gi_xframe == NULL ||  _PyFrameHasCompleted(gen->gi_xframe)) { <------- Crash is here
            /* Generator isn't paused, so no need to close */
            return;
        }
        ...

    It looks like gen->gi_xframe is a junk-but-not-NULL pointer that is getting dereferenced by _PyFrameHasCompleted.

    Maybe related to bpo-44590.

    @sweeneyde
    Copy link
    Member

    I think the import is irrelevant (luckily). This still crashes:

    async def f():
        pass
    frame = f().cr_frame
    frame.clear()

    @sweeneyde
    Copy link
    Member

    Even without garbage-collecting the coroutine, we get a failed assertion in debug mode (but no crash with the assertion removed):

    Python 3.11.0a2+ (heads/main:c8c21bdd19, Nov 21 2021, 13:58:01) [MSC v.1929 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> async def f():
    ...     pass
    ...
    >>> coro = f()
    >>> frame = coro.cr_frame
    >>> frame.clear()
    <stdin>:1: RuntimeWarning: coroutine 'f' was never awaited
    RuntimeWarning: Enable tracemalloc to get the object allocation traceback
    Assertion failed: f->f_frame->generator == NULL, file C:\Users\sween\Source\Repos\cpython2\cpython\Objects\frameobject.c, line 705

    @markshannon
    Copy link
    Member

    I'm seeing a similar failure on a debug build of 3.10 as well.

    ./python 
    Python 3.10.0+ (heads/3.10:9e7a2e4920, Nov 22 2021, 10:51:32) [GCC 9.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> async def f():
    ...     pass
    ... 
    >>> coro = f()
    >>> frame = coro.cr_frame
    >>> frame.clear()
    <stdin>:1: RuntimeWarning: coroutine 'f' was never awaited
    RuntimeWarning: Enable tracemalloc to get the object allocation traceback
    python: Objects/frameobject.c:695: frame_clear: Assertion `f->f_gen == NULL' failed.
    Aborted (core dumped)

    I don't know why that assertion is there, it doesn't seem to be valid.

    @markshannon
    Copy link
    Member

    New changeset 7fd92a8 by Mark Shannon in branch 'main':
    bpo-45813: Make sure that frame->generator is NULLed when generator is deallocated. (GH-29700)
    7fd92a8

    @sweeneyde
    Copy link
    Member

    I think the PR fixed one case, but the other case (when coro is kept around) still fails an assertion:

    Python 3.11.0a2+ (heads/main:734ed35383, Nov 29 2021, 19:29:25) [MSC v.1929 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> async def f(): pass
    ...
    >>> coro = f()
    >>> frame = coro.cr_frame
    >>> frame.clear()
    <stdin>:1: RuntimeWarning: coroutine 'f' was never awaited
    RuntimeWarning: Enable tracemalloc to get the object allocation traceback
    Assertion failed: f->f_frame->generator == NULL, file C:\Users\sween\Source\Repos\cpython2\multiply\Objects\frameobject.c, line 689

    Perhaps there should be some sort of backport as well?

    @sweeneyde sweeneyde reopened this Nov 30, 2021
    @sweeneyde sweeneyde reopened this Nov 30, 2021
    @asvetlov
    Copy link
    Contributor

    asvetlov commented Dec 8, 2021

    Pablo, I think #29990 should be merged as well before the issue closing.

    It covers another execution branch.

    @asvetlov asvetlov reopened this Dec 8, 2021
    @asvetlov asvetlov reopened this Dec 8, 2021
    @markshannon
    Copy link
    Member

    New changeset d4363d2 by Andrew Svetlov in branch 'main':
    bpo-45813: Drop redundant assertion from frame.clear() (GH-29990)
    d4363d2

    @asvetlov asvetlov closed this as completed Dec 8, 2021
    @asvetlov asvetlov closed this as completed Dec 8, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 only security fixes release-blocker topic-asyncio type-crash A hard crash of the interpreter, possibly with a core dump
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants