Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async generator might re-throw GeneratorExit on aclose() #79590

Closed
vxgmichel mannequin opened this issue Dec 4, 2018 · 6 comments
Closed

Async generator might re-throw GeneratorExit on aclose() #79590

vxgmichel mannequin opened this issue Dec 4, 2018 · 6 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@vxgmichel
Copy link
Mannequin

vxgmichel mannequin commented Dec 4, 2018

BPO 35409
Nosy @Fak3, @vxgmichel, @miss-islington, @catern
PRs
  • bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw #14755
  • [3.7] bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw (GH-14755) #17257
  • [3.8] bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw (GH-14755) #17258
  • Files
  • example.py: Reproduce the issue with asyncio
  • test.py: Reproduce the issue without asyncio
  • patch.diff: A possible fix
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-11-19.13:55:48.587>
    created_at = <Date 2018-12-04.13:21:20.827>
    labels = ['interpreter-core', '3.8', 'type-bug', '3.7', '3.9']
    title = 'Async generator might re-throw GeneratorExit on aclose()'
    updated_at = <Date 2020-08-29.20:34:14.782>
    user = 'https://github.com/vxgmichel'

    bugs.python.org fields:

    activity = <Date 2020-08-29.20:34:14.782>
    actor = 'catern'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-11-19.13:55:48.587>
    closer = 'asvetlov'
    components = ['Interpreter Core']
    creation = <Date 2018-12-04.13:21:20.827>
    creator = 'vxgmichel'
    dependencies = []
    files = ['47972', '47973', '47974']
    hgrepos = []
    issue_num = 35409
    keywords = ['patch']
    message_count = 6.0
    messages = ['331043', '356968', '356969', '356970', '376075', '376076']
    nosy_count = 4.0
    nosy_names = ['Roman.Evstifeev', 'vxgmichel', 'miss-islington', 'catern']
    pr_nums = ['14755', '17257', '17258']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue35409'
    versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    @vxgmichel
    Copy link
    Mannequin Author

    vxgmichel mannequin commented Dec 4, 2018

    As far as I can tell, this issue is different than: https://bugs.python.org/issue34730

    I noticed async_gen.aclose() raises a GeneratorExit exception if the async generator finalization awaits and silence a failing unfinished future (see example.py).

    This seems to be related to a bug in async_gen_athrow_throw. In fact, async_gen.aclose().throw(exc) does not silence GeneratorExit exceptions. This behavior can be reproduced without asyncio (see test.py).

    Attached is a possible patch, although I'm not too comfortable messing with the python C internals. I can make a PR if necessary.

    @vxgmichel vxgmichel mannequin added 3.7 (EOL) end of life 3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Dec 4, 2018
    @miss-islington
    Copy link
    Contributor

    New changeset 8e0de2a by Miss Islington (bot) (Vincent Michel) in branch 'master':
    bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw (GH-14755)
    8e0de2a

    @asvetlov asvetlov added the 3.9 only security fixes label Nov 19, 2019
    @miss-islington
    Copy link
    Contributor

    New changeset 6c3b471 by Miss Islington (bot) in branch '3.8':
    bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw (GH-14755)
    6c3b471

    @miss-islington
    Copy link
    Contributor

    New changeset 4ffc569 by Miss Islington (bot) in branch '3.7':
    bpo-35409: Ignore GeneratorExit in async_gen_athrow_throw (GH-14755)
    4ffc569

    @catern
    Copy link
    Mannequin

    catern mannequin commented Aug 29, 2020

    I'm not sure this was the correct fix - or at least, this creates further issues with asynccontextmanager. Consider the following code:

    ---------------

    import contextlib
    import types

    @contextlib.asynccontextmanager
    async def acm():
    # GeneratorExit athrown here from AsyncContextManager __aexit__,
    # propagated from the body of the contextmanager in func()
    yield

    @types.coroutine
    def _yield():
        yield
    
    async def func():
        async with acm():
            # GeneratorExit raised here
            await _yield()
    
    x = func()
    x.send(None) # start running func
    x.close() # raise GeneratorExit in func at its current yield
    # AsyncContextManager __aexit__ fails with "RuntimeError: generator didn't stop after throw()"

    The reason for the failure in AsyncContextManager __aexit__ is that the asyncgenerator raises StopIteration instead of GeneratorExit when agen.athrow(GeneratorExit()) is called and driven, so "await agen.athrow(GeneratorExit())" just evaluates to None, rather than raising GeneratorExit.

    On 3.6 this would work fine, because "await athrow(GeneratorExit())" will raise GeneratorExit. I suspect this was broken by this change.

    @catern
    Copy link
    Mannequin

    catern mannequin commented Aug 29, 2020

    My mistake, I see now this is just https://bugs.python.org/issue33786 and is already fixed.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants