Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

async generator receives wrong value when shared between coroutines #74956

Closed
dimaqq mannequin opened this issue Jun 26, 2017 · 9 comments
Closed

async generator receives wrong value when shared between coroutines #74956

dimaqq mannequin opened this issue Jun 26, 2017 · 9 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@dimaqq
Copy link
Mannequin

dimaqq mannequin commented Jun 26, 2017

BPO 30773
Nosy @njsmith, @cjerdonek, @dimaqq, @1st1, @emilyemorehouse, @jancespivo, @tirkarthi
PRs
  • bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel #7468
  • [3.8] bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (GH-7468) #16486
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/1st1'
    closed_at = <Date 2019-09-30.06:25:31.467>
    created_at = <Date 2017-06-26.18:55:33.730>
    labels = ['interpreter-core', '3.8', 'type-bug', '3.7']
    title = 'async generator receives wrong value when shared between coroutines'
    updated_at = <Date 2019-09-30.06:25:31.466>
    user = 'https://github.com/dimaqq'

    bugs.python.org fields:

    activity = <Date 2019-09-30.06:25:31.466>
    actor = 'yselivanov'
    assignee = 'yselivanov'
    closed = True
    closed_date = <Date 2019-09-30.06:25:31.467>
    closer = 'yselivanov'
    components = ['Interpreter Core']
    creation = <Date 2017-06-26.18:55:33.730>
    creator = 'Dima.Tisnek'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 30773
    keywords = ['patch']
    message_count = 9.0
    messages = ['296931', '296932', '297022', '317563', '317603', '317605', '317607', '353543', '353544']
    nosy_count = 7.0
    nosy_names = ['njs', 'chris.jerdonek', 'Dima.Tisnek', 'yselivanov', 'emilyemorehouse', 'jan.cespivo', 'xtreak']
    pr_nums = ['7468', '16486']
    priority = 'high'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue30773'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    Linked PRs

    @dimaqq
    Copy link
    Mannequin Author

    dimaqq mannequin commented Jun 26, 2017

    MRE

    import asyncio
    
    
    async def generator():
        while True:
            x = yield 42
            print("received", x)
            await asyncio.sleep(0.1)
    
    
    async def user(name, g):
        print("sending", name)
        await g.asend(name)
    
    
    async def helper():
        g = generator()
        await g.asend(None)
    
        await asyncio.gather(*(user(f"user-{x}", g) for x in range(3)))
    
    
    if __name__ == "__main__":
        asyncio.get_event_loop().run_until_complete(helper())
    

    Produces output:

    sending user-0
    received user-0
    sending user-1
    sending user-2
    received None
    received None
    

    Expected output (some variance allowed):

    sending user-0
    received user-0
    sending user-1
    sending user-2
    received user-1
    received user-2
    

    Initial report / discussion: https://mail.python.org/pipermail/async-sig/2017-June/000293.html

    @dimaqq dimaqq mannequin added topic-asyncio type-bug An unexpected behavior, bug, or error labels Jun 26, 2017
    @dimaqq
    Copy link
    Mannequin Author

    dimaqq mannequin commented Jun 26, 2017

    @yuri, this bug doesn't require gather, here's a version with futures and explicit await's instead. It produces same output:

    import asyncio
    
    
    async def generator():
        while True:
            x = yield 42
            print("received", x)
            await asyncio.sleep(0.1)
    
    
    async def user(name, g):
        print("sending", name)
        await g.asend(name)
    
    
    async def helper():
        g = generator()
        await g.asend(None)
    
        u0 = asyncio.ensure_future(user("user-0", g))
        u1 = asyncio.ensure_future(user("user-1", g))
        u2 = asyncio.ensure_future(user("user-2", g))
    
        await u0
        await u1
        await u2
    
    
    if __name__ == "__main__":
        asyncio.get_event_loop().run_until_complete(helper())
    

    Same with asyncio.get_event_loop().create_task as well.

    @cjerdonek
    Copy link
    Member

    Note that the example can be further simplified by replacing user() with:

        async def send_hello(g):
            print("sending: hello")
            await g.asend("hello")

    Then the output is:

    sending: hello
    received hello
    sending: hello
    sending: hello
    received None
    received None

    @jancespivo
    Copy link
    Mannequin

    jancespivo mannequin commented May 24, 2018

    I've reproduced the problem also in 3.7 branch.

    import asyncio
    
    loop = asyncio.get_event_loop()
    
    
    async def consumer():
        while True:
            await asyncio.sleep(0)
            message = yield
            print('received', message)
    
    
    async def amain():
        agenerator = consumer()
        await agenerator.asend(None)
    
        fa = asyncio.create_task(agenerator.asend('A'))
        fb = asyncio.create_task(agenerator.asend('B'))
        await fa
        await fb
    
    
    loop.run_until_complete(amain())
    

    Output:

    received A
    received None
    

    If the line await asyncio.sleep(0) is omitted the output is ok:

    received A
    received B
    

    @jancespivo jancespivo mannequin added the 3.7 (EOL) end of life label May 24, 2018
    @1st1
    Copy link
    Member

    1st1 commented May 24, 2018

    Thanks Jan. Thanks a lot for a short script to reproduce this bug.

    The actual problem here is that asynchronous generators don't control their 'asend' and 'athrow' coroutines in any way. So if you have two of them iterating *in parallel* they will cause their asynchronous generator to be in an inconsistent state.

    The most obvious solution to this problem is to prohibit iterating 'asend'/'athrow' objects in parallel by throwing an exception.

    Nathaniel, what are your thoughts on this?

    @1st1 1st1 added interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.8 only security fixes and removed topic-asyncio labels May 24, 2018
    @1st1 1st1 self-assigned this May 24, 2018
    @njsmith
    Copy link
    Contributor

    njsmith commented May 24, 2018

    @1st1
    Copy link
    Member

    1st1 commented May 24, 2018

    Thanks, I'll look into adding ag_running properly.

    @1st1
    Copy link
    Member

    1st1 commented Sep 30, 2019

    New changeset fc4a044 by Yury Selivanov in branch 'master':
    bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (bpo-7468)
    fc4a044

    @1st1
    Copy link
    Member

    1st1 commented Sep 30, 2019

    New changeset 2f87a7d by Yury Selivanov (Miss Islington (bot)) in branch '3.8':
    bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (GH-7468) (bpo-16486)
    2f87a7d

    @1st1 1st1 closed this as completed Sep 30, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    kristjanvalur added a commit to kristjanvalur/stackless that referenced this issue Oct 23, 2023
    kristjanvalur added a commit to kristjanvalur/stackless that referenced this issue Oct 23, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants