classification
Title: async generator receives wrong value when shared between coroutines
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: yselivanov Nosy List: Dima.Tisnek, chris.jerdonek, emilyemorehouse, jan.cespivo, njs, xtreak, yselivanov
Priority: high Keywords: patch

Created on 2017-06-26 18:55 by Dima.Tisnek, last changed 2019-09-30 06:25 by yselivanov. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7468 merged yselivanov, 2018-06-07 05:02
PR 16486 merged miss-islington, 2019-09-30 05:59
Messages (9)
msg296931 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2017-06-26 18:55
MRE

```
import asyncio


async def generator():
    while True:
        x = yield 42
        print("received", x)
        await asyncio.sleep(0.1)


async def user(name, g):
    print("sending", name)
    await g.asend(name)


async def helper():
    g = generator()
    await g.asend(None)

    await asyncio.gather(*(user(f"user-{x}", g) for x in range(3)))


if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(helper())
```

Produces output:

```
sending user-0
received user-0
sending user-1
sending user-2
received None
received None
```

Expected output (some variance allowed):

```
sending user-0
received user-0
sending user-1
sending user-2
received user-1
received user-2
```

Initial report / discussion: https://mail.python.org/pipermail/async-sig/2017-June/000293.html
msg296932 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2017-06-26 19:03
@Yuri, this bug doesn't require `gather`, here's a version with futures and explicit await's instead. It produces same output:

```
import asyncio


async def generator():
    while True:
        x = yield 42
        print("received", x)
        await asyncio.sleep(0.1)


async def user(name, g):
    print("sending", name)
    await g.asend(name)


async def helper():
    g = generator()
    await g.asend(None)

    u0 = asyncio.ensure_future(user("user-0", g))
    u1 = asyncio.ensure_future(user("user-1", g))
    u2 = asyncio.ensure_future(user("user-2", g))

    await u0
    await u1
    await u2


if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(helper())
```

Same with `asyncio.get_event_loop().create_task` as well.
msg297022 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2017-06-27 11:27
Note that the example can be further simplified by replacing user() with:

    async def send_hello(g):
        print("sending: hello")
        await g.asend("hello")

Then the output is:

sending: hello
received hello
sending: hello
sending: hello
received None
received None
msg317563 - (view) Author: Jan Češpivo (jan.cespivo) Date: 2018-05-24 11:49
I've reproduced the problem also in 3.7 branch.

```
import asyncio

loop = asyncio.get_event_loop()


async def consumer():
    while True:
        await asyncio.sleep(0)
        message = yield
        print('received', message)


async def amain():
    agenerator = consumer()
    await agenerator.asend(None)

    fa = asyncio.create_task(agenerator.asend('A'))
    fb = asyncio.create_task(agenerator.asend('B'))
    await fa
    await fb


loop.run_until_complete(amain())
```

Output:
```
received A
received None
```

If the line `await asyncio.sleep(0)` is omitted the output is ok:
```
received A
received B
```
msg317603 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-24 18:39
Thanks Jan. Thanks a lot for a short script to reproduce this bug.

The actual problem here is that asynchronous generators don't control their 'asend' and 'athrow' coroutines in any way. So if you have two of them iterating *in parallel* they will cause their asynchronous generator to be in an inconsistent state.

The most obvious solution to this problem is to prohibit iterating 'asend'/'athrow' objects in parallel by throwing an exception.  

Nathaniel, what are your thoughts on this?
msg317605 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-05-24 19:14
My thoughts: https://bugs.python.org/issue32526#msg309783
msg317607 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-24 19:19
Thanks, I'll look into adding ag_running properly.
msg353543 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-09-30 05:59
New changeset fc4a044a3c54ce21e9ed150f7d769fb479d34c49 by Yury Selivanov in branch 'master':
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (#7468)
https://github.com/python/cpython/commit/fc4a044a3c54ce21e9ed150f7d769fb479d34c49
msg353544 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-09-30 06:19
New changeset 2f87a7dc5a1ad7f37787f0adee242c931643f878 by Yury Selivanov (Miss Islington (bot)) in branch '3.8':
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel (GH-7468) (#16486)
https://github.com/python/cpython/commit/2f87a7dc5a1ad7f37787f0adee242c931643f878
History
Date User Action Args
2019-09-30 06:25:31yselivanovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-09-30 06:19:09yselivanovsetmessages: + msg353544
2019-09-30 05:59:23miss-islingtonsetpull_requests: + pull_request16072
2019-09-30 05:59:14yselivanovsetmessages: + msg353543
2018-09-22 16:47:30xtreaksetnosy: + xtreak
2018-06-07 05:02:30yselivanovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request7091
2018-05-24 19:19:56yselivanovsetmessages: + msg317607
2018-05-24 19:14:14njssetmessages: + msg317605
2018-05-24 18:39:39yselivanovsetpriority: normal -> high
assignee: yselivanov
components: + Interpreter Core, - asyncio
versions: + Python 3.8
2018-05-24 18:39:04yselivanovsetnosy: + njs
messages: + msg317603
2018-05-24 11:49:24jan.cespivosetnosy: + jan.cespivo

messages: + msg317563
versions: + Python 3.7
2017-06-27 11:27:00chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg297022
2017-06-27 01:33:10emilyemorehousesetnosy: + emilyemorehouse
2017-06-26 19:03:36Dima.Tisneksetmessages: + msg296932
2017-06-26 18:55:33Dima.Tisnekcreate