classification
Title: await anext() returns None when default is given
Type: behavior Stage: resolved
Components: asyncio, Interpreter Core Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Dennis Sweeney, asvetlov, jab, pablogsal, pewscorner, yselivanov
Priority: release blocker Keywords: patch

Created on 2021-04-06 16:00 by pewscorner, last changed 2021-04-11 12:00 by pewscorner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25238 merged Dennis Sweeney, 2021-04-07 00:36
Messages (7)
msg390349 - (view) Author: PEW's Corner (pewscorner) * Date: 2021-04-06 16:00
The new anext() builtin in Python 3.10.0a7 doesn't seem to work properly when a default value is provided as the second argument. Here's an example:

import asyncio

async def f():
    yield 'A'
    yield 'B'

async def main():
    g = f()
    print(await anext(g, 'Z'))  # Prints 'None' instead of 'A'!!!
    print(await anext(g, 'Z'))  # Prints 'None' instead of 'B'!!!
    print(await anext(g, 'Z'))  # Prints 'Z'
    g = f()
    print(await anext(g))       # Prints 'A'
    print(await anext(g))       # Prints 'B'
    print(await anext(g))       # Raises StopAsyncIteration

asyncio.run(main())

As indicated above, anext() works fine when no default is given (in the second half of main()), but produces None in every iteration when a default is given (in the first half of main()) except when the iterator is exhausted.
msg390362 - (view) Author: Dennis Sweeney (Dennis Sweeney) * Date: 2021-04-06 19:03
I can open a PR this evening, but I think this is close to the issue: PyIter_Next() already silences StopIteration, so checking for it afterward fails.

diff --git a/Objects/iterobject.c b/Objects/iterobject.c
index f0c6b79917..95f4659dc9 100644
--- a/Objects/iterobject.c
+++ b/Objects/iterobject.c
@@ -316,7 +316,7 @@ anextawaitable_traverse(anextawaitableobject *obj, visitproc visit, void *arg)
 static PyObject *
 anextawaitable_iternext(anextawaitableobject *obj)
 {
-    PyObject *result = PyIter_Next(obj->wrapped);
+    PyObject *result = (*Py_TYPE(obj->wrapped)->tp_iternext)(obj->wrapped);
     if (result != NULL) {
         return result;
     }
msg390393 - (view) Author: Dennis Sweeney (Dennis Sweeney) * Date: 2021-04-07 00:44
That change fixes that bug, but I think there may be another bug involving when a custom async iterator is passed rather than an async generator. This is at the limit of my knowledge, so any guidance would be appreciated. The test I wrote in the PR currently fails, due to some `tp_iternext` slot being NULL sometimes. Maybe different cases are needed for Coroutine/non-Coroutine?

But it definitely seems like the aiter()/anext() code needs more test coverage.
msg390438 - (view) Author: PEW's Corner (pewscorner) * Date: 2021-04-07 14:53
Regarding the custom async iterator, I don't know if this is the problem you're referring to, but the following code seems to terminate abruptly when running main2() (main1() is fine). This is without your changes, though.

import asyncio

class CustomAsyncIter:
    def __init__(self):
        self.iterator = iter(['A', 'B'])
    def __aiter__(self):
        return self
    async def __anext__(self):
        try:
            x = next(self.iterator)
        except StopIteration:
            raise StopAsyncIteration from None
        await asyncio.sleep(1)
        return x

async def main1():
    iter1 = CustomAsyncIter()
    print(await anext(iter1))       # Prints 'A'
    print(await anext(iter1))       # Prints 'B'
    print(await anext(iter1))       # Raises StopAsyncIteration

async def main2():
    iter1 = CustomAsyncIter()
    print('Before')                 # Prints 'Before'
    print(await anext(iter1, 'Z'))  # Silently terminates the script!!!
    print('After')                  # This never gets executed

asyncio.run(main2())
msg390492 - (view) Author: Dennis Sweeney (Dennis Sweeney) * Date: 2021-04-07 22:33
Okay, the PR should fix those problems now.

I am still apprehensive about whether all of the corner cases are covered, so reviews are welcome, as are suggestions of more test cases.
msg390766 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-04-11 04:51
New changeset dfb45323ce8a543ca844c311e32c994ec9554c1b by Dennis Sweeney in branch 'master':
bpo-43751: Fix anext() bug where it erroneously returned None (GH-25238)
https://github.com/python/cpython/commit/dfb45323ce8a543ca844c311e32c994ec9554c1b
msg390776 - (view) Author: PEW's Corner (pewscorner) * Date: 2021-04-11 12:00
Thanks!
History
Date User Action Args
2021-04-11 12:00:51pewscornersetmessages: + msg390776
2021-04-11 04:52:03pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-04-11 04:51:42pablogsalsetnosy: + pablogsal
messages: + msg390766
2021-04-08 23:57:00pablogsalsetpriority: normal -> release blocker
2021-04-08 19:15:36jabsetnosy: + jab
2021-04-07 22:33:34Dennis Sweeneysetmessages: + msg390492
2021-04-07 14:53:52pewscornersetmessages: + msg390438
2021-04-07 00:44:00Dennis Sweeneysetmessages: + msg390393
2021-04-07 00:36:40Dennis Sweeneysetkeywords: + patch
stage: patch review
pull_requests: + pull_request23976
2021-04-06 19:03:26Dennis Sweeneysetnosy: + Dennis Sweeney
messages: + msg390362
2021-04-06 16:00:14pewscornercreate