Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

await anext() returns None when default is given #87917

Closed
pewscorner mannequin opened this issue Apr 6, 2021 · 7 comments
Closed

await anext() returns None when default is given #87917

pewscorner mannequin opened this issue Apr 6, 2021 · 7 comments
Labels
3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@pewscorner
Copy link
Mannequin

pewscorner mannequin commented Apr 6, 2021

BPO 43751
Nosy @jab, @asvetlov, @1st1, @pablogsal, @pewscorner, @sweeneyde
PRs
  • bpo-43751: Fix anext() bug where it erroneously returned None #25238
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-04-11.04:52:03.772>
    created_at = <Date 2021-04-06.16:00:14.125>
    labels = ['interpreter-core', 'type-bug', 'release-blocker', '3.10', 'expert-asyncio']
    title = 'await anext() returns None when default is given'
    updated_at = <Date 2021-04-11.12:00:51.777>
    user = 'https://github.com/pewscorner'

    bugs.python.org fields:

    activity = <Date 2021-04-11.12:00:51.777>
    actor = 'pewscorner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-04-11.04:52:03.772>
    closer = 'pablogsal'
    components = ['Interpreter Core', 'asyncio']
    creation = <Date 2021-04-06.16:00:14.125>
    creator = 'pewscorner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 43751
    keywords = ['patch']
    message_count = 7.0
    messages = ['390349', '390362', '390393', '390438', '390492', '390766', '390776']
    nosy_count = 6.0
    nosy_names = ['jab', 'asvetlov', 'yselivanov', 'pablogsal', 'pewscorner', 'Dennis Sweeney']
    pr_nums = ['25238']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue43751'
    versions = ['Python 3.10']

    @pewscorner
    Copy link
    Mannequin Author

    pewscorner mannequin commented Apr 6, 2021

    The new anext() builtin in Python 3.10.0a7 doesn't seem to work properly when a default value is provided as the second argument. Here's an example:

    import asyncio
    
    async def f():
        yield 'A'
        yield 'B'
    
    async def main():
        g = f()
        print(await anext(g, 'Z'))  # Prints 'None' instead of 'A'!!!
        print(await anext(g, 'Z'))  # Prints 'None' instead of 'B'!!!
        print(await anext(g, 'Z'))  # Prints 'Z'
        g = f()
        print(await anext(g))       # Prints 'A'
        print(await anext(g))       # Prints 'B'
        print(await anext(g))       # Raises StopAsyncIteration
    
    asyncio.run(main())

    As indicated above, anext() works fine when no default is given (in the second half of main()), but produces None in every iteration when a default is given (in the first half of main()) except when the iterator is exhausted.

    @pewscorner pewscorner mannequin added 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-asyncio type-bug An unexpected behavior, bug, or error labels Apr 6, 2021
    @sweeneyde
    Copy link
    Member

    I can open a PR this evening, but I think this is close to the issue: PyIter_Next() already silences StopIteration, so checking for it afterward fails.

    diff --git a/Objects/iterobject.c b/Objects/iterobject.c
    index f0c6b79917..95f4659dc9 100644
    --- a/Objects/iterobject.c
    +++ b/Objects/iterobject.c
    @@ -316,7 +316,7 @@ anextawaitable_traverse(anextawaitableobject *obj, visitproc visit, void *arg)
     static PyObject *
     anextawaitable_iternext(anextawaitableobject *obj)
     {
    -    PyObject *result = PyIter_Next(obj->wrapped);
    +    PyObject *result = (*Py_TYPE(obj->wrapped)->tp_iternext)(obj->wrapped);
         if (result != NULL) {
             return result;
         }

    @sweeneyde
    Copy link
    Member

    That change fixes that bug, but I think there may be another bug involving when a custom async iterator is passed rather than an async generator. This is at the limit of my knowledge, so any guidance would be appreciated. The test I wrote in the PR currently fails, due to some tp_iternext slot being NULL sometimes. Maybe different cases are needed for Coroutine/non-Coroutine?

    But it definitely seems like the aiter()/anext() code needs more test coverage.

    @pewscorner
    Copy link
    Mannequin Author

    pewscorner mannequin commented Apr 7, 2021

    Regarding the custom async iterator, I don't know if this is the problem you're referring to, but the following code seems to terminate abruptly when running main2() (main1() is fine). This is without your changes, though.

    import asyncio
    
    class CustomAsyncIter:
        def __init__(self):
            self.iterator = iter(['A', 'B'])
        def __aiter__(self):
            return self
        async def __anext__(self):
            try:
                x = next(self.iterator)
            except StopIteration:
                raise StopAsyncIteration from None
            await asyncio.sleep(1)
            return x
    
    async def main1():
        iter1 = CustomAsyncIter()
        print(await anext(iter1))       # Prints 'A'
        print(await anext(iter1))       # Prints 'B'
        print(await anext(iter1))       # Raises StopAsyncIteration
    
    async def main2():
        iter1 = CustomAsyncIter()
        print('Before')                 # Prints 'Before'
        print(await anext(iter1, 'Z'))  # Silently terminates the script!!!
        print('After')                  # This never gets executed
    
    asyncio.run(main2())

    @sweeneyde
    Copy link
    Member

    Okay, the PR should fix those problems now.

    I am still apprehensive about whether all of the corner cases are covered, so reviews are welcome, as are suggestions of more test cases.

    @pablogsal
    Copy link
    Member

    New changeset dfb4532 by Dennis Sweeney in branch 'master':
    bpo-43751: Fix anext() bug where it erroneously returned None (GH-25238)
    dfb4532

    @pewscorner
    Copy link
    Mannequin Author

    pewscorner mannequin commented Apr 11, 2021

    Thanks!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker topic-asyncio type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants