classification
Title: add aiter() and anext() functions
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, Dennis Sweeney, Dima.Tisnek, Eric Wieser, Jelle Zijlstra, John Belmonte, davide.rizzo, gvanrossum, jab, jack1142, jmehnle, pablogsal, rb, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2017-10-24 13:26 by davide.rizzo, last changed 2021-04-07 22:59 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
aiter_comp.py davide.rizzo, 2017-10-27 09:52 tentative Python implementation
Pull Requests
URL Status Linked Edit
PR 8895 closed jab, 2018-08-24 04:56
PR 23847 merged jab, 2020-12-18 21:12
PR 25004 merged pablogsal, 2021-03-23 23:46
PR 25005 merged pablogsal, 2021-03-23 23:53
PR 25008 merged pablogsal, 2021-03-24 01:19
Messages (25)
msg304910 - (view) Author: Davide Rizzo (davide.rizzo) * Date: 2017-10-24 13:26
PEP 525 suggested that adding aiter() and anext() would need to wait until async __aiter__ is dropped in 3.7. Issue 31709 solved that, so now it would be possible to add them.
msg304923 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-10-24 16:17
Guido, do we need a PEP to add aiter() and anext() builtins?
msg304924 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-10-24 16:24
> do we need a PEP to add aiter() and anext() builtins?

No, just this tracker issue, a PR and a reviewer. (Sorry, I can't review
CPython code myself any more.)
msg304926 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-10-24 16:25
> No, just this tracker issue, a PR and a reviewer. (Sorry, I can't review
CPython code myself any more.)

Alright, I'll work on a PR after PEP 55x.
msg305107 - (view) Author: Davide Rizzo (davide.rizzo) * Date: 2017-10-27 09:52
I attempted to make a Python implementation (attached) to use in my code. There are a few questions in the comments.

One of the complications is the async equivalent of next with two arguments like next(iterator, default). It cannot return the result of __anext__() because it needs to catch StopAsyncIteration. So it should return an awaitable wrapper instead (in my Python code this is rendered as a coroutine). A secondary question is whether the default value should be returned as it is passed, or awaited on.
msg319519 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * (Python triager) Date: 2018-06-14 14:51
Do these really need to be builtins?

They seem too specialized to be widely useful; I've personally never needed them in any async code I've written. It would make more sense to me to put them in a module like operators.
msg319520 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-06-14 15:10
> Do these really need to be builtins?

We're only beginning to see async iterators being used in the wild, so we can't have a definitive answer at this point.

> They seem too specialized to be widely useful; I've personally never needed them in any async code I've written. It would make more sense to me to put them in a module like operators.

I think putting them to the operators module makes sense, at least for 3.8.  Do you want to work on a pull request?
msg323989 - (view) Author: Joshua Bronson (jab) * Date: 2018-08-24 04:57
Updating the issue title to reflect the decision to add these to the operator module rather than to built-ins.

Submitted a PR here: https://github.com/python/cpython/pull/8895
msg323997 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-08-24 11:18
Great, thanks! There's someone working on a pr to add them as builtins (so that pr is in C). I'll have to take a look when i'm back from vacation at that pr too, before we can make a decision. 

I've been to europython, pycon ru, and pybay conferences recently and the number of people who want these as builtins was surprisingly high.
msg324009 - (view) Author: Joshua Bronson (jab) * Date: 2018-08-24 16:30
Interesting, thanks Yury!

I only saw the discussion here which implied these wouldn't go directly into builtins for 3.8 (and I searched here and in GitHub, and couldn't find the PR you mentioned either), so I'm curious if that was tracked somewhere. It'd be unfortunate if the work I did on that PR couldn't be used, but I'd be even happier to have these as builtins.

Thanks again for reviewing when you're back, and have a wonderful vacation in the meantime!
msg337061 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2019-03-04 02:13
https://www.python.org/dev/peps/pep-0525/#aiter-and-anext-builtins kinda promised `aiter` and `anext` built-ins.

This ticket seems idle.

Perhaps it's time for the decider club to either remove that language from PEP-525 or make a plan for aiter/anext?
msg337135 - (view) Author: Joshua Bronson (jab) * Date: 2019-03-04 15:49
If the deciders prefer to have these in the operator module for 3.8 (as Yury and Jelle requested above), my PR from a few months ago which implements this (https://github.com/python/cpython/pull/8895) can still be merged with no conflicts. I don't think any other changes to that patch are requested before it can be merged (i.e. it's only stalled on the decision whether to add these to builtins or operator), but I can still make time to address any new requested code changes if these are to go in operator.

If these are to go in builtins instead, @nanjekyejoannah has volunteered to pick that up. So it seems like this can move forward one way or the other once we have a decision on operator vs. builtins.
msg375633 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-08-19 03:28
Might as well re-target for 3.10 as 3.9 seems feature-complete now.
msg380408 - (view) Author: John Belmonte (John Belmonte) Date: 2020-11-05 12:03
Adding this would be a nice compliment to aclosing(), which was already merged for 3.10.

Otherwise, witness the ugliness of using aclosing() with an async iterable object:

    async with aclosing(foo.__aiter__()) as agen:
        async for item in agen:
            ...

I'd like:

    async with aclosing(aiter(foo)) as agen:
        async for item in agen:
            ...
msg381832 - (view) Author: Joshua Bronson (jab) * Date: 2020-11-25 14:18
Nice to see there is still interest in this from someone else! Thanks, John. Are any core developers still interested in this?

If I can get a new PR together that adds C implementations of `aiter` and `anext` to builtins, would a committer still be interested in reviewing the patch?

A week from Friday, I'll have a rare and precious opportunity to spend the day contributing to open source. I have a bunch of different things I could work on, but would work on this if there is still interest. Thanks and hope this finds you all well.
msg383327 - (view) Author: Joshua Bronson (jab) * Date: 2020-12-18 21:18
Please see https://github.com/python/cpython/pull/23847 for the C implementation of aiter and anext added to builtins, as requested.
msg383441 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-12-20 19:12
I don't have anything to add to this beside the name choice is safe and won't clash with anything (but honestly I would prefer it to be discussed on the ML before implementing something after 3 years). I checked a limited dataset to search for aiter and only found examples from 2 different projects. Elastic have something for themselves and the other usages are coming from the tests of aioitertools.
https://github.com/elastic/elasticsearch-py/blob/5fe9ff721ce493fbf2fc8b94d5ab02fc7e55fd5a/elasticsearch/_async/helpers.py#L85-L96
msg389414 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-03-23 22:47
New changeset f0a6fde8827d5d4f7a1c741ab1a8b206b66ffd57 by Joshua Bronson in branch 'master':
bpo-31861: Add aiter and anext to builtins (#23847)
https://github.com/python/cpython/commit/f0a6fde8827d5d4f7a1c741ab1a8b206b66ffd57
msg389417 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-23 23:57
New changeset d9692027f41ee7600fe401c066617ebfc8bac930 by Pablo Galindo in branch 'master':
bpo-31861: Complete the C-API docs for PyObject_GetAiter and PyAiter_Check (GH-25004)
https://github.com/python/cpython/commit/d9692027f41ee7600fe401c066617ebfc8bac930
msg389420 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-24 00:30
New changeset 919d42d477093154a30b55d9d79f023dbbe5614a by Pablo Galindo in branch 'master':
bpo-31861: Fix possible crash in PyAnextAwaitable_New (GH-25005)
https://github.com/python/cpython/commit/919d42d477093154a30b55d9d79f023dbbe5614a
msg389422 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-24 01:04
I reopen the issue.

The commit f0a6fde8827d5d4f7a1c741ab1a8b206b66ffd57 introduced a reference leak:

$ ./python -m test -R 3:3 test_asyncgen

0:00:00 load avg: 4.75 Run tests sequentially
0:00:00 load avg: 4.75 [1/1] test_asyncgen
beginning 6 repetitions
123456
......
test_asyncgen leaked [72, 72, 72] references, sum=216
test_asyncgen leaked [30, 30, 30] memory blocks, sum=90
test_asyncgen failed

== Tests result: FAILURE ==

1 test failed:
    test_asyncgen

Total duration: 6.0 sec
Tests result: FAILURE
msg389426 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-24 01:42
New changeset a02683ac38183fa3a45c32319dfd329c5e622f0e by Pablo Galindo in branch 'master':
bpo-31861: Fix reference leak in builtin_anext_impl() (GH-25008)
https://github.com/python/cpython/commit/a02683ac38183fa3a45c32319dfd329c5e622f0e
msg389427 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-24 01:43
Fixed by PR25008
msg390402 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python triager) Date: 2021-04-07 06:47
A bug was reported in anext(): https://bugs.python.org/issue43751
msg390495 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-07 22:59
I proposed to PR 25266 to rename PyAnextAwaitable_Type to _PyAnextAwaitable_Type, and to initialize the type at Python startup: can someone please have a look?
History
Date User Action Args
2021-04-07 22:59:05vstinnersetnosy: + vstinner
messages: + msg390495
2021-04-07 10:01:16vstinnersetnosy: - vstinner
2021-04-07 06:47:38Dennis Sweeneysetnosy: + Dennis Sweeney
messages: + msg390402
2021-03-24 01:43:07pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg389427

stage: patch review -> resolved
2021-03-24 01:42:29pablogsalsetmessages: + msg389426
2021-03-24 01:19:20pablogsalsetstage: resolved -> patch review
pull_requests: + pull_request23766
2021-03-24 01:04:38vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg389422

resolution: fixed -> (no value)
2021-03-24 00:30:11pablogsalsetmessages: + msg389420
2021-03-23 23:57:13pablogsalsetmessages: + msg389417
2021-03-23 23:53:47pablogsalsetpull_requests: + pull_request23763
2021-03-23 23:46:24pablogsalsetnosy: + pablogsal

pull_requests: + pull_request23762
2021-03-23 22:54:54gvanrossumsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-03-23 22:47:42gvanrossumsetnosy: + gvanrossum
messages: + msg389414
2021-03-17 23:57:27jack1142setnosy: + jack1142
2020-12-20 19:12:44BTaskayasetnosy: + BTaskaya
messages: + msg383441
2020-12-18 21:18:53jabsetmessages: + msg383327
title: add aiter() and anext() functions to operator module -> add aiter() and anext() functions
2020-12-18 21:12:18jabsetpull_requests: + pull_request22708
2020-11-25 14:18:09jabsetmessages: + msg381832
2020-11-05 12:03:22John Belmontesetnosy: + John Belmonte
messages: + msg380408
2020-08-19 03:28:26Dima.Tisneksetmessages: + msg375633
versions: + Python 3.10, - Python 3.8
2020-06-03 08:48:13Eric Wiesersetnosy: + Eric Wieser
2020-05-08 16:10:47rbsetnosy: + rb
2019-03-04 15:58:17giampaolo.rodolasetnosy: - giampaolo.rodola
2019-03-04 15:49:05jabsetmessages: + msg337135
2019-03-04 02:13:24Dima.Tisneksetnosy: + Dima.Tisnek

messages: + msg337061
versions: + Python 3.8, - Python 3.7
2018-08-24 16:30:52jabsetmessages: + msg324009
2018-08-24 15:52:29gvanrossumsetnosy: - gvanrossum
2018-08-24 11:18:08yselivanovsetmessages: + msg323997
2018-08-24 04:57:32jabsetmessages: + msg323989
title: aiter() and anext() built-in functions -> add aiter() and anext() functions to operator module
2018-08-24 04:56:02jabsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8368
2018-06-25 06:03:43giampaolo.rodolasetnosy: + giampaolo.rodola
2018-06-24 19:39:27jabsetnosy: + jab
2018-06-14 15:10:54yselivanovsetmessages: + msg319520
2018-06-14 14:51:41Jelle Zijlstrasetnosy: + Jelle Zijlstra
messages: + msg319519
2018-06-13 00:22:00jmehnlesetnosy: + jmehnle
2017-10-27 09:52:53davide.rizzosetfiles: + aiter_comp.py

messages: + msg305107
2017-10-24 16:25:26yselivanovsetmessages: + msg304926
2017-10-24 16:24:34gvanrossumsetmessages: + msg304924
2017-10-24 16:17:13yselivanovsetmessages: + msg304923
2017-10-24 16:10:26davide.rizzosetnosy: + gvanrossum, yselivanov
2017-10-24 13:26:15davide.rizzocreate