classification
Title: __aiter__ should return async iterator instead of awaitable
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: yselivanov Nosy List: brett.cannon, gvanrossum, larry, lukasz.langa, martin.panter, ncoghlan, ned.deily, python-dev, yselivanov
Priority: release blocker Keywords: patch

Created on 2016-06-06 19:21 by yselivanov, last changed 2016-11-08 20:15 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
fix_aiter.patch yselivanov, 2016-06-06 19:21 review
fix_aiter2.patch yselivanov, 2016-06-07 00:25 review
fix_aiter3.patch yselivanov, 2016-06-08 15:35 review
Messages (22)
msg267544 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-06 19:21
There is a small flaw in PEP 492 design -- __aiter__ should not return an awaitable object that resolves to an asynchronous iterator. It should return an asynchronous iterator directly.

Let me explain this by showing some examples.

I've discovered this while working on a new asynchronous generators PEP.  Let's pretend that we have them already: if we have a 'yield' expression in an 'async def' function, the function becomes an "asynchronous generator function":

   async def foo():
      await bar()
      yield 1
      await baz()
      yield 2

   # foo -- is an `asynchronous generator function`
   # foo() -- is an `asynchronous generator`

If we iterate through "foo()", it will await on "bar()", yield "1", await on "baz()", and yield "2":

   >>> async for el in foo():
   ...     print(el)
   1
   2

If we decide to have a class with an __aiter__ that is an async generator, we'd write something like this:

   class Foo:
      async def __aiter__(self):
          await bar()
          yield 1
          await baz()
          yield 2

However, with the current PEP 492 design, the above code would be invalid!  The interpreter expects __aiter__ to return a coroutine, not an async generator.

I'm still working on the PEP for async generators, targeting CPython 3.6.  And once it is ready, it might still be rejected or deferred.  But in any case, this PEP 492 flaw has to be fixed now, in 3.5.2 (since PEP 492 is provisional).

The attached patch fixes the __aiter__ in a backwards compatible way:

1. ceval/GET_AITER opcode calls the __aiter__ method.

2. If the returned object has an '__anext__' method, GET_AITER silently wraps it in an awaitable, which is equivalent to the following coroutine:

    async def wrapper(aiter_result):
        return aiter_result

3. If the returned object does not have an '__anext__' method, a DeprecationWarning is raised.
msg267547 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-06 20:27
While I agree this needs to be fixed, one key piece of documentation needed will be to cover how to write an __aiter__ method that does the right thing on both 3.5.1 and 3.5.2+ (and also avoids the deprecation warning in the latter case).

(I've also added Larry to the cc list as release manager)
msg267548 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-06-06 20:31
Would it be easier to handle for everyone if this did not vary between
3.5.0/1 and 3.5.2, and instead was an incompatibility in 3.6? (That would
still be allowed given 492's provisional status.)
msg267551 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-06 20:46
Since the old behaviour is only deprecated with Yury's changes, rather than disallowed entirely, I think it makes sense to provide a 3.5.x version that also supports the new behaviour.

In addition to my docs comment above, the other thing I noticed in reviewing the patch is that the proposed tests don't currently check that we *don't* emit a deprecation warning for the newly permitted forward compatible cases that return an asynchronous iterator (rather than an awaitable) directly from __aiter__.

There are definitely some tests that exercise that path, so we could rely on the general principle of "the test suite shouldn't emit deprecation warnings", but we could also add some tests that specifically check that no warning is emitted in that case.
msg267557 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-06-06 20:57
As RM my default position is naturally "don't change behavior in point releases".  I'm willing to be overruled by the BDFL, less so by anybody else.
msg267559 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-06 20:58
> While I agree this needs to be fixed, one key piece of documentation needed will be to cover how to write an __aiter__ method that does the right thing on both 3.5.1 and 3.5.2+ (and also avoids the deprecation warning in the latter case).

Oh, this is tough.

How about we do this:

1. In 3.5.2 we start to support new behaviour. We raise PendingDeprecationWarning.  We update documentation and PEP 492.  The recommended way to write 3.5 code is to keep returning awaitables from __aiter__.  We can add a snippet of code for a compatibility decorator to the docs.

2. In 3.6 we start to raise DeprecationWarning.  The recommended way to write code for 3.6 & 3.7 is to return async iterators from __aiter__.

3. In 3.7 we remove support of the old behaviour.

It's a rather long & conservative process, but it will be
msg267560 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-06 21:00
It's a rather long & conservative process, but it will be easier for people to migrate their code.
msg267561 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-06-06 21:26
Unless Nick disagrees (or unless we can't figure out how to implement it) I
think Yury's proposal makes sense. So if Larry is asking for my fiat, this
is it.
msg267563 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-06-06 21:48
Okay.

I'm hoping to not delay 3.5.2 RC1, and the schedule calls for me to tag the release Saturday afternoon.  Can you guys get this solid and checked in before then?
msg267565 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-06 22:04
> Can you guys get this solid and checked in before then?

Will do my best.  I'll update the patch soon with some code comments and docs.
msg267572 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-07 00:25
Updated patch (fix_aiter2.patch)
msg267577 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-07 00:50
Yury's proposal sounds good to me - I'll have time to do a proper review tomorrow (at a quick glance, my one suggestion is to move the if statement outside the example decorator, so the decorator is defined differently based on the Python version, rather than checking the version when called)
msg267865 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-08 15:35
Nick,

Please see the updated patch.  Do you think it's ready to be merged?  I want buildbots to have some time with it before RC.
msg268035 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-09 16:39
+1 from me - my only comments were on the docs updates and one of the explanatory comments in the code.
msg268054 - (view) Author: Roundup Robot (python-dev) Date: 2016-06-09 19:13
New changeset 93ad47d63b87 by Yury Selivanov in branch '3.5':
Issue #27243: Fix __aiter__ protocol
https://hg.python.org/cpython/rev/93ad47d63b87

New changeset 9ff95c30a38e by Yury Selivanov in branch 'default':
Merge 3.5 (issue #27243)
https://hg.python.org/cpython/rev/9ff95c30a38e
msg268055 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-09 19:14
Thanks a lot, Nick!  I've merged the patch.
msg268063 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-09 22:20
I've also updated PEP 492: https://hg.python.org/peps/rev/fef4b9969b9d  

Please feel free to post to this issue if you think that I should have covered it differently or in more detail.
msg268089 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-10 05:42
Test suite emits a new warning, and fails under python -Werror:

======================================================================
ERROR: test_readline (test.test_asyncio.test_pep492.StreamReaderTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/disk/home/proj/python/cpython/Lib/test/test_asyncio/test_pep492.py", line 89, in test_readline
    data = self.loop.run_until_complete(reader())
  File "/media/disk/home/proj/python/cpython/Lib/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/media/disk/home/proj/python/cpython/Lib/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/media/disk/home/proj/python/cpython/Lib/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/media/disk/home/proj/python/cpython/Lib/test/test_asyncio/test_pep492.py", line 85, in reader
    async for line in stream:
PendingDeprecationWarning: 'StreamReader' implements legacy __aiter__ protocol; __aiter__ should return an asynchronous iterator, not awaitable
msg268117 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-10 12:05
This is because sys.version_info is 3.5.1 (not 3.5.2 yet) in the "3.5" branch.  Once 3.5.2 RC is tagged the warning will disappear.

https://github.com/python/cpython/blob/master/Lib/asyncio/streams.py#L693
msg268184 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-11 05:26
I didn’t realize, sorry for the noise
msg268216 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-06-11 15:17
> I didn’t realize, sorry for the noise

Actually thanks for reporting this, Martin.  I didn't realize that sys.version_info was 3.5.1 in 3.5 branch.
msg280343 - (view) Author: Roundup Robot (python-dev) Date: 2016-11-08 20:15
New changeset a0d50aad7b02 by Yury Selivanov in branch '3.6':
Issue #27243: Change PendingDeprecationWarning -> DeprecationWarning.
https://hg.python.org/cpython/rev/a0d50aad7b02

New changeset 9550f0d22d27 by Yury Selivanov in branch 'default':
Merge 3.6 (issue #27243)
https://hg.python.org/cpython/rev/9550f0d22d27
History
Date User Action Args
2016-11-08 20:15:49python-devsetmessages: + msg280343
2016-06-11 15:17:40yselivanovsetmessages: + msg268216
2016-06-11 05:26:38martin.pantersetstatus: open -> closed

messages: + msg268184
2016-06-10 12:05:53yselivanovsetmessages: + msg268117
2016-06-10 05:42:25martin.pantersetstatus: closed -> open
nosy: + martin.panter
messages: + msg268089

2016-06-09 22:20:23yselivanovsetmessages: + msg268063
2016-06-09 19:14:45yselivanovsetstatus: open -> closed
type: enhancement
messages: + msg268055

resolution: fixed
stage: resolved
2016-06-09 19:13:30python-devsetnosy: + python-dev
messages: + msg268054
2016-06-09 16:39:24ncoghlansetmessages: + msg268035
2016-06-08 15:35:50yselivanovsetfiles: + fix_aiter3.patch

messages: + msg267865
2016-06-07 00:50:03ncoghlansetmessages: + msg267577
2016-06-07 00:25:35yselivanovsetfiles: + fix_aiter2.patch

messages: + msg267572
2016-06-06 22:58:56vstinnersetnosy: - vstinner
2016-06-06 22:04:23yselivanovsetmessages: + msg267565
2016-06-06 21:48:10larrysetmessages: + msg267563
2016-06-06 21:26:57gvanrossumsetmessages: + msg267561
2016-06-06 21:00:12yselivanovsetmessages: + msg267560
2016-06-06 20:58:43yselivanovsetmessages: + msg267559
2016-06-06 20:57:42larrysetmessages: + msg267557
2016-06-06 20:46:04ncoghlansetmessages: + msg267551
2016-06-06 20:31:34gvanrossumsetnosy: + ned.deily
messages: + msg267548
2016-06-06 20:27:56ncoghlansetnosy: + larry
messages: + msg267547
2016-06-06 19:21:36yselivanovcreate