classification
Title: Syntax error with async generator inside dictionary comprehension
Type: behavior Stage: patch review
Components: asyncio Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, ericvw, gvanrossum, josh.r, levkivskyi, njs, pablogsal, serhiy.storchaka, xtreak, yselivanov
Priority: normal Keywords: patch

Created on 2018-04-24 11:05 by pablogsal, last changed 2020-06-30 18:51 by pablogsal.

Pull Requests
URL Status Linked Edit
PR 6766 open serhiy.storchaka, 2018-05-11 09:46
Messages (19)
msg315692 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-04-24 11:05
Given this async function:

async def elements(n):
    yield n
    yield n*2
    yield n*3
    yield n*4

This definition is considered invalid:

async def test():
    return { n: [x async for x in elements(n)] for n in range(3)}

SyntaxError: asynchronous comprehension outside of an asynchronous function

The reason (I suspect) is because the dict comprehension is not an async context (it would be if range was an async iterable and we use `async for n in range(3)`). 

Is this expected behaviour or something it needs to change?
msg316115 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-05-03 09:48
My guess this is a consequence of the implicit function scope in comprehensions, see https://bugs.python.org/issue3692

I would say a proper solution would be to drop the implicit function scope in favour of other mechanisms, but this will require some work and discussions.
msg316390 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-11 09:52
I think this can be fixed simpler. Currently a comprehension become asynchronous in two cases:

1. Explicitly, when it contains 'async for'. This is visible at AST level.

2. Implicitly, when any of inner expressions contains 'await'.

But asynchronous comprehensions should behave the same way as 'await'. I think that a comprehension should be made implicitly asynchronous if any of inner expressions contains explicit or implicit asynchronous comprehension. This is implemented in PR 6766.
msg316398 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-05-11 15:13
@Ivan: Please stop bringing up that we should drop the implicit scope for comprehensions. I know you feel this way, but it's not going to happen.

@Serhiy: What does the PEP for async/await say? (Or is there a separate PEP for allowing async/await inside comprehensions?)

@Yury: Your thoughts?

I do think the code from the OP's example should be expected to work.

Does it / should it work the same way for generator expressions?
msg316404 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-11 16:31
There are several related PEPs:

PEP 492 -- Coroutines with async and await syntax
PEP 525 -- Asynchronous Generators
PEP 530 -- Asynchronous Comprehensions

I haven't found anything explicit about this case. PEP 492 says that just "await" is not enough for converting a function into a coroutine. Explicit "async def" is required. PEP 530 says nothing about implementation details, it omits the fact that comprehensions are implemented by creating and calling an implicit function. From the implementation's point of view PEP 530 means that "async for" and "await" inside an implicit function make it an asynchronous function, and implicit "await" is added in the place of it's call. The natural extension of this is than an implicit "await" should have the same effect as explicit "await", in particular it should make an outer implicit function an asynchronous function and add other implicit "await" in the place of it's call, and so forth. But all this is implied, and is not said explicitly.

I don't understand one paragraph in PEP 530:

"""
In principle, asynchronous generator expressions are allowed in
any context.  However, in Python 3.6, due to ``async`` and ``await``
soft-keyword status, asynchronous generator expressions are only
allowed in an ``async def`` function.  Once ``async`` and ``await``
become reserved keywords in Python 3.7, this restriction will be
removed.
"""

Does it mean that even more restrictions should be removed than PR 6766 does? And what is the relation between this restriction and making "async" and "await" reserved keywords?
msg316549 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-14 18:47
[Serhiy]

> But asynchronous comprehensions should behave the same way as 'await'. I think that a comprehension should be made implicitly asynchronous if any of inner expressions contains explicit or implicit asynchronous comprehension. This is implemented in PR 6766.

[Guido]

> @Yury: Your thoughts?

> I do think the code from the OP's example should be expected to work.


I agree with Serhiy and I like his proposal. Essentially, a comprehension is asynchronous when it contains an "await" or an "async for" in it. We want to add another case: make it async when any of its inner-expressions is an async comprehension.  Essentially:

    [f: [x async for x in f(x)] for f in fs]

The nested comprehension is obviously asynchronous, so the outer comprehension should become asynchronous too.  I think this is a fairly obvious and easy to follow semantics.

Guido, if you agree that this is a reasonable proposition I can update PEP 530 about this new behaviour (for Python 3.8) and review Serhiy's PR.
msg316561 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-05-14 19:38
> [f: [x async for x in f(x)] for f in fs]

Did you mean {} for the outer brackets intead of []?

I think it is reasonable that if the presence of 'async for' or 'await' in a comprehension makes it async, then this should also apply if that comprehension is nested inside another.

All of these should only be allowed inside 'async def' though, right?
msg316584 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-14 21:28
> Did you mean {} for the outer brackets intead of []?

Yes, my bad.

> All of these should only be allowed inside 'async def' though, right?

Yep, except async generator expressions which are allowed to appear in synchronous contexts, e.g.:

   def foo():
       return (i async for i in ai)

(this already works in 3.7).
msg316590 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-05-14 22:06
Is there any problem that is solved by allowing this example? The asymmetry
with using [...async for...] in the same position (which is not allowed)
worries me. Do you have a real use case where it's clearer to write an
async generator expression instead of a nested async function?

def foo():
    async def inner():
        async for i in ai:
            yield i
    return inner

I would encourage you to think about various ways of nesting async
generator expressions and comprehensions to see if you can poke a hole in
the design.
msg321285 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-08 18:34
PR 6766 adds numerous examples of nesting async
generator expressions and comprehensions as tests.
msg326089 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-09-22 13:43
What is the status of this issue?

Similar change to the Grammar was just merged in issue32117. Both issue32117 and this issue extend already implemented PEPs (PEP 448 and PEP 525 correspondingly) to the corner case missed in the original PEP.

Pablo, Yury, could you please start a discussion about this on the Pythod-Dev mailing list?
msg337146 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-03-04 20:17
Hm, I think that if Yury still thinks this is a good idea you two should just do this. No need to write a PEP or wait for the Steering Committee. Unless there's a downside? Does it break real existing code?
msg337150 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-03-04 21:16
There are some tricky subtleties here around the distinction between list/dict/set comprehensions and generator expressions.

For list/dict/set comprehensions, they're evaluated eagerly, so an async comprehension can only occur in async context. For generator expressions, they're evaluated lazily, so you can write an async generator expression inside a non-async context. *Consuming* the async generator will require an async context, but all the expression itself does is instantiate an object, and that's synchronous.

(Like Guido, I'm not 100% sure that it's useful to support async generator expressions inside sync contexts, but we've already shipped it, so I'll assume we're going to keep it.)

So, I would expect the rule to be, precisely: if an async list/dict/set comprehension occurs inside either a list/dict/set comprehension or a generator expression, that should force the enclosing expression to become async.

So this is a synchronous comprehension, even though it has an async generator expression in it, and must occur inside an 'async def':

[(i async for i in range(j)) for j in range(n)]

And this is an async generator expression, which can legally be placed inside a regular 'def' (but can only be consumed inside an 'async def'):

([i async for i in range(j)] for j in range(n))

This might be what Yury/Serhiy/etc. already had in mind, but it's complicated enough that it seemed like it's worth spelling out in detail...
msg337340 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-03-06 21:30
Yes, I have exactly the same thoughts as Nathaniel about this.  The bug should be fixed, and the algorithm should be as follows (quoting Nathaniel):

> So, I would expect the rule to be, precisely: if an async list/dict/set comprehension occurs inside either a list/dict/set comprehension or a generator expression, that should force the enclosing expression to become async.
msg337363 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-03-07 06:47
It is what PR 6766 implements, Nathaniel.

Except that your first example (an asynchronous generator in a synchronous comprehensions) is allowed in the current code and can occur inside a synchronous function.
msg337364 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-03-07 06:54
> Except that your first example (an asynchronous generator in a synchronous comprehensions) is allowed in the current code and can occur inside a synchronous function.

Oh yeah, you're right of course.
msg344194 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-01 17:27
Ping, should we include this in beta1 or as is a "bugfix" this can be backported?
msg372592 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-06-29 15:17
What can I do to move this issue forward?

We already missed 3.7, 3.8 and 3.9.
msg372717 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-30 18:51
Yeah, I am +1 on moving this and on the current solution in PR6766
History
Date User Action Args
2020-06-30 18:51:07pablogsalsetmessages: + msg372717
2020-06-29 15:17:04serhiy.storchakasetmessages: + msg372592
2020-06-29 14:58:26serhiy.storchakalinkissue41159 superseder
2019-06-01 17:27:57pablogsalsetmessages: + msg344194
2019-03-07 06:54:17njssetmessages: + msg337364
2019-03-07 06:47:30serhiy.storchakasetmessages: + msg337363
2019-03-06 21:42:42josh.rsetnosy: + josh.r
2019-03-06 21:30:15yselivanovsetmessages: + msg337340
2019-03-04 21:16:45njssetnosy: + njs
messages: + msg337150
2019-03-04 20:17:13gvanrossumsetnosy: + gvanrossum
messages: + msg337146
2018-09-22 15:39:17xtreaksetnosy: + xtreak
2018-09-22 13:43:22serhiy.storchakasetmessages: + msg326089
2018-07-08 22:40:48gvanrossumsetnosy: - gvanrossum
2018-07-08 18:34:16serhiy.storchakasetmessages: + msg321285
2018-05-14 22:06:04gvanrossumsetmessages: + msg316590
2018-05-14 21:28:58yselivanovsetmessages: + msg316584
2018-05-14 19:38:35gvanrossumsetmessages: + msg316561
2018-05-14 18:47:06yselivanovsetmessages: + msg316549
2018-05-11 16:31:09serhiy.storchakasetmessages: + msg316404
2018-05-11 15:13:54gvanrossumsetmessages: + msg316398
2018-05-11 09:52:19serhiy.storchakasetnosy: + serhiy.storchaka, gvanrossum
messages: + msg316390
2018-05-11 09:46:17serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request6452
2018-05-03 09:48:38levkivskyisetnosy: + levkivskyi
messages: + msg316115
2018-04-24 19:24:04ericvwsetnosy: + ericvw
2018-04-24 11:05:41pablogsalcreate