classification
Title: Asynchronous comprehensions don't work in asyncio REPL
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, asvetlov, hroncok, jack1142, lukasz.langa, mbussonn, miss-islington, pablogsal, serhiy.storchaka, vstinner
Priority: release blocker Keywords: patch

Created on 2020-02-05 17:28 by jack1142, last changed 2020-07-06 20:18 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18968 merged BTaskaya, 2020-03-12 19:19
PR 19010 merged pablogsal, 2020-03-15 03:59
PR 19070 merged miss-islington, 2020-03-19 11:36
PR 19230 merged BTaskaya, 2020-03-30 13:51
PR 19835 merged pablogsal, 2020-05-01 11:42
PR 19838 merged pablogsal, 2020-05-01 15:23
Messages (27)
msg361443 - (view) Author: (jack1142) * Date: 2020-02-05 17:28
asyncio REPL doesn't allow using asynchronous comprehensions outside of async func. Same behavior can also be observed when using `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` flag in `compile()`

Example with `async for`:
>>> async def async_gen():
...     for x in range(5):
...         yield await asyncio.sleep(1, x)
... 
>>> [x async for x in async_gen()]
  File "<console>", line 0
SyntaxError: asynchronous comprehension outside of an asynchronous function


Example with `await`:
>>> [await asyncio.sleep(1, x) for x in range(5)]
  File "<console>", line 0
SyntaxError: asynchronous comprehension outside of an asynchronous function
msg361445 - (view) Author: (jack1142) * Date: 2020-02-05 17:33
I also noticed that putting `await` before the async comprehension will make the code inside the comprehension run (though after it runs, it will fail on awaiting list):

>>> await [await asyncio.sleep(1, print(x)) for x in range(5)] 
0
1
2
3
4
Traceback (most recent call last):
  File "C:\Python38\lib\concurrent\futures\_base.py", line 439, in result
    return self.__get_result()
  File "C:\Python38\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
TypeError: object list can't be used in 'await' expression
msg364600 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-03-19 11:35
New changeset 9052f7a41b90f2d34011c8da68f9a4facebc8a97 by Batuhan Taşkaya in branch 'master':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
https://github.com/python/cpython/commit/9052f7a41b90f2d34011c8da68f9a4facebc8a97
msg364601 - (view) Author: miss-islington (miss-islington) Date: 2020-03-19 11:54
New changeset ec8a973f7cf080d9c0679f058b2371f0b7c7862c by Miss Islington (bot) in branch '3.8':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
https://github.com/python/cpython/commit/ec8a973f7cf080d9c0679f058b2371f0b7c7862c
msg365311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-30 13:17
This change caused python-gmpy2 tests to fail:
https://bugzilla.redhat.com/show_bug.cgi?id=1817710

Reproducer:
---
from __future__ import print_function, division
import doctest

filename = "doctest.txt"
with open(filename, "w") as fp:
    print("""
Test
====

    >>> all(x == 1 for x in [1, 1, 1])
    True
    """.strip(), file=fp)

result = doctest.testfile(filename, globs=globals())
print("result:", result)
---
msg365312 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-03-30 13:28
Both PyCF_ALLOW_TOP_LEVEL_AWAIT and CO_FUTURE_DIVISION has same value.
msg365313 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-03-30 13:39
Sending a patch that would prevent this collision.
msg365347 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-30 20:23
If CO_FUTURE_DIVISION conflicts with PyCF_ALLOW_TOP_LEVEL_AWAIT, does not CO_ITERABLE_COROUTINE conflict with PyCF_SOURCE_IS_UTF8 and CO_ASYNC_GENERATOR with PyCF_DONT_IMPLY_DEDENT?
msg365881 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-04-06 22:34
#define IS_COMPILER_FLAG_ENABLED(c, flag) printf("%s: %d\n", #flag, c->c_flags->cf_flags & flag)

> If CO_FUTURE_DIVISION conflicts with PyCF_ALLOW_TOP_LEVEL_AWAIT, does not CO_ITERABLE_COROUTINE conflict with PyCF_SOURCE_IS_UTF8 and CO_ASYNC_GENERATOR with PyCF_DONT_IMPLY_DEDENT?

Yes, they do.

Compiling without anything
PyCF_SOURCE_IS_UTF8: 256
CO_ITERABLE_COROUTINE: 256
PyCF_DONT_IMPLY_DEDENT: 0
CO_ASYNC_GENERATOR: 0
Compiling with CO_ASYNC_GENERATOR
PyCF_SOURCE_IS_UTF8: 256
CO_ITERABLE_COROUTINE: 256
PyCF_DONT_IMPLY_DEDENT: 512
CO_ASYNC_GENERATOR: 512

This result is a side affect of merging future flags with compiler flags. Even if we access from cf_flags (or the other way around, ff_features) it doesnt change anything because we are merging both flags before we start the process. 

Two ways of escaping this is changing flags to not to conlict with each other or not merging. Not merging is out of this box because it will break user level compile function (it takes both flags in a single parameter, flags). The most reasonable solution I thought was making this flags not to conflict with each other.
msg366838 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-20 15:04
Is there an update of this *release blocker* issue? Should we revert the commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97?
msg366841 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-04-20 15:13
> Is there an update of this *release blocker* issue? Should we revert the commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97?

I dont think we can gain much by reverting 9052f7a41b90f2d34011c8da68f9a4facebc8a97 because it doesn't introduce a new issue, it just adds "another" path to notice it. 

Test:
>>> await __import__('asyncio').sleep(1)

Expected (and the behavior of 3.6):

(.venv) [  6:09PM ]  [ isidentical@threeheadedgiant:~ ]
 $ python3.6 z.py       
**********************************************************************
File "doctest.txt", line 4, in doctest.txt
Failed example:
    await __import__('asyncio').sleep(1)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest doctest.txt[0]>", line 1
        await __import__('asyncio').sleep(1)
                       ^
    SyntaxError: invalid syntax
**********************************************************************
1 items had failures:
   1 of   1 in doctest.txt
***Test Failed*** 1 failures.
result: TestResults(failed=1, attempted=1

Current master (+ reverted GH 18968):
(.venv) [  6:09PM ]  [ isidentical@threeheadedgiant:~ ]
 $ ./cpython/python z.py
/home/isidentical/cpython/Lib/doctest.py:1336: RuntimeWarning: coroutine '<module>' was never awaited
  exec(compile(example.source, filename, "single",
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
result: TestResults(failed=0, attempted=1)
msg366843 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-04-20 15:18
GH 19230:
(.venv) [  6:17PM ]  [ isidentical@threeheadedgiant:~ ]
 $ ./cpython/python z.py
**********************************************************************
File "/home/isidentical/doctest.txt", line 4, in doctest.txt
Failed example:
    await __import__('asyncio').sleep(1)
Exception raised:
    Traceback (most recent call last):
      File "/home/isidentical/cpython/Lib/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest doctest.txt[0]>", line 1
    SyntaxError: 'await' outside function
**********************************************************************
1 items had failures:
   1 of   1 in doctest.txt
***Test Failed*** 1 failures.
result: TestResults(failed=1, attempted=1)
msg366849 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-20 16:05
Either the regression should be fixed, or the commits which introduced the regression should be reverted.

I'm talking about python-gmpy2 doctest which pass on Python 3.8: msg365311.
msg366851 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-20 16:08
> Either the regression should be fixed, or the commits which introduced the regression should be reverted.

If you do that we will have now two problems:

* The asyncio repr will not work correctly.
* The flags in compile still have a clash.

The solution is to merge https://github.com/python/cpython/pull/19230 or some version of it, which should fix the issue in  python-gmpy2 doctest .
msg367015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-22 16:09
New changeset 4454057269b995341b04d13f0bf97f96080f27d0 by Batuhan Taşkaya in branch 'master':
bpo-39562: Prevent collision of future and compiler flags (GH-19230)
https://github.com/python/cpython/commit/4454057269b995341b04d13f0bf97f96080f27d0
msg367018 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-22 16:34
> bpo-39562: Prevent collision of future and compiler flags (GH-19230)
> https://github.com/python/cpython/commit/4454057269b995341b04d13f0bf97f96080f27d0

I tested manually: this change fix my msg365311 reproducer. Thanks!

In Python 3.8, PyCF_ALLOW_TOP_LEVEL_AWAIT = CO_FUTURE_DIVISION = 0x2000. The commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97 was backported to 3.8:

New changeset ec8a973f7cf080d9c0679f058b2371f0b7c7862c by Miss Islington (bot) in branch '3.8':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
https://github.com/python/cpython/commit/ec8a973f7cf080d9c0679f058b2371f0b7c7862c

And now 3.8 branch also has the bug: msg365311 reproducer fails as well in 3.8.

What should be done? Revert ec8a973f7cf080d9c0679f058b2371f0b7c7862c? Change constants value in Python 3.8.3? (backport 4454057269b995341b04d13f0bf97f96080f27d0 to 3.8)

IMHO reverting the commit ec8a973f7cf080d9c0679f058b2371f0b7c7862c (fix async comprehension in asyncio REPL) is the safest option.

First, I understood that the asyncio REPL was experimental in Python 3.8: https://bugs.python.org/issue37028

But it's not documented as provisional or experimental in What's New in Python 3.8:
https://docs.python.org/dev/whatsnew/3.8.html#asyncio

But I'm also fine with backporting the fix 4454057269b995341b04d13f0bf97f96080f27d0 (change constants) to 3.8.
msg367734 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-30 04:52
I think we should backport the fix 4454057269b995341b04d13f0bf97f96080f27d0 (change constants) to 3.8 given that the likelihood of users using the actual hardcoded value of the constants instead of the constants themselves is very low. Also, given the collision, it would be fixing a bug present still in 3.8. 

If we revert 9052f7a41b90f2d34011c8da68f9a4facebc8a97 we would have two bugs in 3.8: collision of constants and asyncio repr not working properly.
msg367847 - (view) Author: miss-islington (miss-islington) Date: 2020-05-01 14:18
New changeset 5055c274c6e4f2bb8025910dedf0ff89f4bdd170 by Pablo Galindo in branch '3.8':
[3.8] bpo-39562: Prevent collision of future and compiler flags (GH-19230) (GH-19835)
https://github.com/python/cpython/commit/5055c274c6e4f2bb8025910dedf0ff89f4bdd170
msg367855 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 15:36
New changeset 71e6122b4428ae43e868e34db4f072635f58a555 by Pablo Galindo in branch '3.8':
bpo-39562: Correctly updated the version section in the what's new document (GH-19838)
https://github.com/python/cpython/commit/71e6122b4428ae43e868e34db4f072635f58a555
msg367890 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-01 21:44
Fedora packagers report that this problem is now showing up in 3.8.3rc1. What can be done to ensure that 3.8.3 final will contain the fix?
msg367892 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 22:42
The merge is in 3.8 master, so we need to make sure that Łukasz includes this on the 3.8.3 release. Victor sent an email to python-dev already about this issue. If you want to make absolutely sure this happens, maybe send an email directly to Łukasz.
msg367897 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-01 23:21
I just did.
msg368334 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-07 12:12
I reopen the bug to ensure that the fix will land into Python 3.8.3. The issue is marked as release blocker.
msg368835 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-14 12:56
I tested manually the just released Python 3.8.3 with msg365311: I confirm that it's fixed. Thanks!
msg373131 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-07-06 15:05
This make non-await list comprehension coroutine-code-objects as well: 

https://bugs.python.org/issue41218

    import ast
    import inspect
    cell = '[x for x in l]'
    code = compile(cell, "<>", "exec", flags=getattr(ast,'PyCF_ALLOW_TOP_LEVEL_AWAIT', 0x0))

    inspect.CO_COROUTINE & code.co_flags == inspect.CO_COROUTINE  # this is now TRUE. 

This leads to weird things in Jupyter/IPython when we try to detect wether a block of code is, or os not async.
msg373159 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-07-06 19:27
Pablo, I see you reopened and marked as blocker, 

As the changes here has been released maybe you want to reclose and marked https://bugs.python.org/issue41218 as blocker it should also be fixed by
https://github.com/python/cpython/pull/21357
msg373168 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-07-06 20:18
> As the changes here has been released maybe you want to reclose and marked https://bugs.python.org/issue41218 as blocker it should also be fixed by
https://github.com/python/cpython/pull/21357

Thanks for the heads up, Matthias!

Yes, I think is the best thing to do. For some reason, I thought this was not still released :(
History
Date User Action Args
2020-07-06 20:18:05pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg373168
2020-07-06 19:27:57mbussonnsetmessages: + msg373159
2020-07-06 15:33:19pablogsalsetpriority: release blocker
2020-07-06 15:32:52pablogsalsetstatus: closed -> open
resolution: fixed -> (no value)
2020-07-06 15:05:55mbussonnsetnosy: + mbussonn
messages: + msg373131
2020-05-14 12:56:45vstinnersetstatus: open -> closed
priority: release blocker -> (no value)
resolution: fixed
messages: + msg368835
2020-05-07 12:12:45vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg368334
2020-05-01 23:21:18hroncoksetmessages: + msg367897
2020-05-01 22:44:25yselivanovsetnosy: - yselivanov
2020-05-01 22:42:38pablogsalsetmessages: + msg367892
2020-05-01 21:44:32hroncoksetnosy: + hroncok
messages: + msg367890
2020-05-01 15:36:59pablogsalsetmessages: + msg367855
2020-05-01 15:23:45pablogsalsetpull_requests: + pull_request19156
2020-05-01 14:24:57pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-05-01 14:18:30miss-islingtonsetmessages: + msg367847
2020-05-01 11:42:54pablogsalsetpull_requests: + pull_request19153
2020-04-30 04:52:17pablogsalsetmessages: + msg367734
2020-04-22 16:34:52vstinnersetmessages: + msg367018
2020-04-22 16:09:06vstinnersetmessages: + msg367015
2020-04-20 16:08:50pablogsalsetmessages: + msg366851
2020-04-20 16:05:44vstinnersetmessages: + msg366849
2020-04-20 15:18:25BTaskayasetmessages: + msg366843
2020-04-20 15:13:30BTaskayasetmessages: + msg366841
2020-04-20 15:04:26vstinnersetmessages: + msg366838
2020-04-06 22:34:56BTaskayasetmessages: + msg365881
2020-03-30 20:23:37serhiy.storchakasetmessages: + msg365347
2020-03-30 13:51:26BTaskayasetstage: resolved -> patch review
pull_requests: + pull_request18590
2020-03-30 13:39:45BTaskayasetmessages: + msg365313
2020-03-30 13:28:22BTaskayasetmessages: + msg365312
2020-03-30 13:20:31vstinnersetpriority: normal -> release blocker
nosy: + lukasz.langa
2020-03-30 13:17:19vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg365311

resolution: fixed -> (no value)
2020-03-19 12:00:48BTaskayasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-03-19 11:54:20miss-islingtonsetmessages: + msg364601
2020-03-19 11:36:06miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request18425
2020-03-19 11:35:47pablogsalsetmessages: + msg364600
2020-03-15 03:59:05pablogsalsetpull_requests: + pull_request18357
2020-03-15 03:58:26pablogsalsetpull_requests: - pull_request18355
2020-03-15 03:52:58pablogsalsetnosy: + pablogsal
pull_requests: + pull_request18355
2020-03-12 19:19:07BTaskayasetkeywords: + patch
stage: patch review
pull_requests: + pull_request18316
2020-02-14 21:11:17BTaskayasetnosy: + BTaskaya
2020-02-05 20:47:03serhiy.storchakasetnosy: + serhiy.storchaka
2020-02-05 17:33:21jack1142setmessages: + msg361445
2020-02-05 17:28:25jack1142create