asyncio REPL doesn't allow using asynchronous comprehensions outside of async func. Same behavior can also be observed when using `ast.PyCF_ALLOW_TOP_LEVEL_AWAIT` flag in `compile()`

Example with `async for`:
>>> async def async_gen():
...     for x in range(5):
...         yield await asyncio.sleep(1, x)
... 
>>> [x async for x in async_gen()]
  File "<console>", line 0
SyntaxError: asynchronous comprehension outside of an asynchronous function


Example with `await`:
>>> [await asyncio.sleep(1, x) for x in range(5)]
  File "<console>", line 0
SyntaxError: asynchronous comprehension outside of an asynchronous function

I also noticed that putting `await` before the async comprehension will make the code inside the comprehension run (though after it runs, it will fail on awaiting list):

>>> await [await asyncio.sleep(1, print(x)) for x in range(5)] 
0
1
2
3
4
Traceback (most recent call last):
  File "C:\Python38\lib\concurrent\futures\_base.py", line 439, in result
    return self.__get_result()
  File "C:\Python38\lib\concurrent\futures\_base.py", line 388, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
TypeError: object list can't be used in 'await' expression

New changeset 9052f7a41b90f2d34011c8da68f9a4facebc8a97 by Batuhan Taşkaya in branch 'master':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
https://github.com/python/cpython/commit/9052f7a41b90f2d34011c8da68f9a4facebc8a97

New changeset ec8a973f7cf080d9c0679f058b2371f0b7c7862c by Miss Islington (bot) in branch '3.8':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
https://github.com/python/cpython/commit/ec8a973f7cf080d9c0679f058b2371f0b7c7862c

This change caused python-gmpy2 tests to fail:
https://bugzilla.redhat.com/show_bug.cgi?id=1817710

Reproducer:
---
from __future__ import print_function, division
import doctest

filename = "doctest.txt"
with open(filename, "w") as fp:
    print("""
Test
====

    >>> all(x == 1 for x in [1, 1, 1])
    True
    """.strip(), file=fp)

result = doctest.testfile(filename, globs=globals())
print("result:", result)
---

Both PyCF_ALLOW_TOP_LEVEL_AWAIT and CO_FUTURE_DIVISION has same value.

Sending a patch that would prevent this collision.

If CO_FUTURE_DIVISION conflicts with PyCF_ALLOW_TOP_LEVEL_AWAIT, does not CO_ITERABLE_COROUTINE conflict with PyCF_SOURCE_IS_UTF8 and CO_ASYNC_GENERATOR with PyCF_DONT_IMPLY_DEDENT?

#define IS_COMPILER_FLAG_ENABLED(c, flag) printf("%s: %d\n", #flag, c->c_flags->cf_flags & flag)

> If CO_FUTURE_DIVISION conflicts with PyCF_ALLOW_TOP_LEVEL_AWAIT, does not CO_ITERABLE_COROUTINE conflict with PyCF_SOURCE_IS_UTF8 and CO_ASYNC_GENERATOR with PyCF_DONT_IMPLY_DEDENT?

Yes, they do.

Compiling without anything
PyCF_SOURCE_IS_UTF8: 256
CO_ITERABLE_COROUTINE: 256
PyCF_DONT_IMPLY_DEDENT: 0
CO_ASYNC_GENERATOR: 0
Compiling with CO_ASYNC_GENERATOR
PyCF_SOURCE_IS_UTF8: 256
CO_ITERABLE_COROUTINE: 256
PyCF_DONT_IMPLY_DEDENT: 512
CO_ASYNC_GENERATOR: 512

This result is a side affect of merging future flags with compiler flags. Even if we access from cf_flags (or the other way around, ff_features) it doesnt change anything because we are merging both flags before we start the process. 

Two ways of escaping this is changing flags to not to conlict with each other or not merging. Not merging is out of this box because it will break user level compile function (it takes both flags in a single parameter, flags). The most reasonable solution I thought was making this flags not to conflict with each other.

Is there an update of this *release blocker* issue? Should we revert the commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97?

> Is there an update of this *release blocker* issue? Should we revert the commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97?

I dont think we can gain much by reverting 9052f7a41b90f2d34011c8da68f9a4facebc8a97 because it doesn't introduce a new issue, it just adds "another" path to notice it. 

Test:
>>> await __import__('asyncio').sleep(1)

Expected (and the behavior of 3.6):

(.venv) [  6:09PM ]  [ isidentical@threeheadedgiant:~ ]
 $ python3.6 z.py       
**********************************************************************
File "doctest.txt", line 4, in doctest.txt
Failed example:
    await __import__('asyncio').sleep(1)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest doctest.txt[0]>", line 1
        await __import__('asyncio').sleep(1)
                       ^
    SyntaxError: invalid syntax
**********************************************************************
1 items had failures:
   1 of   1 in doctest.txt
***Test Failed*** 1 failures.
result: TestResults(failed=1, attempted=1

Current master (+ reverted GH 18968):
(.venv) [  6:09PM ]  [ isidentical@threeheadedgiant:~ ]
 $ ./cpython/python z.py
/home/isidentical/cpython/Lib/doctest.py:1336: RuntimeWarning: coroutine '<module>' was never awaited
  exec(compile(example.source, filename, "single",
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
result: TestResults(failed=0, attempted=1)

GH 19230:
(.venv) [  6:17PM ]  [ isidentical@threeheadedgiant:~ ]
 $ ./cpython/python z.py
**********************************************************************
File "/home/isidentical/doctest.txt", line 4, in doctest.txt
Failed example:
    await __import__('asyncio').sleep(1)
Exception raised:
    Traceback (most recent call last):
      File "/home/isidentical/cpython/Lib/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest doctest.txt[0]>", line 1
    SyntaxError: 'await' outside function
**********************************************************************
1 items had failures:
   1 of   1 in doctest.txt
***Test Failed*** 1 failures.
result: TestResults(failed=1, attempted=1)

Either the regression should be fixed, or the commits which introduced the regression should be reverted.

I'm talking about python-gmpy2 doctest which pass on Python 3.8: msg365311.

> Either the regression should be fixed, or the commits which introduced the regression should be reverted.

If you do that we will have now two problems:

* The asyncio repr will not work correctly.
* The flags in compile still have a clash.

The solution is to merge https://github.com/python/cpython/pull/19230 or some version of it, which should fix the issue in  python-gmpy2 doctest .

New changeset 4454057269b995341b04d13f0bf97f96080f27d0 by Batuhan Taşkaya in branch 'master':
bpo-39562: Prevent collision of future and compiler flags (GH-19230)
https://github.com/python/cpython/commit/4454057269b995341b04d13f0bf97f96080f27d0

> bpo-39562: Prevent collision of future and compiler flags (GH-19230)
> https://github.com/python/cpython/commit/4454057269b995341b04d13f0bf97f96080f27d0

I tested manually: this change fix my msg365311 reproducer. Thanks!

In Python 3.8, PyCF_ALLOW_TOP_LEVEL_AWAIT = CO_FUTURE_DIVISION = 0x2000. The commit 9052f7a41b90f2d34011c8da68f9a4facebc8a97 was backported to 3.8:

New changeset ec8a973f7cf080d9c0679f058b2371f0b7c7862c by Miss Islington (bot) in branch '3.8':
bpo-39562: Allow executing asynchronous comprehensions in the asyncio REPL (GH-18968)
https://github.com/python/cpython/commit/ec8a973f7cf080d9c0679f058b2371f0b7c7862c

And now 3.8 branch also has the bug: msg365311 reproducer fails as well in 3.8.

What should be done? Revert ec8a973f7cf080d9c0679f058b2371f0b7c7862c? Change constants value in Python 3.8.3? (backport 4454057269b995341b04d13f0bf97f96080f27d0 to 3.8)

IMHO reverting the commit ec8a973f7cf080d9c0679f058b2371f0b7c7862c (fix async comprehension in asyncio REPL) is the safest option.

First, I understood that the asyncio REPL was experimental in Python 3.8: https://bugs.python.org/issue37028

But it's not documented as provisional or experimental in What's New in Python 3.8:
https://docs.python.org/dev/whatsnew/3.8.html#asyncio

But I'm also fine with backporting the fix 4454057269b995341b04d13f0bf97f96080f27d0 (change constants) to 3.8.

I think we should backport the fix 4454057269b995341b04d13f0bf97f96080f27d0 (change constants) to 3.8 given that the likelihood of users using the actual hardcoded value of the constants instead of the constants themselves is very low. Also, given the collision, it would be fixing a bug present still in 3.8. 

If we revert 9052f7a41b90f2d34011c8da68f9a4facebc8a97 we would have two bugs in 3.8: collision of constants and asyncio repr not working properly.

New changeset 5055c274c6e4f2bb8025910dedf0ff89f4bdd170 by Pablo Galindo in branch '3.8':
[3.8] bpo-39562: Prevent collision of future and compiler flags (GH-19230) (GH-19835)
https://github.com/python/cpython/commit/5055c274c6e4f2bb8025910dedf0ff89f4bdd170

New changeset 71e6122b4428ae43e868e34db4f072635f58a555 by Pablo Galindo in branch '3.8':
bpo-39562: Correctly updated the version section in the what's new document (GH-19838)
https://github.com/python/cpython/commit/71e6122b4428ae43e868e34db4f072635f58a555

Fedora packagers report that this problem is now showing up in 3.8.3rc1. What can be done to ensure that 3.8.3 final will contain the fix?

The merge is in 3.8 master, so we need to make sure that Łukasz includes this on the 3.8.3 release. Victor sent an email to python-dev already about this issue. If you want to make absolutely sure this happens, maybe send an email directly to Łukasz.

I just did.

I reopen the bug to ensure that the fix will land into Python 3.8.3. The issue is marked as release blocker.

I tested manually the just released Python 3.8.3 with msg365311: I confirm that it's fixed. Thanks!

This make non-await list comprehension coroutine-code-objects as well: 

https://bugs.python.org/issue41218

    import ast
    import inspect
    cell = '[x for x in l]'
    code = compile(cell, "<>", "exec", flags=getattr(ast,'PyCF_ALLOW_TOP_LEVEL_AWAIT', 0x0))

    inspect.CO_COROUTINE & code.co_flags == inspect.CO_COROUTINE  # this is now TRUE. 

This leads to weird things in Jupyter/IPython when we try to detect wether a block of code is, or os not async.

Pablo, I see you reopened and marked as blocker, 

As the changes here has been released maybe you want to reclose and marked https://bugs.python.org/issue41218 as blocker it should also be fixed by
https://github.com/python/cpython/pull/21357

> As the changes here has been released maybe you want to reclose and marked https://bugs.python.org/issue41218 as blocker it should also be fixed by
https://github.com/python/cpython/pull/21357

Thanks for the heads up, Matthias!

Yes, I think is the best thing to do. For some reason, I thought this was not still released :(

this maybe shouldn't have been backported to 3.8.x as the change in compiler flags appears to break pyc files in subtle ways: https://stackoverflow.com/q/68910329/812183