classification
Title: implement "Async exec"
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, brett.cannon, mbussonn, minrk, miss-islington, ned.deily, njs, pmpp, vstinner, willingc, xtreak, yselivanov
Priority: normal Keywords: patch

Created on 2018-09-09 21:57 by mbussonn, last changed 2019-05-22 19:38 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13148 merged mbussonn, 2019-05-07 03:18
PR 13473 merged yselivanov, 2019-05-21 21:06
PR 13484 merged mbussonn, 2019-05-22 05:08
Messages (40)
msg324900 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2018-09-09 21:57
Hi, 

This is an issue, prompted by twitter (https://twitter.com/Mbussonn/status/1038866329971961859) and by the imminent release of IPython 7.0 that provides an async REPL to discuss the introducion of something I'll call "Async exec", the exact form can vary, but I believe the name si relatively self explanatory. 

The short description would be to allow something akin to `exec` but for asynchronous code. Typically for one to be able to write an async-repl in the generic sens that is not say not obviously liked to asyncio.

For example IPython 7.0 (current master branch) allow the following:

```

In [1]: import asyncio, trio, curio

In [2]: await asyncio.sleep(0)

In [3]: %autoawait trio

In [4]: await trio.sleep(0)

In [5]: %autoawait curio

In [6]: await curio.sleep(0)
Out[6]: 30980.70591396
```


Sleep is here and example, but you can play with aoihttp, asks, and other library and it "just works". Alternatively when using IPython via Jupyter, you can also schedule background tasks that will execute in the currently running loop. 

To reach this, we had to work around a large number of roadblock, and there is a number of missing pieces (or things especially prevented) in core Python we had to work around. To see how we did that see https://github.com/ipython/ipython/pull/11265

The core being if we have a block of async code like `await sleep(0)`, we need an asynchronous way to exec it without blocking, hence the proposal for async-exec.

During the development and test of the above feature of IPython here are some of the challenges we got with top-level async code. 

1) top-level async is invalid syntax. 

It make sens for a module for this to be invalid syntax, but not in a repl.
Since Python 3.7 we can (at least) compile it to AST, but not to bytecode.

2) It would also be extremely convenient to have a util function to tell you whether what you just compiled need to be ran async or not, from Source Code, ast tree, code object. It's surprisingly not trivial to get it always right.

So far in IPython we have to carry-over and recompute whether to actually run the compiled byte code in classical `exec` or our pseudo async-exec. You may think that `await exec()` always cover the case, but when you are already running under asyncio, you may want to tell user "no you can't run async code", and use the normal `exec` path.


3) Have  distinction in this `async exec`/`compile` between "I am compiling a module", currently `exec` mode for both exec and compile, and a "I'm compiling a _multiline_ interactive statement".

4) Not be coupled to a specific async library.

Allow new AIO library like trio/curio/... to use this.

Note that despite both using IPython, the above cover 2 use cases. 
- Terminal IPython were the loop is started/stopped between each user input. 
- Notebook kernel/Jupyter IPython where the loop is already running and task/user code can be process in background w/o pausing the loop. 

AFAICT, this would also be of potential use for other REPL (Xonsh, Bpython).

I'm happy to give more details, but I think that's a enough of a broad overview, as we should be releasing this in IPython in a few days/week, we will likely have further feedback from a wider range of users that can inform the design.
msg324915 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-09-10 06:17
I think the first thing is to add async "modes" to compile: in particular "async-exec" and "async-single". These would be like the current "exec" and "single" modes respectively, except that they act like the code is inside an "async def", so "await" is allowed, and executing the resulting code object produces a coroutine object that has to be iterated to actually run the code.

I guess we might want "async-eval" too just for completeness, though I'm not currently aware of any use cases for that.

A utility to check whether an AST requires async mode should be fairly straightforward. So if you want to choose on the fly, you would do:

1. ast = compile(source, filename, "async-exec", ast.PyCF_ONLY_AST)
2. use a utility check whether 'ast' contains any top-level 'await'/'async with'/'async for'
3. if so, create bytecode with compile(ast, filename, "async-exec"). If not, create bytecode with compile(ast, filename, "exec").

Once you have a code object, I think it's too late: if you use "async-exec" mode to compile a code object that doesn't actually use 'await', then it should still return a coroutine object that needs iterating, etc., just like an 'async def' that has no 'await' in it. So if you want to do this check, the AST phase is the time to do it. Maybe ast.is_ast_async(ast_obj)?

> Have distinction in this `async exec`/`compile` between "I am compiling a module", currently `exec` mode for both exec and compile, and a "I'm compiling a _multiline_ interactive statement".


This seems like a separate problem from the async stuff... I'm curious to hear how what distinction you want to make between 'exec' and a new 'multi-single' (?) mode, but maybe that should be a new issue?
msg324925 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2018-09-10 14:03
> I think the first thing is to add async "modes" to compile: in particular "async-exec" and "async-single". These would be like the current "exec" and "single" modes respectively, except that they act like the code is inside an "async def", so "await" is allowed, and executing the resulting code object produces a coroutine object that has to be iterated to actually run the code.

> This seems like a separate problem from the async stuff... I'm curious to hear how what distinction you want to make between 'exec' and a new 'multi-single' (?) mode, but maybe that should be a new issue?

Mell, in classical `exec` there is always a tension between "this is for a module" and "this is for REPL". We saw that close to 3.7 release where strings literal were moved into the AST Module `docstring` attribute.

In IPython[1] we also have to dance to trigger the display hook in a multi-line context by executing each top-level node one by one. So while we are designing a new `async-exec` while not tackle that issue at the same time and cover both use case ? That should let one category of user do the optimization they wish and get a `Module` object, without being slow down by the other. 

1: https://github.com/ipython/ipython/blob/869480ed70944ca70ad9ed70779b9c3e4320adb7/IPython/core/interactiveshell.py#L3179-L3190
msg325339 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2018-09-14 09:53
> A utility to check whether an AST requires async mode should be fairly straightforward.

Here is one case we forgot in IPython apparently :

In [1]: x = 1
   ...: def f():
   ...:     nonlocal x
   ...:     x = 10000

This is not detected as a syntax error, but considered as asyn, we took an approach that prefer false positive (try to run invalid syntax as async) than false negative (reject valid async-code). Add this to the test suite for this feature.
msg331525 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2018-12-10 17:11
So through time our heuristic to check wether a code should be async or not grew: 

https://github.com/ipython/ipython/blob/320d21bf56804541b27deb488871e488eb96929f/IPython/core/async_helpers.py#L94-L165

There also seem to be some code that uses `codeop.compile_command` to figure out wether the user code is valid syntax (or not), so that would need some update too. 

I'm thinking of submitting a talk at PyCon to explain what we've discover so far in IPython.
msg331526 - (view) Author: pmp-p (pmpp) * Date: 2018-12-10 17:35
indeed adding async flag to compile and providing some 'aexec' is a very good idea ! 

*an async repl is really usefull when stuck with a threadless python*

( specific engines, or emscripten cpython )

"top-level async is invalid syntax" : 
Rewinding the readline history stack to get code "async'ified" is probably not the best way : readline is specific to some platforms.
see https://github.com/pmp-p/aioprompt for a hack using that.

First raising an exception "top level code is async" and allowing user to get source code from exception would maybe a nice start to an async repl.
msg331529 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2018-12-10 18:05
In IPython we use `prompt_toolkit`  which does already provide a async readline alternative.

Also have a look at https://github.com/ipython/ipython/blob/320d21bf56804541b27deb488871e488eb96929f/IPython/core/interactiveshell.py#L121-L150

Seem to be equivalent to what you aare trying to do with updating your locals here https://github.com/pmp-p/aioprompt/blob/93a25ea8753975be6ed891e8d45f22db91c52200/aioprompt/__init__.py#L78-L94

It just sets the function to not create a new local scope
msg331531 - (view) Author: pmp-p (pmpp) * Date: 2018-12-10 18:53
i already use prompt_toolkit on droid as it uses concurrent futures for completion and threads are allowed on that platform, and yeah it is quite good.

but no way to use it on emscripten where cpython is 100% async ( it uses dummy_threading to load asyncio ). best you can do is fill an history buffer with the indented input, eval the whole thing when it's done with PyRun_SimpleString. 

having cpython storing code until sync/async path can  be choosen could save a lot of external hacks with minimal impact on original repl loop, unless somebody is willing to make it *fully* async ( i know i can't ). 

The original repl input loop is really not made for async and i don't know if Sylvain Beuclair's work on "emterpreted" cpython covers also python3.

thx for the pointers anyway and your article on async and ast was inspiration.
msg331535 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-12-10 21:06
> I'm thinking of submitting a talk at PyCon to explain what we've discover so far in IPython.

You totally should!

Or actually there are two options to think about: you can submit a general talk, or submit a talk to the language summit. (Or write two talks and do both, I guess.) They're pretty different – the summit is a more informal thing (no video, smaller room), mostly just core devs, more of a working meeting kind of thing where you can argue about technical details.
msg331539 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2018-12-10 21:38
> Or actually there are two options to think about: you can submit a general talk, or submit a talk to the language summit. (Or write two talks and do both, I guess.) They're pretty different – the summit is a more informal thing (no video, smaller room), mostly just core devs, more of a working meeting kind of thing where you can argue about technical details.

Thanks, I may do that then – if a core dev invite me to do so – I wouldn't have dared otherwise. I'm not even sure you can suggest a language summit proposal yet.

For the normal talk proposal here is what I have so far: 

https://gist.github.com/Carreau/20881c6c70f1cde9878db7aa247d432a
msg341202 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-01 14:50
Slides and demo (videos) during PyCon2019 Language Summit: https://github.com/Carreau/talks/tree/master/2019-05-01-Language-Summit

One question was how to handle non-asyncio in Core Python, REPL.

My response was to not take care of that in the first time, but provide the building blocks for alternative REPL, in a second time provide an async-input, and a way to register runner for alternative async libraries.
msg341203 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-05-01 14:56
> My response was to not take care of that in the first time, but provide the building blocks for alternative REPL, in a second time provide an async-input, and a way to register runner for alternative async libraries.

Yeah, I think this is pretty simple: the runtime/stdlib should provide the primitives to compile top-level async code, get coroutines, etc., and then a REPL like ipython can take care of handing that off to asyncio or whatever library they want.

Maybe in the long run the builtin REPL should get async support "out of the box", but that's much less clear, and anyway we should split that off into a separate issue if we want to discuss it.

async-input isn't even useful for ipython, is it? you use prompt-toolkit :-)
msg341209 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-01 15:13
> Yeah, I think this is pretty simple: the runtime/stdlib should provide the primitives to compile top-level async code, get coroutines, etc., and then a REPL like ipython can take care of handing that off to asyncio or whatever library they want.

Exactly.  FWIW I'm +1 to have this in 3.8.  Do you have a patch for this?  If not I can take a look at this myself tomorrow or the day after.
msg341222 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-01 18:17
> async-input isn't even useful for ipython, is it? you use prompt-toolkit :-)


You don't have to use prompt_toolkit (https://github.com/ipython/rlipython), but yes we don't need this. 

> Do you have a patch for this?  If not I can take a look at this myself tomorrow or the day after.

No, I do not have a patch. I can provide a few test case and I am happy to spend some task to sit-down and discuss what we exactly do so far in IPython. I'm flying back Sunday morning.
msg341224 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-01 18:27
> No, I do not have a patch. I can provide a few test case

That would be helpful!

> and I am happy to spend some task to sit-down and discuss what we exactly do so far in IPython. I'm flying back Sunday morning.

Yes, let's do that around Friday/Saturday (feel free to ping me via a Twitter DM).  I'm also leaving on Sunday.
msg341230 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-01 19:48
Extra notes from in-person discussion; 
We might want a both a Sync-REPL and Async-REPL `compile()` mode depending on wether the REPL support async. 

One of the other question was wether `exec` should look at wether an eventloop is running, or if it should just return a coroutine and it's the -programmer job to check wether there is an event loop or not and do the right thing. 

Also for info all the top level async-await PR and issues on IPython can be found here: https://github.com/ipython/ipython/issues?q=label%3Aasync%2Fawait+is%3Aclosed
msg341232 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-01 20:22
I recall the idea of passing a specific flag to `compile()` for accepting await and family on top level of passed code string.

Than compile can return a code object with CO_COROUTINE flag set.

Returned code object can be analyzed for this flag and executed with `await exec(code)`.

Actual coroutine executor can be asyncio/trio/twisted/whatever, it depends on what code calls this `await`.

Did I miss something?
msg341388 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-04 15:38
> Did I miss something?

No I think for the async stuff that should be a big improvement already. 
Either a flag, or different "MODE", like 'single','exec' and 'eval'.

I'l love for something like 'exec' but where not only we can have multiple statement, but where the last statement behave like single and "expression statements that evaluate to something other than None will be printed".

The other thing would be for this mode to not turn the the fist statement into a module docstring in the AST if it is a string.

I know that most of these are slightly orthogonal but do have backward compatibility consequences.

I'll try to be at PyCon "Mentored Sprints" this afternoon, I'll be ha[[y to be mentored into contributing this to CPython. 

I'm also happy to discuss writing this in a pep.
msg341396 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-04 18:10
Sorry, I don't know all the compilation workflow details to help you quickly (and will be very busy on other tasks during the sprint).

Yuri will be absent on sprints.
msg341409 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-04 21:50
Here's a VERY rough first implementation to play with: https://github.com/1st1/cpython/commit/ad2ed0aed922d7c36f2fced64264124613e37f09
msg341681 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-07 04:10
Thank Yuri for the guidance; I worked on it, cleaned things up a bit and posted a draft PR (#13148) with some example. 

That helps cleaning up a lot of code; and in the PR is a ~30 line example that implement an asyncio-repl.
msg341823 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-07 21:01
Matthias, please use GH-13148 for pointing on github pull requests.
#xxxx is for links to this tracker issues.
msg342598 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-15 19:54
I see the 3.8 feature freeze seem to be end of next-week; do you think that async-exec (gh-13148) has a chance of getting in ? I'm happy to do modification to the PR but would need some more reviews. I'm happy to take some other tasks of your plate is that allow this to squeeze in before feature freeze. 

Thanks.
msg342602 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-15 21:20
I'll be working on CPython on Friday. Will take a look at your PR first thing.
msg342604 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-15 21:55
> I'll be working on CPython on Friday. Will take a look at your PR first thing.

Thanks, let me know if the is anything I can do for you in exchange; also thanks again for sitting with me and stepping me through the things at PyCon.
msg343089 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-21 20:12
New changeset 565b4f1ac7304d1e690c404ca8316f383ba60862 by Yury Selivanov (Matthias Bussonnier) in branch 'master':
bpo-34616: Add PyCF_ALLOW_TOP_LEVEL_AWAIT to allow top-level await (GH-13148)
https://github.com/python/cpython/commit/565b4f1ac7304d1e690c404ca8316f383ba60862
msg343090 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-21 20:16
Great!
msg343099 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-21 21:20
New changeset eca18aac7b5952d23854d27dc8e502dfb0bb0505 by Ned Deily (Yury Selivanov) in branch 'master':
bpo-34616: Fix code style and unbreak buildbots (GH-13473)
https://github.com/python/cpython/commit/eca18aac7b5952d23854d27dc8e502dfb0bb0505
msg343109 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 22:26
Would it be possible to document this change in What's New in Python 3.8?
msg343111 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-21 22:31
> Would it be possible to document this change in What's New in Python 3.8?

Yes, Elvis and I will take care of that later.
msg343112 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 22:32
Any idea why PR 13148 has been linked to unrelated bugs.python.org issues? I saw 3 of them: bpo-35363, bpo-25234, bpo-33725.

https://mail.python.org/pipermail/python-dev/2019-May/157592.html
msg343113 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 22:33
> Yes, Elvis and I will take care of that later.

Well, it would be nice to get a first mention before the next release, to see all new shiny Python 3.8 features ;-) The text can be reworded later if needed ;-)
msg343116 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-21 22:40
> Any idea why PR 13148 has been linked to unrelated bugs.python.org issues?

Could it be due to rebasing and force-pushing ? Cause I did force-push on this branch a couple of times...
msg343117 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-21 22:42
> Yes, Elvis and I will take care of that later.

And I'm guessing you will collect what's in NEWS.d ? Otherwise I'm happy to contribute some text. let me know the best way.
msg343118 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 22:43
> Could it be due to rebasing and force-pushing ? Cause I did force-push on this branch a couple of times...

Maybe you used "git merge" and your PR "contained" changes from other issues?
msg343119 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-05-21 22:43
> And I'm guessing you will collect what's in NEWS.d ? Otherwise I'm happy to contribute some text. let me know the best way.

Would be great if you could make a PR to add an entry to whatsnew/3.8.rst (as Victor suggests)
msg343120 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-21 22:45
> And I'm guessing you will collect what's in NEWS.d ? Otherwise I'm happy to contribute some text. let me know the best way.

One option to document it is to add a new "builtins" section to document the new flag inside "Improved Modules" category:
https://docs.python.org/dev/whatsnew/3.8.html#improved-modules
msg343121 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-21 22:47
> Maybe you used "git merge" and your PR "contained" changes from other issues?

I quasi-nerver merge, either `rebase interactive` or `reset --hard HEAD`... I even proscribed "git pull" from my CLI.. but you know everybody does mistakes. 

> Would be great if you could make a PR to add an entry to whatsnew/3.8.rst (as Victor suggests)

Ok, i'll do this.
msg343146 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-22 05:13
> section to document the new flag inside "Improved Modules" category

Done in https://github.com/python/cpython/pull/13484
msg343231 - (view) Author: miss-islington (miss-islington) Date: 2019-05-22 19:38
New changeset 2ddbd21aec7f0e2f237a1073d3e0b313e673413f by Miss Islington (bot) (Matthias Bussonnier) in branch 'master':
bpo-34616: Document top level async in whatsnew/3.8. (GH-13484)
https://github.com/python/cpython/commit/2ddbd21aec7f0e2f237a1073d3e0b313e673413f
History
Date User Action Args
2019-05-22 19:38:51miss-islingtonsetnosy: + miss-islington
messages: + msg343231
2019-05-22 05:13:42mbussonnsetmessages: + msg343146
2019-05-22 05:08:21mbussonnsetpull_requests: + pull_request13396
2019-05-21 22:47:19mbussonnsetmessages: + msg343121
2019-05-21 22:45:57vstinnersetmessages: + msg343120
2019-05-21 22:43:47yselivanovsetmessages: + msg343119
2019-05-21 22:43:23vstinnersetmessages: + msg343118
2019-05-21 22:42:49mbussonnsetmessages: + msg343117
2019-05-21 22:40:35mbussonnsetmessages: + msg343116
2019-05-21 22:33:45vstinnersetmessages: + msg343113
2019-05-21 22:32:36vstinnersetmessages: + msg343112
2019-05-21 22:31:12yselivanovsetmessages: + msg343111
2019-05-21 22:26:54vstinnersetnosy: + vstinner
messages: + msg343109
2019-05-21 21:20:24ned.deilysetnosy: + ned.deily
messages: + msg343099
2019-05-21 21:06:43yselivanovsetpull_requests: + pull_request13385
2019-05-21 20:16:32asvetlovsetmessages: + msg343090
2019-05-21 20:13:44yselivanovsetstatus: open -> closed
resolution: fixed
stage: resolved
2019-05-21 20:12:06yselivanovsetmessages: + msg343089
2019-05-15 21:55:23mbussonnsetmessages: + msg342604
2019-05-15 21:20:05yselivanovsetmessages: + msg342602
2019-05-15 19:54:12mbussonnsetmessages: + msg342598
2019-05-07 21:01:04asvetlovsetmessages: + msg341823
stage: patch review -> (no value)
2019-05-07 04:10:21mbussonnsetmessages: + msg341681
2019-05-07 03:18:07mbussonnsetkeywords: + patch
stage: patch review
pull_requests: + pull_request13063
2019-05-04 21:50:23yselivanovsetmessages: + msg341409
2019-05-04 18:10:12asvetlovsetmessages: + msg341396
2019-05-04 15:38:36mbussonnsetmessages: + msg341388
2019-05-01 20:22:16asvetlovsetmessages: + msg341232
2019-05-01 19:48:10mbussonnsetmessages: + msg341230
2019-05-01 18:27:03yselivanovsetmessages: + msg341224
2019-05-01 18:17:31mbussonnsetmessages: + msg341222
2019-05-01 15:13:24yselivanovsetmessages: + msg341209
2019-05-01 14:56:53njssetmessages: + msg341203
2019-05-01 14:50:54mbussonnsetmessages: + msg341202
2019-05-01 14:34:53brett.cannonsetnosy: + brett.cannon
2019-05-01 14:32:36asvetlovsetnosy: + asvetlov
2019-03-26 00:09:01xtreaklinkissue36428 superseder
2019-03-26 00:08:11xtreaksetnosy: + xtreak
2018-12-10 21:38:16mbussonnsetmessages: + msg331539
2018-12-10 21:06:47njssetmessages: + msg331535
2018-12-10 18:53:41pmppsetmessages: + msg331531
2018-12-10 18:05:03mbussonnsetmessages: + msg331529
2018-12-10 17:35:06pmppsetnosy: + pmpp
messages: + msg331526
2018-12-10 17:11:39mbussonnsetmessages: + msg331525
2018-09-14 09:53:53mbussonnsetmessages: + msg325339
2018-09-10 14:03:26mbussonnsetmessages: + msg324925
2018-09-10 06:17:18njssetnosy: + njs
messages: + msg324915
2018-09-09 21:59:21mbussonnsetnosy: + yselivanov, willingc, minrk
2018-09-09 21:57:12mbussonncreate