This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
Type: enhancement Stage: resolved
Components: asyncio Versions: Python 3.8
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, blazespinnaker, flying sheep, fried, lukasz.langa, yselivanov
Priority: normal Keywords: patch

Created on 2018-05-15 16:21 by fried, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6866 closed fried, 2018-05-15 16:23
Messages (9)
msg316678 - (view) Author: Jason Fried (fried) * Date: 2018-05-15 16:21
At Facebook and Instagram we have large interconnected codebases without clear boundaries of ownership. As we move more and more services to utilize asyncio we are finding that once blocking (but fast) code paths, are now cropping up with asyncio code using run_until_complete().  Now this is fine for all the blocking callers, but some times we have async callers to that blocking code path and now it doesn't work.  

So we have two options revert the change to not use asyncio deep in the dep tree or Convert all functions in the stack to be asyncio.  Both are not possible and engineers have solved them in two crazy ways.

1. Nested Event Loops, when you hit a run_until_complete, create a new eventloop and do the async and return the result.
2. Like the first, but each library creates its own eventloop, and swaps it with the running loop for the duration of the run_until_complete, restoring the original loop when its done. 

Both of these ultimately have the same problem, everything on the primary event loop stops running until the new loop is complete. What if we could instead start servicing the existing eventloop from the run_until_complete. This would insure that tasks don't timeout.

This would allow us to convert things to asyncio faster without having to have absolute knowledge of a codebase and its call graph, and not have to have all engineers completely synchronized.
msg317221 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-05-21 08:55
Sorry, no.
The feature was requested many times but was constantly rejected.
By this, you are adding a BLOCKING call to your async function.
At least it leads to log warning about too long callback execution.

Moreover, I suspect that `run_until_complete` reentrancy requirement breaks the existing third-party loop implementations, not all loops can be fixed easily.

The last: calling blocking code from async code is the anti-pattern, asyncio explicitly discourages it.

You should convert all your stack to async functions and add sync stubs
when needed like

def sync_call(arg):
    asyncio.get_event_loop().run_until_complete(async_call(arg))

Yuri, do you agree with me?
msg317254 - (view) Author: Jason Fried (fried) * Date: 2018-05-21 22:02
For loops not supporting this throwing NotImplmentedError from the method to enable reentrancy seems appropriate. 

"You should convert all your stack to async functions..."

That may not be practical for large code bases in transition to asyncio. The fixes for reentrancy that I find in reality are not adding async logic through out the call stack but instead its one of the two I listed.
msg349561 - (view) Author: (flying sheep) * Date: 2019-08-13 14:12
There’s this monkeypatching solution: https://pypi.org/project/nest-asyncio/

But yes, it’s a very practical problem that you can’t call async code from sync code that’s being called from async code.
msg350517 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-08-26 10:04
The solution produces subtle and error-prone code.  It can live in third party library but not good enough for stdlib I think.
msg350595 - (view) Author: Jason Fried (fried) * Date: 2019-08-27 04:32
Yeah I have to agree at this point, from a naive point of view it looks
awesome, but its just as bad as loosing gil guarantees.

On Mon, Aug 26, 2019 at 03:04 Andrew Svetlov <report@bugs.python.org> wrote:

>
> Andrew Svetlov <andrew.svetlov@gmail.com> added the comment:
>
> The solution produces subtle and error-prone code.  It can live in third
> party library but not good enough for stdlib I think.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33523>
> _______________________________________
>
msg351424 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-09 10:50
Let's close the issue as "won't fix".

Third-party loop implementation *can* be reentrant but we don't want to encourage people to use this pattern in stdlib.
msg351425 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-09-09 10:54
🙏
msg379212 - (view) Author: Blaze Spinnaker (blazespinnaker) Date: 2020-10-21 15:08
I know this issue is closed, but can you folks review:

https://github.com/ipython/ipykernel/issues/548#issuecomment-713637954

The nest_asyncio patch is becoming a defacto contract to fixing this amongst a lot of people.  But the patch doesn't really event work (IMHO).

It would be ideal to address this in a more central manner, either with change of code, better errors, more clear APIs, better documentation, etc.   Lots of options, but hopefully it will be addressed.
History
Date User Action Args
2022-04-11 14:59:00adminsetgithub: 77704
2020-10-21 15:08:19blazespinnakersetnosy: + blazespinnaker
messages: + msg379212
2019-09-09 10:54:57yselivanovsetmessages: + msg351425
2019-09-09 10:50:55asvetlovsetstatus: open -> closed
resolution: wont fix
messages: + msg351424

stage: patch review -> resolved
2019-08-27 04:32:08friedsetmessages: + msg350595
2019-08-26 10:04:44asvetlovsetmessages: + msg350517
2019-08-13 14:12:17flying sheepsetnosy: + flying sheep
messages: + msg349561
2018-05-21 22:02:49friedsetmessages: + msg317254
2018-05-21 08:55:44asvetlovsetmessages: + msg317221
2018-05-15 16:23:14friedsetkeywords: + patch
stage: patch review
pull_requests: + pull_request6540
2018-05-15 16:21:44friedsetnosy: + lukasz.langa
2018-05-15 16:21:06friedcreate