classification
Title: loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio
Type: enhancement Stage: patch review
Components: asyncio Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, flying sheep, fried, lukasz.langa, yselivanov
Priority: normal Keywords: patch

Created on 2018-05-15 16:21 by fried, last changed 2019-08-13 14:12 by flying sheep.

Pull Requests
URL Status Linked Edit
PR 6866 open fried, 2018-05-15 16:23
Messages (4)
msg316678 - (view) Author: Jason Fried (fried) * Date: 2018-05-15 16:21
At Facebook and Instagram we have large interconnected codebases without clear boundaries of ownership. As we move more and more services to utilize asyncio we are finding that once blocking (but fast) code paths, are now cropping up with asyncio code using run_until_complete().  Now this is fine for all the blocking callers, but some times we have async callers to that blocking code path and now it doesn't work.  

So we have two options revert the change to not use asyncio deep in the dep tree or Convert all functions in the stack to be asyncio.  Both are not possible and engineers have solved them in two crazy ways.

1. Nested Event Loops, when you hit a run_until_complete, create a new eventloop and do the async and return the result.
2. Like the first, but each library creates its own eventloop, and swaps it with the running loop for the duration of the run_until_complete, restoring the original loop when its done. 

Both of these ultimately have the same problem, everything on the primary event loop stops running until the new loop is complete. What if we could instead start servicing the existing eventloop from the run_until_complete. This would insure that tasks don't timeout.

This would allow us to convert things to asyncio faster without having to have absolute knowledge of a codebase and its call graph, and not have to have all engineers completely synchronized.
msg317221 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-05-21 08:55
Sorry, no.
The feature was requested many times but was constantly rejected.
By this, you are adding a BLOCKING call to your async function.
At least it leads to log warning about too long callback execution.

Moreover, I suspect that `run_until_complete` reentrancy requirement breaks the existing third-party loop implementations, not all loops can be fixed easily.

The last: calling blocking code from async code is the anti-pattern, asyncio explicitly discourages it.

You should convert all your stack to async functions and add sync stubs
when needed like

def sync_call(arg):
    asyncio.get_event_loop().run_until_complete(async_call(arg))

Yuri, do you agree with me?
msg317254 - (view) Author: Jason Fried (fried) * Date: 2018-05-21 22:02
For loops not supporting this throwing NotImplmentedError from the method to enable reentrancy seems appropriate. 

"You should convert all your stack to async functions..."

That may not be practical for large code bases in transition to asyncio. The fixes for reentrancy that I find in reality are not adding async logic through out the call stack but instead its one of the two I listed.
msg349561 - (view) Author: (flying sheep) * Date: 2019-08-13 14:12
There’s this monkeypatching solution: https://pypi.org/project/nest-asyncio/

But yes, it’s a very practical problem that you can’t call async code from sync code that’s being called from async code.
History
Date User Action Args
2019-08-13 14:12:17flying sheepsetnosy: + flying sheep
messages: + msg349561
2018-05-21 22:02:49friedsetmessages: + msg317254
2018-05-21 08:55:44asvetlovsetmessages: + msg317221
2018-05-15 16:23:14friedsetkeywords: + patch
stage: patch review
pull_requests: + pull_request6540
2018-05-15 16:21:44friedsetnosy: + lukasz.langa
2018-05-15 16:21:06friedcreate