Currently if one needs lazily resolve event loop depending on where awaitable is being awaited have to pass loop everywhere explicitly. That quickly becomes an unnecessary noise in interfaces of callables. 

Consider an example where a coroutine which constructs other awaitables can be executed in arbitrary loop:

    import asyncio
    import datetime

    async def foo_coro(loop):
        await some_other_coro(loop)

    async def bar_coro(loop):
        await asyncio.ensure_future(..., loop=loop)

    async def baz_coro(loop):
        await asyncio.gather(foo_coro(loop), bar_coro(loop), loop=loop)

    loop = asyncio.get_event_loop()
    loop.run_until_complete(multiple_coros(loop))
    loop.close()

It would be nice, if all functions that belong to an event loop instance as well as asyncio helpers that accept a loop would set default event loop to one that was passed to these functions. So that the example above could be rewritten as:

    import asyncio
    import datetime

    async def foo_coro():
        await some_other_coro()

    async def bar_coro():
        await asyncio.ensure_future(...)

    async def baz_coro():
        await asyncio.gather(foo_coro(), bar_coro())

    loop = asyncio.get_event_loop()
    loop.run_until_complete(multiple_coros())
    loop.close()

Can't you easily write such a policy yourself? Why does it have to be a
standard part of asyncio?

Can we at least make it part of the documentation?  It's not obvious to me, at least, how to do it.  (Maybe I just haven't thought about it carefully enough.)  For that matter, it wasn't obvious to me it could even be done, so I've been passing loop everywhere in my code rather than thinking about how to avoid it; which is another argument for documenting it at least.

> Currently if one needs lazily resolve event loop depending on where awaitable is being awaited have to pass loop everywhere explicitly. That quickly becomes an unnecessary noise in interfaces of callables. 

Why don't you just use 'loop = asyncio.get_event_loop()' in places where you need the loop?

Because, as indicated by the OP, I'm not using the default event loop of the thread.

Thinking about my code, I now remember that the reason I went down this path was because I wanted to make sure that my functions, like asyncio functions, accepted the optional loop parameter, and that it worked. For it to work my code pretty much has to pass loop everywhere, because that was a design decision I made early on.  So maybe there's nothing interesting for me here after all.

> Why does it have to be a standard part of asyncio?

I've only seen few libraries that deal with asyncio so far (aiohttp, pyzmq), but my general impression is that this is a generic problem.

With asyncio code should be (ideally) written as a set of coroutines which schedule each other or are called in response to monitored events. That means (and asyncio very implementation shows that) loop has to be "self" for all coroutines it executes.

Thread-local default event loop is a good solution to get an entry point into the event loop from an out-of-event-loop execution location. But (ideally) there will be exactly one place in code where this behavior is convenient: when you "instantiate" and run event loop. After that the event loop becomes "self" for each coroutine it may run and therefore it's now convenient for get_event_loop to return currently running event loop.

Probably it would make sense to modify behavior of the default policy to make get_event_loop to return Thread-local event loop when called from out-of-event-loop and return currently running event loop when called from within a coroutine.

> Why don't you just use 'loop = asyncio.get_event_loop()' in places where you need the loop?

As David pointed out, I'm not using thread-local event loop.

> Probably it would make sense to modify behavior of the default policy to make get_event_loop to return Thread-local event loop when called from out-of-event-loop and return currently running event loop when called from within a coroutine.

The problem with this approach is that you're relying on the presence of GIL.  If, let's say, PyPy releases tomorrow with 3.5 and STM, this won't work.

To clarify: the problem lies in "return currently running event loop when called from within a coroutine" -- what if there is no GIL and we have a few event loops running?  You'd need to use a threadlocal storage.  In this case I'm not sure how this is different from the current "get_event_loop".

> In this case I'm not sure how this is different from the current "get_event_loop"

Thread may have multiple event loops, but only one, explicitly associated, is default. And it's not necessary one which is currently running.

I think what I propose here can be expressed in Python terms as an implicit context manager that replaces Thread's default event loop once while it "runs" particular event loop then switches it back (possible to None) once loop is stopped.

> Thread may have multiple event loops, but only one, explicitly associated, is default. And it's not necessary one which is currently running.

Ah, I got it.  You know what, this actually is starting to make sense.

Guido, what do you think about this?  Essentially, (Ilya, feel free to correct me if I'm wrong), we can implement the following:

1. Add a new thread-local storage, say "_running_loop"

2. Add another method "asyncio.get_running_loop()"

3. Update some places in asyncio where we currently use "get_event_loop()", such as Future constructor, Task.current_task, etc.

This might actually solve an awkwardness of get_event_loop(), which when called, instantiates an event loop if it doesn't find one, which makes it difficult to know if an event loop was running before the call.

Honestly I think it's pretty crazy and out there to have multiple event
loops in the same thread. That feels like an anti-pattern inspired by some
other event loop APIs (in other languages) that encourage this. But asyncio
was not designed for that.

> Honestly I think it's pretty crazy and out there to have multiple event
loops in the same thread. That feels like an anti-pattern inspired by some
other event loop APIs (in other languages) that encourage this. But asyncio
was not designed for that.

I agree.

OTOH, if you're designing a library for asyncio, you want it to be as foolproof as possible, so many people simply pass an event loop everywhere.  It's especially annoying when you have a huge chunk of code that didn't need the loop, and then when something needs it you have to refactor everything or use "get_event_loop".

In terms of performance, I don't think this is gonna affect anything, it's just a couple of additional thread-local sets in the Loop.run method.

And we can design "get_running_loop" to raise a clear exception if no loop is currently running.

Yury, that would do it.

Guido, that's indeed might be an anti-pattern. But it looks like passing event loop around is just a worse version of it.

> Update some places in asyncio where we currently use "get_event_loop()", such as Future constructor, Task.current_task, etc.

Yury, do you have an idea how it could be done?

The way I avoid the errors of failing to always pass the loop through in my application library, currently, is to run my tests in a thread with no event loop set up, so that get_event_loop will raise an error.  Take that for what it is worth, since I don't claim to know what best practices are in asyncio programming.

>> Update some places in asyncio where we currently use "get_event_loop()", such as Future constructor, Task.current_task, etc.

> Yury, do you have an idea how it could be done?

I think I do.  To keep the backwards compatibility, we'll need to update the "get_event_loop()" function as follows:

  def get_event_loop():
     loop = _running_loop.loop
     if loop is None:
         loop = get_event_loop_old_impl()
     return loop

No need to actually update Future or Task or any other asyncio code.

The get_running_loop() would look like this:

  def get_running_loop():
     loop = _running_loop.loop
     if loop is None:
         raise RuntimeError('no running event loop')
     return loop

FWIW asyncio's own test suite makes sure that the loop is indeed passed
everywhere by setting the default loop to None. If a library chooses to
pass the loop around like this it should structure its tests the same way.

Guido, Probably a legitimate example of having multiple event loops in a single thread: you want to run certain coroutine synchronously without running thread's own event loop.

No, that specifically sounds like asking for trouble to me. You don't know
what else that coroutine is waiting for, and it may be waiting for some I/O
whose socket is registered with the other event loop. Since the other event
loop won't make progress, you'd be deadlocked.

PS. If you have something you sometimes want to run synchronously and
sometimes as a coroutine, you probably need to refactor it somehow.

> You don't know what else that coroutine is waiting for, and it may be waiting for some I/O whose socket is registered with the other event loop. Since the other event loop won't make progress, you'd be deadlocked.

As an end user I know what my coroutines may be waiting for. It's out of question to expect an ability to reuse coroutine and its associated context in multiple loops.

> PS. If you have something you sometimes want to run synchronously and sometimes as a coroutine, you probably need to refactor it somehow.

We have a service that is responsible for reporting various stages about our application's lifetime. Most of the time app does not care about when (and whether) data will be actually sent, so we run this service in its own thread, the rest of the app just schedules coroutines in its event loop (hidden and managed by service's API). However sometimes it does care and moreover needs to do that synchronously (i.e. when we handle fatal / unhandled exception. In order to reuse remaining code that constructs coroutines, in the place of invocation we create a temporary event loop and use run_until_complete.

This is all practical problem. The conceptual problem is that current API is not flexible enough to create an event policy that would do something more useful that changing default type of an event loop.

Having the ability to get currently running event loop from within executing coroutine sounds pretty convenient and in my opinion would greatly reduce the amount of passing loops everywhere for end users (library developery, unfortunately, have no chance to avoid this).

Instead of starting a new event loop, you should figure out a way to wait
for an event in the existing loop. IIUC that loop runs in a different
thread -- I think you can solve this by using a threading.Event that you
set from a wrapper coroutine running in your event loop and waited for in
the main loop. (If you're worried about blocking the event loop just to
acquire the threading lock that's part of threading.Event, you can use
run_in_executor().

Good luck!

> If you're worried about blocking the event loop just to acquire the threading lock that's part of threading.Event, you can use run_in_executor()

I'm more worried to delay invocation of a "synchronous" report because of outstanding tasks in event loop's queue. Given my situtation, that may not even happen because outstanding tasks may stop the loop by raising their own exceptions.

I'm willing to sacrifice them (i.e. I'm ok with not sending them) in order to send information about exception "for sure".

Yury, could you submit a patch implements this feature?

> Yury, could you submit a patch implements this feature?

Yes, the patch part is easy. The hard part is to get a green light from Guido on adding the 'get_running_loop()' function...

Still no green light. Ilya just seems to want something misguided.

Ilya, at this point I'd suggest you to craft a very clear and simple to digest email to python-ideas describing 'get_running_loop()', how it can be used, your case, other cases, etc.

Yury, what do you think of this yourself? Maybe you can explain it better
than Ilya?

Yury, we're building our own CPython anyway (and we just updated to 3.5.1). I'd be glad to test the patch during the next iteration.

Guido, I think my use case mixes up other things I find confusing about asyncio: e.g. inablitity to synchronously perform code written as standalone coroutine. It deserves its own task and discussion.

The purpose of this request is that access to a running loop is handy within coroutine. It's already there, since coroutine is executed within its context. However, instead of "self" library provides API to only access some globally default loop.

Python does not require a user to pass self whenever instance methods needs to be called. I consider relationship between event loop and awaitables alike.

As long as there are still things you find confusing about asyncio's model,
you should probably consider yourself unqualified to start proposing new
features...

> Yury, what do you think of this yourself? Maybe you can explain it better
than Ilya?

TBH, I don't fully understand Ilya's case with threads, synchronous coroutines, possible deadlocks etc.  So I'll just explain what I would want out of this.

I don't like a few aspects of the current design of 'get_event_loop'.  Mainly because it creates an event loop implicitly, or it can just fail with a RuntimeError under some conditions.  When one has a few event loops, implicitly called 'get_event_loop' from Future and other asyncio functions won't work correctly.  IMO, this is something that we at least should try to fix (even though it's not an idiomatic use case for asyncio).

I now think that we don't need a new function for getting the currently running event loop.  Instead we can try to improve the `get_event_loop()` function a little.

The core idea is to add a thread-local object `running_loop`, which will point to the currently running event loop in the current thread.  And `Loop._run` method will set/reset a reference to the loop to `running_loop`.

`get_event_loop()` will then try to use the `running_loop` object first, and if nothing is there, fall back to its current implementation.

Now, this is a point where everything becomes complicated. To quote myself,--"Yes, the patch part is easy",--I don't think it is that easy anymore ;)

Currently, event loops are decoupled from policies.  This means that we can't make loops to use some hidden shared thread-local object (`running_loop`) that policies and loops will work with.  There has to be another new public policy APIs that loops will use, for instance: 'set_running_event_loop()'.  This won't be the first case of event loop policy APIs being called from event loops -- another example is the `get_child_watcher` method.

With this new method, `Loop._run` will then look like this:

  def _run(self):
      policy = get_event_loop_policy()
      policy.set_running_event_loop(self)
      try:
         ... # current Loop._run implementation
      finally:
          policy.set_running_event_loop(None)

`policy.set_running_event_loop` call would raise a RuntimeError if it's called if another event loop is currently running.

And the `get_event_loop()` function will look like this:

  def get_event_loop():
      policy = get_event_loop_policy()
      loop = policy._get_running_event_loop()
      if loop is not None:
          return loop
      # ... current get_event_loop implementation

So it all boils down to:

1. Adding a new policy API 'set_running_event_loop'.

2. Updating 'get_event_loop' to return the currently running event loop if it's available, or else resorting to its current behaviour.

With this change it will be completely safe to use "get_event_loop" from running coroutines or just about any code that runs in a context of an event loop.

> TBH, I don't fully understand Ilya's case with threads, synchronous coroutines, possible deadlocks etc.

I feel sorry for sharing that example. It didn't help and made my points regarding original issue unclear, hidden behind example's complexity.

Yury,

> I now think that we don't need a new function for getting the currently running event loop.

May I ask you to elaborate on this? Asynchronous API I'm aware of (including  other languages) typically allows to get "main" (which in asyncio is lazily created for main thread only), and "current". Sometimes there is also a convenient "global" which can be used when you want to do some async work outside. E.g. take a look at public API of Apple's Grand Central Dispatch, specifically `dispatch_get_current_queue`.

Even Python's thread module allows to get "current_thread" instead of passing it into run callback and requiring a user to carry it around by hand.

>> I now think that we don't need a new function for getting the currently running event loop.

> May I ask you to elaborate on this?

In my last message I propose to modify 'get_event_loop' to return the running loop reliably.  To me this is to me the most important thing -- this will make things like Task.current more reliable.

Adding another function (get_running_loop) won't change much. It would have a slightly different behaviour, but likely it won't be used anywhere in asyncio, or other libraries.

Yury, 

> `get_event_loop()` will then try to use the `running_loop` object first, and if nothing is there, fall back to its current implementation.

Do you think hiding "default" event loop completly from a currently executing coroutine is safe?

> Do you think hiding "default" event loop completly from a currently executing coroutine is safe?

Not sure I understand the question.  Anyways, let's see what Guido thinks about this all.

Yury,

> Not sure I understand the question.

If I understood it correctly, get_event_loop() would never return "default" event loop (in terms of current implementation) for a running task, because it always be overridden with "running" event loop. If it's so, are you sure it won't raise any problems with backward compatibility?

> If I understood it correctly, get_event_loop() would never return "default" event loop (in terms of current implementation) for a running task, because it always be overridden with "running" event loop. If it's so, are you sure it won't raise any problems with backward compatibility?

I think this is extremely unlikely.  I hope that there is not much code out there that relies on this behaviour.  But you have a point.

Yury, as you suggested posted to python-ideas (https://groups.google.com/forum/#!topic/python-ideas/ABOe22Mib44)

I agree with Yury's ideas about the implementation of this feature. However, it is a bit confusing to have `asyncio.get_event_loop` defined as:

def get_event_loop():
    policy = get_event_loop_policy()
    return policy.get_running_loop() or policy.get_event_loop()

One would expect `asyncio.get_event_loop` to simply work as a shortcut for:

    get_event_loop_policy().get_event_loop()

The root of the problem is that we're trying to define 3 concepts with only 2 wordings. I think it is possible to solve this issue quite easily by renaming `AbstractLoopPolicy.get_event_loop` with `AbstractLoopPolicy.get_default_loop`. We'd end up with the following definitions:

- default_loop: current default loop as defined in the policy
- running_loop: current running loop (thread-wise) if any 
- event_loop: running loop if any, default_loop otherwise

Changing the API is always annoying, but in this case it would only affect the event loop policies. This is a pretty specific use case, and they'll have to be updated anyway in order to implement `set_running_loop`. asyncio.set_event_loop might be affected too, although it could be kept or deprecated.

Do you think it's worth the trouble?

See https://github.com/python/asyncio/pull/452. I believe that the issue is not resolved (almost).

What's left is documenting how to use get_event_loop and when to use explicit/implicit loop passing.

Closing this issue.  Will open a new one for the documentation update.

I'm very happy that the issue is finally resolved.

But a bit offended that it took Andrew only 4 days to push it :)

> On 07 Nov 2016, at 17:04, Yury Selivanov <report@bugs.python.org> wrote:
> 
> 
> Yury Selivanov added the comment:
> 
> See https://github.com/python/asyncio/pull/452. I believe that the issue is not resolved (almost).
> 
> What's left is documenting how to use get_event_loop and when to use explicit/implicit loop passing.
> 
> Closing this issue.  Will open a new one for the documentation update.
> 
> ----------
> resolution:  -> fixed
> stage:  -> resolved
> status: open -> closed
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26969>
> _______________________________________