The last week of last February, there was a discussion of this topic on python-ideas as part of "How the heck does async/await work in Python 3.5".  I would like to re-start the discussion along with rescuing the two big chunks of code that were posted.

Guido, you said "it would be very useful to have an asyncio loop integrated with Tkinter".  Were you thinking of something added to either the asyncio or tkinter modules? What would be the minimum that you think would be worth adding?  What would a mininal test look like?

There are, of course, multiple strategies.  Maxime posted the code I copied into the attached tkselector.py.  No example usage was given.  I cannot run it on Windows because it uses a unix-only function.

Maxime, for us to use this, you need to sign the contributor agreement, which can be done via the net.  See https://www.python.org/psf/contrib/

As soon as I submit this, I will also upload my tkloop.py.  It defines a TkEventLoop class that adds root.update in the run_forever method.  It then uses the class for working example of a asyncio callback modifying a tk widget at times intervals.  As I note in the doc string, there is an issue with asyncio's _run_once() blocking forever.  I recently noticed that idlelib.run.main does something similar, running a loop that gets user input with 1/20 second timeout and calling tkapp.eval('update') whenever there is none.

Motivations:

1. Mix gui events (key, mouse, and others) with i/o events.

2. Use gui events with the new async syntax.  My simple example, in the callback typical of tkinter apps, has widgets, callback, and dynamics defined in different places.  I would like to be able to write the app in a more 'natural' style, using loops: something like

async def display_date(interval, end_time, loop=ai.get_event_loop()):
    label = tk.Label(root)
    label.pack()
    async for tick in timer(interval, end_time, loop):
         label['text'] = datetime.datetime.now()

Some python-tkinter beginners try to do something like this, except that for time delays they use while and time.sleep.  There have been many Stackoverflow questions about the resulting failures.  I would like to find out if 

I presume that converting my example to something like the above, using TkEventLoop, could be done easily enough by someone who knows how.

For only supporting tk events, it would be better to have a tkinter implementation of the needed subset of asyncio.events.AbstractEventLoop. The run and call methods are needed, the io methods not, but I am not sure where to draw the line in between.  The implementation of a few methods should be fairly easy: run_forever is mainloop, close is destroy; run_until_complete will be harder.

I expect it should eventually be added to tkinter. But I also think it
might be worthwhile to first develop it as a 3rd party package on
PyPI, to see if it can actually be done well enough to put it in the
stdlib.

I guess a demo app should be part of the project; the demo should show
that some independently developed asyncio code (perhaps derived from
examples/crawl.py in the asyncio repo) can run while also maintaining
a UI built using Tkinter responsive. Ideally there would also be a
demonstration of how the UI and the crawl code interact. And no
threads.

I've signed the CLA.

Guido, that seems like a reasonable roadmap.  The examples directory is not in the CPython repo, but I found it here.
https://github.com/python/asyncio/tree/master/examples

For a demo based on crawl.py, the goal would be a live status report.  Perhaps as files are fetched, they could be entered into a Treeview-based url tree, with columns for the statistics. (I still have to learn to practical use of Treeview.)  This would require adding widget data insertion commands within the crawl code.  Who actually wrote it, that would understand it?

Responsiveness should be a matter of prioritizing events.  Using the  asyncio loop as a base, I did the minimum needed for the first demo and for the initial part of the follow-up below.  For general use, _run_once should be modified to always let gui events be handled 'soon' by either not blocking or having a short timeout.  The root.update call could be moved from run_forever to multiple places within _run_once.  At an extreme, the _process_events implementations could be modified to call root.update() after processing each io event.  According to timeit.timeit(root.update), a do-nothing call takes less than 4 microseconds on my machine.

---

I implemented my idea of updating a tk widget in an async for loop.  The attached tkaloop does so, with the syntax I proposed previously.  There is only a change to the timer api.  With respect to the datetime clock, the behavior is essentially the same as with the first code.

Given that one can do the same thing with a normal for loop and explicit update (as some beginners try to do), is this useful?

1. Responsiveness: A normal for-loop blocks, freezing the gui.  To test that this is not so here, I added a button to do something visible - change a background color.  While the clock is running, it only sort-of works, because clicks are only processed after each 1 second tick.  The reason is that the select call in _run_once gets a timeout that is the minimum time to a scheduled event.  (The same should be true in tkloop.py if similarly modified.)  The next step is to modify _run_once as discussed above.

Tk allows one to specify delays to the nearest millisecond and on my machine, that precision is real.

from tkinter import *
import time
        
root = Tk()
root.withdraw()
timer = time.perf_counter
##n = 999
##delay = 1
n=1
delay=337
def tick():
    global n
    if n:
        n -= 1
        root.after(delay, tick)
    else:
        print(timer() - start)

tick()
start = timer()
root.mainloop()

prints .3370... in multiple runs. 1000 loops with a delay of 1 ms takes 1.05... seconds.  Similar performance would be a target for run_forever.

2. Parallel looping.  Supposed we wanted to automate the background color changes, with an interval of, say x seconds.  If x is, say, .3, then one could use .1 second ticks and an update block that flips the background every 3 ticks and the clock every 10.

But what if the intervals are not so conveniently commensurable or the separate updates not so easily mashed together into one loop block?
Parallel looping with chained callbacks, whether with tk or asyncio is fairly easy.  I presume it could be done with asyncio coroutines (but have no experience).  What I would like, but cannot see, is a way to do so with async for.  Within a function, async for blocks execution of remaining code after the for statement the same as normal for statement.

My conclusion thus far: with the response issue solved, a single async for loop could work for some time-driven applications, including some that beginners write, but not all.

An alternative for reducing the callback boilerplate is an animate function.  I have thought of writing one for tkinter, but have not yet because I have not yet needed one for IDLE.

> Who actually wrote it [crawl.py], that would understand it?

I wrote it, and I can probably still recover my understanding of it.  Basically you say "python3 crawl.py xkcd.com -q" and it crawls the XKCD.com website -- that takes about 4 seconds on my machine, fetching 1764 urls. The -q is to make it more quiet while it's working -- but somehow it doesn't suppress the final report, which includes a full list of all crawled URLs -- this could use some surgery to pass the log level into Fetcher.report().  A test site that has only one URL is www.pythonlabs.com.

I ran the test program (tkaloop.py) and it appears to work, with one caveat: updates in response to Tk events (specifically clicking the button that toggles the color of the "Hello" label) seem to take an effect only once the self.root.update() call in your run_forever() is made, which appears to be only once per "interval" (i.e. 1 second).

Hopefully future development will make it so that when a Tk event happens, *somehow* the asyncio selector gets woken up.

(PS. Please don't abbreviate asyncio to aio. It makes your code harder to read for asyncio veterans. :-)

New file loop_tk.py solve the responsiveness problem, at least for this example, by using the asyncio loop as it is and doing tk updates in a callback loop. It works with both SelectorEventLoop and ProactorEventLoop.

I was inspired to try this, instead of my planned modification of _run_once, but Yury's post on uvloop http://magic.io/blog/uvloop-make-python-networking-great-again/.  I realized that any modification of asyncio.XyzLoop would only work for that class, where as a standard callback loop would work with any sufficiently compatible loop.

Yury, if you happen to read this, could you try loop_tk.py with the two uvloop lines added and get_event_loop uncommented?

Using the callback loop from loop_tk.py, 3.6 repository IDLE runs with an asyncio loop.  See idle-async.diff.  So far, it seems about as snappy.  I need to do a more stressful (longer running) gui operation test to be sure: change syntax highlighting with 10 or 20 editor windows open.

Speed tests:
First -- raw loops/second.  The attached loopspeed.py has code for tk and asyncio that is equivalent as I know how.  The tkasync loop is for the asyncio loop with a root.update() call added, where the root is a fresh instance of Tk().  (Reusing the old root resulted in the tk loop restarting.)
  tcl/tk loop:  13300 l/s
  asyncio loop: 14900 l/s
  tkasync loop: 10900 l/s (uncomment commented lines)
The two bare loops have a std.dev. of roughly around 100, with +- up to 300.  This suggests to me that the tcl loop might be written in tcl, not C.  This result reduces my motivation to write a tcl/tk based replacement for BaseEventLoop, thought I still might.

Second -- IDLE syntax highlighting speed.  I opened 9 idlelib files in installed 3.6.0a3 and patched repository 3.6.0a3+.  I switched between a light default and custom custom theme and hit apply.  For stock IDLE, it took 2 subjective mental 'thousand and one...' counts.  For patched IDLE, three.  Reducing the call_later delay from .01 to .001 reduced the count to about two and a half.  Replacing call_later with call_soon may have given a small further speedup.

Changing highlight theme (or changing font or font size, which trigger re-highlighting) with at least a few thousand lines of code to process, is sufficiently rare that a slowdown of 25% or less does not bother me.  When I want to try to experiment with using asyncio with IDLE, I will use the current patch until it proves not to work.  I am now thinking of this as perhaps just a doc issue.

On what platform did you measure that? On a Mac, with Python 3.5.2, I
get very different numbers:

tkloop: 9000/sec
asloop: 90000/sec (about 10x!)
tkasyncloop: 2500/sec

Windows 10, repository default (3.6), with tcl/tk 8.6.4. I am not sure what to make of your results.  I have an impression that tk on Mac was considerably rewritten for 8.6.   Ned would know more about that.  Which were you using? "print(tkinter.TkVersion)"

OK, I'm playing with this too. I merged your loop_tk.py example with crawl.py and am attaching the result as tkcrawl.py. I don't like global state so I added root and loop as parameters to a few places. Hopefully you have the asyncio repo checked out so you can compare this with the original crawl.py -- it basically has your code (with globals replaced by locals or instance variables) and these three lines in main() once loop is set:

+    root = tk.Tk()
+    t = asyncio.Task(display_date(root, loop, 100, 1))
+    tk_update(root, loop)

and a loop.run_forever() call just before the loop.close() call, to keep the app running once it's done.

There are also two imports near the top (datetime and tkinter as tk).

(Oh, and I just realized I made some changes to crawl.py to suppress displaying the URL of every HTML page retrieved when -q is given. I will commit that.)

Running this will display the widget with the hello label and the button to change the color, *and* update the date every second, and it's totally responsive, all *while* it is fetching contents from e.g. xkcd.com.

PS. I recommend this command line to play with tkcrawl.py:

python3 tkcrawl.py xkcd.com -q

The fetching process takes about 6.8 seconds to fetch 1765 URLs on my machine. The original crawl.py takes 6.2 seconds.

I guess the next step would be to hook up the logger to a Tk text widget. It would also be nice to have a set of Tk widgets keep track of the state of the connections in the ConnectionPool, showing for each connection what its state it. It's a little tricky because connections get opened and closed at random; I would recommend using a table assigning connections slots in the widget. The number of lines in the widget is limited by max_pool.

(I would code up more of this myself except my Tkinter is rusty!)

Oh, my tkinter.TkVersion is 8.6.

I was thinking of starting (Mon. afternoon/evening, its now 2:40 am) with updating done stats with every done fetcher, updating stat display every second, and listing urls with html body, with a way to view the body.  Log to text easy too.  Table of connections will require experiments with ttk.Treeview, which I want to do anyway.

[On #27579, Lodovic Gasc volunteered to open and help with an issue on the github asyncio-doc project.]

Ludovic, here is my current suggestion for the asyncio tutorial or how-to.

Adding a Tkinter GUI to an Asyncio Program
-------------------------------------------

Assume that the tkinter Tk instance is called 'root' and the asyncio loop is called 'loop'. Add a callback loop like the following.

def tk_update():
    root.update()
    loop.call_soon(tk_update)  # or loop.call_later(delay, tk_update)

Call tk_update before each loop.run_forever() or loop.run_until_complete(...) call. Do not call root.mainloop() or root.quit().  Loop.stop() or completion of run_until_complete will stop the tk_update loop.  This method is used in the following example.
---

Follow this with loop_tk2.py as an example.  It is carefully written to be a template that people can build on.  Some notes:

0. I REALLY like being able to write tkinter animations with async for.

1. I initially called loop.close immediately after loop.stop in App.close.  This is a natural think for a tkinter programmer to do.  But it does not work because loop.stop does not *immediately* take effect, and loop.close will typically get called before it does.  Now I understand that loop.close() must physically follow loop.run_xxx() to ensure that it temporally follows the actual stopping of the loop.

2. I did not initially make a list of tasks.  This worked once, but on the next run I got a "Task was destroyed but it is pending!:" message.  Cancelling all 'forever' tasks makes for a clean shutdown.

If/when you open a thread, please post url here and I will subscribe.

[Guido, I will adapt this example for crawl.py.]

Isn't this going to busy-wait, esp. with the loop.call_soon() version?

Even with loop.call_later(0.1, tk_update) you're wasting battery power
even if no network activity and no UI activity is happening. I'm not
eager to document this as the right way to do things, even if it
unblocks some apps in the short term.

A proper solution IMO should somehow merge the selectors so that a
single select() or whatever wakes up when either network I/O happens
or a UI event comes in (which could be something that Tk transparently
handles but it still needs to be given the chance, through
root.update()).

This afternoon, I realized that I would replace the async for loops with while loops with await sleep and eliminate the Timer class.  I presume the tutorial will have other examples of how to write an async iterator.  (If not, I would put is back in.)  I also notices that the updater could also be turned into a coroutine, but decided to leave it as it is.

> wasting battery power ?!  We live in slightly different computing universes ;-).  But I get the point.  The last two files I uploaded use call_later and I should stick with that.  I should also add a note that the innermost asyncio loop function sleeps when there is nothing to do and that the tk updater wakes it up, if necessary, to check for gui events even when there are none.  Updater is polling rather than interrupt based.  Or in other words, it interrupts asyncio to poll tk.

I should also say that the update interval is passed in to the class so one can make an instance-specific tradeoff between overhead and responsiveness and display update frequency.  A read-only display might be updated just once a minute.

If loop.call_later(0.1, tk_update) is actually a problem on a system, then IDLE would likely be twice as bad, as it also has a busy loop in the user process, polling both the socket connection and calling tk update 20 times a second.  There are also loops in the IDLE process.


I agree on the ideal solution and on the key component, which is to sleep until there is a ready Task or file event.  _run_once does this in the select call by adjusting the timeout to the minimum time to the next ready task (possibly 0).

Tcl has such a component, with the addition of simultaneously waiting for window events -- but it only only includes file events on unix.  (It took a brave and talented group to try to reconcile the Unix and Windows models.)

Here is a simplified Python version of Tcl_DoOneEvent. http://www.tcl.tk/man/tcl8.6/TclLib/DoOneEvent.htm  

Tcl has window, file, timer, and idle events.  Window events include user key and mouse events and other from the graphics system.  The first three types all go in one ready queue. Idle events are those that affect what the user sees on the screen and go in a separate, lower-priority queue.

def do_one_event(sleep_ok):
    if ready:
        process(ready.pop())
        return True
    load_ready()
    if ready:
        process(ready.pop())
        return True
    if idle:
        for event in idle:
            process(event)
        return True
    if sleep_ok:
        sleep_until_event()  # the hard part
        load_ready()
        process(ready.pop())
        return True
    else:
        return False

def load_ready():
    # In some unspecified order
    ready.extend(get_window_events)  # graphics system
    ready.extend(get_file_events)  # select
    ready.extend(get_timer_events)  # priority queue pops

Update processes all events available without sleeping. http://www.tcl.tk/man/tcl8.6/TclCmd/update.htm
Mainloop continues (with sleeps) while there are toplevels and not stopped.

def update():
    while(do_one_event(sleep_ok=False)): pass

def mainloop():
    while toplevels and not stop:
        do_one_event()

Sleep_ok is actually a dont_sleep bit flag. DoOneEvent has other flags to select which types of event to process.  Hence

def update_idletasks()  # all currently ready
    do_one_event(dont_sleep | idletasks)

It is possible for the idle queue to get starved for attention.  Hence the existence of update_idletasks and recommendations to call it in certain situations.


It would also be possible to call (from Python, via tcl) do_one_event(dont_sleep | gui_events) IF it were known that a gui event was ready to be retrieved.  It is knowing that, without polling in a 'busy loop' that is hard to impossible.  If it were possible, an extra call to do idletasks would also be needed..

In summary, I see this as a situation where practicality beats a possibly unattainable purity.

> A proper solution IMO should somehow merge the selectors so that a
> single select() or whatever wakes up when either network I/O happens
> or a UI event comes in (which could be something that Tk transparently
> handles but it still needs to be given the chance, through
> root.update()).

I think we can do something similar to self-pipe trick: have a pipe
and a reader for it registered with 'loop.add_reader'. Whenever
a UI thread have an event to process, it should write a byte to the
pipe.

I'm not sure if there's a way to do that with Tk.

OK, in the context of IDLE it probably doesn't matter (though I recall
that IDLE was given a hard time many years ago by people complaining
about that same busy-waiting -- I guess batteries have improved
somewhat since then).

If you're interesting in this, you should take a look at what Twisted has done, since they have a history of integrating with various GUI toolkits.
You'll find everything in the "twisted.internet" package:
https://github.com/twisted/twisted/tree/trunk/src/twisted/internet

It appears that for Tk, they use an ugly busy-loop trick as proposed here.

See also (perhaps)

https://bugs.python.org/issue47190

I posted look_tk3, I believe, as an answer to an SO question. 

https://stackoverflow.com/questions/47895765/use-asyncio-and-tkinter-or-another-gui-lib-together-without-freezing-the-gui/47896365#47896365