(This touches a bit on things mentioned in #14111)

I'm looking through the debugger code in IDLE, and now understand that it's essentially trying to make a GUI program act like a command line program. Hence the nested call to mainloop(), when we receive a trace execution callback. This is the equivalent of saying "stop here until I input something, and then continue along" at which point the callback completes and we return to the running program.

Right now, if you run a program with the debugger on, and just hit Go, it's just like you're not running it in the debugger at all.

I've mentioned elsewhere (#15347) that I suspect the nested mainloop is behind some of the instability problems related to quitting when the debugger is active.

But if we don't assume that each time we print where we are we have to wait for input, we can do some more interesting things. Most importantly, we should be able to say "Go", and actually watch our program being executed. In other words, whenever it goes to the next statement, the debugger actually shows that statement, highlights the line in the source file, updates variables, etc. -- all without user interaction. 

Someone can sit back and watch, getting a better understanding of what their program is doing while running. You'd be able to see if a program was stuck in a loop, without necessarily going through it one statement at a time, step by step, or setting breakpoints. 

It also makes it clearer that the program is running because you see something happening... now there's not great feedback in that regard. Similarly, some of the issues with pausing/continuing or stopping the program become a bit easier to deal with.

I agree that without breakpoints set, Go is currently useless. I believe that you are proposing that it become a 'Start slide show' button, where a 'slide' is the execution of one statement.

I really like this idea. Like music or dance notation, a programs is a static representation of a dynamic performance.  An expert programmer, like an expert musician or dancer, can imagine a performance.  But without seeing a calculation performance, novice programmers may have trouble knowing what they are supposed to be doing.  For Python, calculation results are made visible by namespace bindings. 

Suggestions based on experience with automated slide shows:
* a user-settable delay between statement execution (possible 0).
* use space bar to stop and start
* allow single stepping while stopped, and space-bar restart after

A debugger-specific suggestion:
* step over functions whose code is not in a clean editor window (no unsaved edits)

try:
    filename = f.__globals__['__file__']
except AttributeError:  # builtin function or method
    <step over>
else:
    if filename not in filelist or buffer dirty:
        <step over>
    else:
        <step>

> trying to make a GUI program act like a command line program.
I don't think I understand that completely, except perhaps that the reason for the nested event loop is to avoid blocking the main event loop.  And maybe that you believe we can avoid the likely troublesome nested loop without blocking the main loop. (It does seem like judicious use of .after, commands, and callbacks should avoid needing a nested loop.)

Regarding the nested loops, what's happening is:
- IDLE tells interpreter to run program
- Interpreter now has control and is running program, start to end
- Because execution is being traced, before every statement we get a callback

If we didn't use the nested loop and just returned back immediately after we're called, the program would go onto the next statement, and the next one, etc. (it's in control).

Instead, we block to force the program to wait until we do something, at which point it continues to the next statement, after possibly modifying the conditions under which the execution traces fire.

If instead, we tell the execution tracing to actually stop on every statement (rather than running through start to end), we'll get a callback for each statement (to update our display), but can decide to stay stopped, or immediately continue on, based on whether we last did single-step, in/out, go, etc. That way it is more event-based: we don't need the nested event loop to 'block' the program from running.

Totally agree on your suggestions (speed setting, and whether or not to show where we are outside our own code).

Ok, I lied. The program runs through start to finish regardless, and all the 'stopping' and breakpoints is really just very selectively deciding which subset of execution tracing events to pass back to the debugger. So you really do need to 'block' in those callbacks.

Doing this in the remote debugger is relatively easy with just a small tweak (turning a synchronous call into an async one, so we control when the callback 'completes'). 

Doing so in the 'run without subprocess' scenario would be tougher, and we'd need to either keep the nested loops or do some funky thing with threads.

I had a wild idea, possible a result of too much ignorance.  Run the main debugger in the user process, with a separate root and event loop.  Add to the communication protocol options to send a list of breakpoints and modules to be traced into and to send (module, lineno) pairs back, for highlighting in the editor.

I am willing for a revised debugger to not work with -n.  We want that to go away anyway.  Roger Serwy sent some of us a prototype pipe implementation a year ago when I was busy with GSOC, and I did not pursue it yet.

Frighteningly, your wild idea is close to how it actually works now, as per the ASCII art at the top of RemoteDebugger.py. :-) 

The authors of RemoteDebugger and rpc.py went to fantastic lengths to make this transparent so that everything looks local. If in my previous life I didn't develop groupware infrastructure I'd have been screwed trying to follow what was actually going on!

If you want to be impressed, turn on debugging in rpc.py (at start of RPCHandler class), uncomment the print statements in RemoteDebugger.py, start IDLE, open up the debugger, and type "1+1" into the shell.