classification
Title: IDLE threading + stdout/stdin observed blocking behavior
Type: behavior Stage: needs patch
Components: IDLE Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: John Smith, gvanrossum, taleinat, terry.reedy
Priority: normal Keywords:

Created on 2020-02-22 10:29 by John Smith, last changed 2020-05-27 00:58 by terry.reedy.

Files
File name Uploaded Description Edit
IDLE_threading_issue_min_example1.py John Smith, 2020-02-22 16:50
tem4.py terry.reedy, 2020-05-27 00:58
Messages (12)
msg362456 - (view) Author: John Smith (John Smith) Date: 2020-02-22 10:29
preamble: I am aware that I am not the first to encounter this issue but neither I could identify a preexisting ticket which fully matches nor is the commonly recommended "solution" (stay away from IDLE) satisfying.

environment: win10, python 3.7 (tested with 32 and 64 bit version)

description: If the attached script is started from IDLE the "alive" only shows up once for every input, while the script output "alive" frequently if ran from the terminal with python. So there is a discrepancy between the behavior of IDLE and "plain" python, which can lead to serious "irritations". If the print is replaced with logging.info and the logging is setup to write into a file everything works as expected and equal in both environments.

thoughts: the input call seems to block access to stdout(?) in "IDLE mode". I noticed that there are several topics/post regarding IDLE's stdout/in behavior but I was unabled to find a (convinient) solution besides "just quit using IDLE". It feels strange that the editor bundled with python has such a reputation and features such a deviation in behavior from "plain" python.
msg362467 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-02-22 15:01
If I understand, input() in one thread blocks print() in another thread. 
This sound like a new issue for this tracker, and possibly a bug in IDLE's implementation of std streams through a socket, but I need to see your code.  Please try attaching it again.

Also, try running >py -m idlelib -n path
msg362468 - (view) Author: John Smith (John Smith) Date: 2020-02-22 16:53
py -m idlelib -n path output:

Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
==== No Subprocess ====

WARNING: Running IDLE without a Subprocess is deprecated
and will be removed in a later version. See Help/IDLE Help
for details.

>>> 
waiting for inputalive
alive
alive
alive
alive
alive

 -> so its working as intended
msg362469 - (view) Author: John Smith (John Smith) Date: 2020-02-22 16:54
P.S. here (other computer but with same observed behavior) I have a 3.6 32bit as one can see from the output.
msg362484 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-02-22 23:55
Summary: With IDLE running normally, with 2 processes, "input('prompt')" in the main thread blocks execution of "print('whatever')" in a separate thread.  (Verified on Windows with 3.9 and macOS with 3.8.)  The user can respond to the prompt without being disrupted by continued output.

For standard single process python or IDLE (started with -n), the thread output continues and can be mixed in with both the input prompt and user response and in the example, scroll prompt and even initial response off the screen.

I presume the difference is related to IDLE routing user-code use of the std streams through a 3rd thread that manages the inter-process socket connection.  But I did not see why after more than an hour of looking.

Overall, I think the experience with input() in IDLE is more useful and less confusing, especially to beginners.  (I am having trouble imagining when the intermixing and interference would be useful.)

Guido, do you have any recollection of whether the IDLE difference, when using an execution process, is intended?  Do you regard intermixing input prompts and responses and thread output as a required feature of Python, or as an (unfortunate, IMHO) side-effect of using a dumb text interface?

Note: in IDLE, '>>>' come from IDLE, not the usercode execution process.  Hence a) asynchonous outputs from the execution process can appear on the screen, and b) IDLE, unlike the standard REPL, can and does prevents such text from mixing into and spoiling user code input.  (It may appear after '>>>' but is put before the input area.  This is an intentional interface difference.

My *immediate* inclination is to regard the IDLE difference as overall being a positive feature and leave it alone.  But it should be documented in "Running user code", which is already mostly about stdio differences.  

I don't deny that blocking output while a user responds is a negative.  I just see mixing streams as worse.  To work towards not blocking while not mixing, I replaced the thread loop with
    for i in range(10): sys.stdout.write(f'some text {i}\n')
and separated input(prompt) into sys.stdout.write(prompt) and sys.stdin.readline().

Observations:
1. .write is better as it is atomic, while print() prints '\n' separately, with sometimes undesirable results.
2. The block is from readline.
3. While output to Shell is blocked on readline, typing a continuation-invoking '.' or calltip-invoking '(' in either Shell or editor allows one line of output.  Both send a non-stdio message from Shell to the run process and back.  This might be a clue as to how read/readline blocks.

In the longer term, after some changes to Shell, I could imagine replacing __builtins__.input with a function that would check if sys.stdin/out were IDLE's replacements and if so, send the prompt to Shell tagged as 'prompt' rather than as 'stdout', so that Shell could keep prompt and user response  separate from regular output, similar to the way now done for code input.  If .readline cannot be made to not block output, it might work for Shell to send  the response back tagged as 'response' rather than as 'stdin' and not use readline.
msg362497 - (view) Author: John Smith (John Smith) Date: 2020-02-23 09:26
Interesting finding. Just some thoughs: I have to disagree on the "the experience with input() in IDLE is more useful and less confusing, especially to beginners" part.

Here are some reasons:

1. If a beginner starts with threading and expect a concurrent execution the confusion will be high if there is non in IDLE => implicitly changes of the runtime behavior siliencly enforced by an IDE can lead to misconcepts.

2. It took me (and my team) more than one hour to track a strange issue down to this IDLE related behavior - mainly because I assumed that IDLE would not change the runtime behavior of a program in that way => the deviation in behavior can even confuse experienced developers.

3. During my search I came across several reports of similar issues and most people were told to simply quit IDLE => "quirks" have impact on the reputation.

Personally I like and mainly use IDLE and would like to continue with it.
msg362504 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-02-23 12:06
I tend to think that we should avoid discrepancies with a simple Python command-line shell by default, and in this case I am not convinced that there is a good reason for a divergence in behavior.

I'd be happy to look into debugging this issue in the next few days.
msg362528 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-02-23 17:52
It sounds like either an old implementation restriction or an old misguided attempt at keeping the IDLE console clean. I’m with Tal, let’s fix it and make it more like regular Python. Thanks John for taking the time to report this so clearly.
msg362535 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-02-23 19:59
I found a stackoverflow question (and my then answer) about the same issue:
https://stackoverflow.com/questions/50108354/python-idle-running-code-differently-than-other-ides

Although the code posted does not run, it gives a very legitimate use case (<Enter> is the signal) in which stream mixing is the lesser evil compared to the clearly wrong blocking of the output needed to decide when to give the signal. I now agree we should fix blocking now, if possible.

Tal, I posted my observations partly in the hope that you might be able to extend them.  So please take a look.

I am thinking about how to test a fix.  Test code needs to be able to 1. send code to be executed by run.py, 2. send a response to input() calls in the code, and 3. retrieve all output from the code.  New test machinery, needed not just for this issue, should be a separate issue.

A possibility including the minimum of transport machinery would be to replace the StdInputFile and StdOutputFile in run.py with test classes giving the needed test access.  Without knowing where the block is, it is unclear how much of the run machinery has to be included.

Another possibility, including all of the transport machinery except for the gui display, would be a test interpreter connected to an unaltered run process through a socket, as usual.  pyshell.PyShell (the shell window itself) initializespyshell.ModifiedInterpreter with itself.  A no-gui TestShell could do the same.  The TestShell would also need TestIn/OutputFile replacements.
msg369993 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-26 15:00
This seems to be at least partly intentional: When waiting for user input, IDLE starts a nested Tk mainloop (!), which is stopped by the end-of-line handler. It seems to me that this is what is causing displaying other outputs to be delayed.
msg369994 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-05-26 15:01
Can someone talk about the reasoning behind the nested Tk mainloop? I haven't managed to find anything useful in code comments and such. Guido?
msg370038 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-27 00:58
Researching 'nested mainloop': the one we are concerned with is in pyshell.PyShell.readline, currently line 1078.

            self.top.mainloop()  # nested mainloop()

This was in David Scherer's 'Initial Revision' of 2000 Aug 14 without the comment.  It has since been touched once to add the comment.

2004 Dec 22, KBK, 5c3df35b6b1f48cb48c91b0c7a8754590a694171.
The GUI was hanging if the shell window was closed while a raw_input() was pending.  Restored the quit() of the readline() mainloop().
http://mail.python.org/pipermail/idle-dev/2004-December/002307.html

So no hint of reason.
---

https://stackoverflow.com/questions/17252056/tkinter-nested-mainloop
OP asked about using a nested mainloop for a video player.  There is discussion (dispute) of whether or not this blocked the mainloop.  Tkinter expert Bryan Oakley opined "While it's possible, there is very rarely ever a need to call it. reason to do it, and what you are trying to do probably won't work the way you think it will. –" 

OP Raoul quoted FL "Event loops can be nested; it's ok to call mainloop from within an event handler." (dead link to draft version). It is now in mainloop entry of https://effbot.org/tkinterbook/widget.htm.  There is no explanation of why or what effect.
---

Mark Lutz, Programming Python, discusses recursive mainloop calls as an alternative way to make modal dialogs, without 'wait' and other setup calls.  In the 3rd edition, displayed by Google Books, the half page is expanded to a full page (442-443, Example 9-14, PP3E/Gui/Tour/dlg-recursive.py) and he adds that calling quit() is mandatory (see KBK patch above) and that using 'wait', etc, is 'probably better'.  The latter is what IDLE does.  Since 'modal' means 'disable event handling outside the dialog', the intent must be to suspend the outer event loop, as we see for this issue.
---

While I still think that IDLE should protect user code input in response to *IDLE's* '>>>' prompt, even more than it does now, I now agree that IDLE should not do the same with try user *input()* calls.  The usability of the latter is the responsibility of users who write them.

Not blocking thread prints may just be a matter of removing mainloop() and all corresponding quit() calls.  Proper sequencing may be trickier.  Tem4.py calls input() and print() from both main and thread.  Run directly with 3.9.0b1, it results in

thread start
main input: m  # wait before entering 'm\n'.
main got:  m
thread input: t
thread got:  t

What surprised me is 'thread input:' being held up until 'main got' was printed.  Run from IDLE, the result is

thread startmain input: 
m
thread input: main got: t
 thread got: m 
t

This is more jumbled, not less.  'main input:' is printed before the '\n' after 'thread start'.  etc.  Separating print(...input()) into two statements had no effect.  Neither did removing sleep(.1).  We will have to retest after blocking is removed.
History
Date User Action Args
2020-05-27 00:58:53terry.reedysetfiles: + tem4.py

messages: + msg370038
2020-05-26 15:01:26taleinatsetmessages: + msg369994
2020-05-26 15:00:34taleinatsetmessages: + msg369993
2020-02-23 19:59:56terry.reedysetmessages: + msg362535
versions: + Python 3.8, Python 3.9
2020-02-23 17:52:37gvanrossumsetmessages: + msg362528
2020-02-23 12:06:26taleinatsetmessages: + msg362504
2020-02-23 09:26:15John Smithsetmessages: + msg362497
versions: - Python 3.8, Python 3.9
2020-02-22 23:55:18terry.reedysetversions: + Python 3.8, Python 3.9
nosy: + gvanrossum, taleinat

messages: + msg362484

stage: needs patch
2020-02-22 16:54:31John Smithsetmessages: + msg362469
2020-02-22 16:53:36John Smithsetmessages: + msg362468
2020-02-22 16:50:15John Smithsetfiles: + IDLE_threading_issue_min_example1.py
2020-02-22 15:01:33terry.reedysetmessages: + msg362467
2020-02-22 10:29:37John Smithcreate