classification
Title: IDLE: thread hang, possibly related to print
Type: behavior Stage: resolved
Components: IDLE Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: PythonInTheGrass, kbk, pitrou, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2011-01-14 17:37 by PythonInTheGrass, last changed 2017-06-30 00:12 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
e.py PythonInTheGrass, 2011-01-14 17:37
e10909.py terry.reedy, 2011-01-21 23:42 3.2 version
Messages (11)
msg126282 - (view) Author: Scott M (PythonInTheGrass) Date: 2011-01-14 17:37
New to Python; be gentle if I've simply missed something. i'M running on Windows XP, using a recently downloaded 2.7.1. I'm running by hitting F5 in IDLE.

The attached .py creates 2 threads, one which updates a Tkinter label 10 times a second forever, and one which sleeps for a second and then thrashes the CPU for ~4 seconds, forever. I wrote this to see how Python dealt with CPU-pig threads, and was pleased to see both threads got to run in a decently interleaved way, even in the face of a tight loop.

But at a random point, one of the threads (which one varies) stops running. The other continues fine. The problem manifests in less then 5 minutes (often less than 1).

At least once, but not ever time, the thread that stopped gave a traceback:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "D:\Python27\lib\threading.py", line 530, in __bootstrap_inner
    self.run()
  File "F:\Python27\MyProjects\e.py", line 24, in run
    print "evil!"              #<--- point 1
  File "D:\Python27\lib\idlelib\rpc.py", line 595, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python27\lib\idlelib\rpc.py", line 211, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python27\lib\idlelib\rpc.py", line 240, in asyncreturn
    response = self.getresponse(seq, wait=0.05)
  File "D:\Python27\lib\idlelib\rpc.py", line 280, in getresponse
    response = self._getresponse(myseq, wait)
  File "D:\Python27\lib\idlelib\rpc.py", line 305, in _getresponse
    cvar = self.cvars[myseq]
KeyError: 8680

BUT in some cases there wasn't a traceback (the last time there wasn't, it was BThread that had stopped running; the label was no longer updating.)

If you comment out the print at point 2, or point 1, it seems to work fine, at least for as long as I cared to watch it.

Also, I've noticed that if I close the app's window, that at least one thread keeps running and writing to the Python shell console. (One generally dies because the Tkinter label has gone away). They are both marked as daemonic; shouldn't they stop more or less instantly?
msg126790 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-01-21 22:34
What happens if you run your program without IDLE? 
(either with right-click and run or run from Command Prompt window?) 

I would not be surprised if your problems go away. IDLE runs a saved file in a separate pythonw process. Printing (or writing) to stdout requires sending output back to the IDLE process. I can imagine that doing that from 2 subthreads could lead to conflicts.

If the problems do go away, then we should probably close this as won't fix. IDLE is a development environment, not a production running environment
msg126792 - (view) Author: Scott M (PythonInTheGrass) Date: 2011-01-21 23:06
It hasn't failed yet when run "straight".

Here's the issue, though. I'm going to be introducing Python as the scripting language of choice, to a bunch of people who are less than fluent in programming. Because debugging is not an obvious concept to them, they are going to hunt for problems by sticking in print statements everywhere, and the environment is inherently multithreaded, so it's hard for me to imagine they won't experience this bug. I hit it in my very first multithreaded experiment. And I can't possibly ask them not to use IDLE - without syntax coloring and a friendly editor to show them which line is wrong, they will never get off the ground.

It would be tolerable if there was at least a traceback, but if threads vanish silently, they will not realize it's a environment problem; they will either assume that they did something inexplicably wrong and that "scripting is too hard", or it will get around that I provided a "broken" solution.

So even if it's "just the IDE", it's still life and death for this project. "Won't fix" probably means I have to abandon Python.
msg126799 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-01-21 23:42
FWIW, I downloaded, edited for 3.2, and ran. 45 minutes later, with counter in box near 16000, and output sequence looking about the same as initially, I killed with Task Manager (also on winxp). You might try same on your system.
msg126815 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-01-22 03:43
I ran the 2.7 file (on 2.7) and after several minutes, threadA stopped printing, but no error message.

Antoine, could improvement from 2.7 to 3.2 have anything to new with the new implementation of the Global Interpreter Lock?

Scott, if you can do your project with 3.2, I strongly recommend it. Even with 2.7, one can run files outside of IDLE even though you edit with IDLE. You just give up the F5 convenience. Or, I believe there are other editors without a shell which do so directly. Or, you could write a dp() (debug print) function that writes to a scrolled window in your toplevel app window (where running numbers appear now.
msg126837 - (view) Author: Scott M (PythonInTheGrass) Date: 2011-01-22 13:25
Moving to 3.x means redoing large swaths of the extension I just wrote. It's only a couple thousand lines, but it was my first extension and it cost me a week of my life in Google, and it does a lot with strings.

I haven't pulled down the source code for Python's internals yet, but from the one trace back I did get in the failing thread test app, the problem was inside print and it looked like it was inside the mechanism where print gets redirected. I hope the problem isn't anything as fundamental as the GIL, because that throws the stability of all of 2.7 into question (and it's been solid in all my tests; I do a lot with threads in my extension, all going after the GIL and diving into Python code.)
msg126839 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-22 14:39
> Antoine, could improvement from 2.7 to 3.2 have anything to new with
> the new implementation of the Global Interpreter Lock?

Perhaps, but then it's pure luck. 
Looking at the traceback, IDLE seems to replace sys.stdout with its own thingie when spawning a separate Python program. Perhaps that thingie isn't thread-safe.
msg173088 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-16 20:13
Is this issue reproduced now, with Python 2.7.3+?

Is this issue reproduced with issue9290 patch?
msg183776 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-09 00:08
I re-ran in 2.7.3. First time, B froze at 81, second time neither froze for a few minutes until I used task manager to stop. I then remembered that accessing the gui from a 2nd thread is not supported.

Quoting from msg179101 on #16823, which I turned into a doc issue:
'''
What you are doing appears to be unsupported (invalid). From
http://www.astro.washington.edu/users/rowen/TkinterSummary.html
"all Tkinter access must be from the main thread (or more precisely,
from the thread that calls the mainloop). Violating this is likely to
cause nasty and mysterious symptoms such as freezes and core dumps."
'''
You got the unpredictable random freeze part. But perhaps the quote is wrong (#11077).

But to be sure that was the problem, I reran with B's gui access commented out. It stopped again as about 500. So there is a problem with just printing.

I tested 3.3 and stopped it with TaskManager at 18000 and A still running too.

> Is this issue reproduced with issue9290 patch?

That was just last January. Unless and until I can get new executable with a working tkinter, I cannot test that and other recent patches. Let's hope that 3.2/3.3 still run correctly.
msg296685 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-23 07:23
If there is no bug in 3.6/7, this should be closed.
msg297332 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-30 00:12
In 3.6.2rc1, the counter has reached nearly 40000 after 40 minutes.  So no  bug in current 3.x.  I ran a couple of times with 2.7 and got 3000-4000 both times.  I am no longer patching IDLE for 2.7, so closing.
History
Date User Action Args
2017-06-30 00:12:10terry.reedysetstatus: open -> closed
versions: - Python 3.4, Python 3.5
messages: + msg297332

assignee: terry.reedy
resolution: wont fix
stage: patch review -> resolved
2017-06-23 07:23:57vstinnersettitle: thread hang, possibly related to print -> IDLE: thread hang, possibly related to print
2017-06-23 07:23:27terry.reedysetmessages: + msg296685
2014-10-03 03:44:46terry.reedysetstage: needs patch -> patch review
versions: + Python 3.4, Python 3.5
2013-03-09 00:08:43terry.reedysetmessages: + msg183776
stage: needs patch
2012-10-16 20:13:55serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg173088
2011-01-22 14:41:47pitrousetnosy: + kbk
2011-01-22 14:39:46pitrousetnosy: terry.reedy, pitrou, PythonInTheGrass
messages: + msg126839
components: + IDLE, - Interpreter Core
2011-01-22 13:25:28PythonInTheGrasssetnosy: terry.reedy, pitrou, PythonInTheGrass
messages: + msg126837
2011-01-22 03:43:16terry.reedysetnosy: + pitrou
messages: + msg126815
2011-01-21 23:42:44terry.reedysetfiles: + e10909.py
nosy: terry.reedy, PythonInTheGrass
messages: + msg126799
2011-01-21 23:06:34PythonInTheGrasssetnosy: terry.reedy, PythonInTheGrass
messages: + msg126792
2011-01-21 22:34:48terry.reedysetnosy: + terry.reedy
messages: + msg126790
2011-01-14 17:37:25PythonInTheGrasscreate