This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Unsupported provider

classification
Title: IDLE and pythonw.exe stderr problem
Type: behavior Stage: patch review
Components: IDLE Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: anthonypjshaw, asvetlov, cheryl.sabella, ggenellina, ned.deily, terry.reedy
Priority: normal Keywords: patch

Created on 2011-12-11 19:38 by roger.serwy, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
idle_pyw.patch roger.serwy, 2011-12-11 19:38 review
Messages (15)
msg149244 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-11 19:38
Running IDLE on Windows typically uses pythonw.exe. Unfortunately any error messages written to stderr will cause IDLE to terminate abruptly without an error message. This is due to __stderr__ == None.

Attached is a patch against 3.3a0 for idle.pyw to redirect stderr messages to a dialog box. This allows IDLE to keep running so that the user can at least save their work before closing IDLE.
msg149316 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-12-12 15:35
"terminate abruptly"? I thought that print(file=None) silently returned, without printing but without an error.
A delayed popup to display (otherwise discarded) output is a nice feature, though.
msg149317 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-12 15:46
You can try triggering bug #8900 quite simply. From a shell or an editor, press Ctrl+N and then Ctrl+O. Open a file and watch IDLE terminate abruptly.

Also, see #12274.

If you want to play with this problem further, try adding a "raise Exception" to a constructor for one of the extensions. Running IDLE using python.exe will display a traceback in the console, but IDLE keeps running. However, IDLE won't even bring up a window when using pythonw.exe; it just fails to launch from the user's perspective.
msg149414 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-14 01:44
Here is a list of open issues that describe IDLE suddenly crashing, which can be traced back to pythonw.exe:

#4765, #5707, #6257, #6739, #9404, #9925, #10365, #11437, #12274, #12988, #13052, #13071, #13078, #13153

This patch does not fix these errors, but at least these errors will no longer be fatal. This patch also "future-proofs" against yet to be discovered (or yet to be introduced) bugs into IDLE.
msg149427 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-12-14 05:54
IDLE.app on OS X also has the issue of stderr messages not being presented to users unless you know to look in the system log where they get written.  So writing to stderr is not fatal but displaying them in a popup would be an improvement.
msg149481 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-14 21:54
I don't have a Mac to test against. Is there anything I need to do to improve the patch?
msg192092 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-06-30 23:38
Todd, do you have a Mac to test this on?

This patch treat sending messages to a widget as a backup option. In 18318 I propose we make Idle a true gui app, with all messages other than 'No tkinter' handled by the gui. Console writing, when available, would then be a secondary option for those who want it. But in the meanwhile, I would like this or something like it applied.
msg192104 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-07-01 03:23
Yes I have a Mac and I am glad to help, so I gave it a test run tonight.  The first thing I did was apply the patch  then I ran idle from the console like so:
./python.exe Lib/idlelib/idle.py

For testing I used a simple print command to print to stderr:
sys.stderr.write("spam/n")

which I got the output of
spam/n6

I also tried to use the newer print function:
print("fatal error", file=sys.stderr)

Python 3.4 on the Mac behaved exactly the same way with or without the patch.  I got the stderr output in the Python shell and nothing appeared in the console.  With the patch applied I saw no dialog box to capture the stderr output.  Maybe I didn't perform the test correctly?

Another thing to consider is for Mac IDLE runs in a special mode via macosxSupport.py which I turn on by forcing runningAsOSXApp() to always return True.  Even after setting runningAsOSXApp() to true a dialog box does not appear when writing to stderr.

Maybe I am not testing this patch correctly? Let me know if I can do anything else to help, thanks.
msg192105 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-07-01 04:14
Print in the user process goes to shell window. You need to stimulate (or just add) print or warn in the idle process, which normally goes to console, or nowhere. It is hard (intentionally, I am sure) to dynamically manipulate idle process code. Roger said "try adding a "raise Exception" to a constructor for one of the extensions". I would start with a warning in PyShell. See doc or test_warnings for an example warnings call.
msg192223 - (view) Author: Dev Player (devplayer) Date: 2013-07-03 07:24
I may be mistaken but I thought, as of not too long ago, that pythonw.exe is no longer needed by current versions. 2.7.5
msg192238 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-07-03 14:02
No change. pythonw.exe is still present and in use.
msg192512 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-07-07 04:26
Terry,
   Bottom line I can't seem to get this patch to do anything for me.  Before the patch is applied IDLE seems to be handling warnings and exceptions just fine in PyShell on the Mac.  I get no crash and the output matches the normal console.  Here is a small test program I wrote:
import warnings

def fxn():
    # user warnings are not ignored by default and will cause a dump of information
    # to standard error.
    warnings.warn("User warning: Warn on purpose for IDLE", UserWarning)

if __name__ == "__main__":
    fxn()
    print("the program should not terminate with the warning, but keep on running")
    a = 10 * 1000
    print(a)

    # exception testing each of these will stop the program
    # divide by zero
    b = 10 * (1/0)
    print(b)
    # variable not defined
    c = 4 + spam*3
    print(c)
    # can't convert 'int' object o str implicitly
    d = '2' + 2
    print(d)

Then I wanted to make sure I was executing the patched code so I made sure I called idle.pyw (normally I wouldn't do that I would use idle.py).  After I called the correct script I changed the code to force std error to ErrorNotify class around line 101:
import sys

##if 0: # For testing
##    sys.__stderr__ = None
##    sys.stderr = None

if sys.__stderr__ is None:
    sys.__stderr__ = ErrorNotify()

if sys.stderr is None:
    sys.stderr = ErrorNotify()

if sys.__stdout__ is None:
    sys.__stdout__ = ErrorNotify(devnull=True)

if sys.stdout is None:
    sys.stdout = ErrorNotify(devnull=True)

sys.__stderr__ = ErrorNotify()
sys.stderr = ErrorNotify()

I would expect after this code runs any message sent to stderr would go through the ErrorNotify class and a widget should appear with the stderr output.  Even after this change I can't get the error widget to appear.  I have not given up yet but I wanted to provide a status update.
msg212310 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-02-26 21:56
Idle start up seems unnecessarily fragmented into multiple files.
  idlelib/__main__.py 
  idlelib/idle.py
  idlelib/idlew.py
can all be started from the command line by name with either python or pythonw or run once by import. idlelib/__main__.py can also be started by 'python(w) -m idlelib'.

I checked that in Command Prompt
  C:\Programs\Python34>python lib/idlelib/idle.py
  C:\Programs\Python34>python lib/idlelib/idle.pyw
  C:\Programs\Python34>pythonw lib/idlelib/idle.py
  C:\Programs\Python34>pythonw lib/idlelib/idle.pyw
all do the same thing except that the first two caused a new console python icon to appear on the taskbar.

All three files contain
  import idlelib.PyShell
  idlelib.PyShell.main()
which are equivalent to
  from idlelib.PyShell import main; main()
PyShell can also by run by 'python(w) -m idlelip.PyShell' though I believe that is somewhat 'deprecated'.

idle.py also has a path addition that seems obsolete. I believe the addition was intended for developing idle with installed python and a subrepository consisting of idlelib/*. Subrepositories were possible with svn but are not with hg. In any case, proper development testing ultimately requires testing revised idle with current tkinter.py and compiled _tkinter.c. I suppose that the addition would still work to let someone clone the repository and run the repository Lib/*.py, etc, with installed binaries, but that seems ultimately chancy.

idle.pyw (which is also called by idle.bat) has additions for a scenario that I don't understand: idle.pyw is present and running but apparently not in idlelib, Idle is 'not installed', but PyShell and the rest of idlelib are somewhere on sys.path. There is nothing that I see as pythonw specific. I think this file should be dropped and any needed path manipulations added to idle.py.

A single start up file (idle.py) should first import tkinter (and _tkinter) as in the patch, but in try;except. If the import fails print to stderr if there is one (a console) or use subprocess to start python in a console to display the message. Ditto for creating root. Once that succeeds, I think stderr or the idle process should be replaces  unconditionally. A message box as in the patch is one possibility. An error log window is another. The latter could accumulate all non-fatal error messages to be edited and saved if the user wishes. I think the arg parsing code in PyShell that decides whether to open a Shell or an Editor should be moved to the startup file. Ditto for any other configuration stuff that precedes opening one or the other.
msg341610 - (view) Author: anthony shaw (anthonypjshaw) * (Python triager) Date: 2019-05-06 19:10
Revisiting this issue as it's still open. The original patch was for 3.3a0, there have been many changes to both IDLE and the Windows build since, including the recent IDLE entry point in the new Windows installer.

Steve, Terry, please could you review to check if this issue is still relevant or can be closed off.
msg341643 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-05-06 20:50
Good question.  There are two issues when starting IDLE with pythonw.exe, so that sys.__stderr__ and sys.stderr are initially None.

1. None.write is an attribute error that crashes Python.  Therefore, don't do that. Instead use print.  By default, and when 'file=None' is given directly or by reference, print writes to sys.stdout if it exists or gives up otherwise.

I checked for possible (None).write code and did not find any except where commented out.  These should be deleted or converted to prints.  Possible None files are passed to traceback.print_exception and traceback.print_stack and these both use print, not write.

All but two of the stderr crash issues Roger listed in msg149414 are closed, and the last two either should be or should have a different problem.

Still relevant is

2. Unprinted messages cannot be read by the user.  So display them in a text widget.

Open question 1: backup to print?, default display method?, or both (when possible)?  Ned implies that both would be good on Mac, and maybe
same would be true everywhere.

Open question 2: what patch?  I need to consider Roger's patch in relation to idlelib changes since, including new PseudoFiles, changes to text viewer, and new query module.

The above is about sys in the IDLE GUI process.  The user code process is started with sys.executable, and sys.stdout and sys.stderr are replaced with Pseudofiles that direct output to the idle process shell.  The value of sys.__stdout__ and __stderr__ depend on sys.executable.  If None, prints will go to sys.stdout and hence Shell, and writes raise AttributeError, which also appears in the Shell.  If normal, prints and writes go to the console.  I should check if enough of this is in the docs.

It might be a good idea to wrap both processes in a top-level try-except to attempt to display any unexpected internal error.


I will clean the nosy list as this issue does not involve Steve Dower and many others are long inactive.
History
Date User Action Args
2022-04-11 14:57:24adminsetgithub: 57791
2019-05-06 20:50:56terry.reedysetnosy: - amaury.forgeotdarc, roger.serwy, Todd.Rovito, devplayer, steve.dower

messages: + msg341643
versions: + Python 3.8, - Python 3.6
2019-05-06 19:10:29anthonypjshawsetnosy: + anthonypjshaw, cheryl.sabella, steve.dower
messages: + msg341610
2017-06-30 01:00:45terry.reedysetassignee: terry.reedy
stage: test needed -> patch review
versions: + Python 3.6, Python 3.7, - Python 2.7, Python 3.3, Python 3.4
2014-02-26 21:56:17terry.reedysetmessages: + msg212310
2013-07-07 04:26:50Todd.Rovitosetmessages: + msg192512
2013-07-03 14:02:09terry.reedysetmessages: + msg192238
2013-07-03 07:24:14devplayersetnosy: + devplayer
messages: + msg192223
2013-07-01 05:41:46ggenellinasetnosy: + ggenellina
2013-07-01 04:14:35terry.reedysetmessages: + msg192105
stage: test needed
2013-07-01 03:23:05Todd.Rovitosetmessages: + msg192104
2013-06-30 23:38:25terry.reedysetnosy: + Todd.Rovito
messages: + msg192092
2013-06-15 18:45:31terry.reedysetversions: + Python 3.4, - Python 3.2
2013-05-22 07:05:55serhiy.storchakalinkissue18030 superseder
2012-12-24 10:59:10asvetlovsetnosy: + asvetlov
2012-05-19 04:49:54loewislinkissue14838 superseder
2011-12-16 23:54:17terry.reedysetnosy: + terry.reedy
2011-12-14 21:54:00roger.serwysetmessages: + msg149481
2011-12-14 05:54:29ned.deilysetnosy: + ned.deily
messages: + msg149427
2011-12-14 01:44:43roger.serwysetmessages: + msg149414
2011-12-12 15:46:17roger.serwysetmessages: + msg149317
2011-12-12 15:35:17amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg149316
2011-12-11 19:38:09roger.serwycreate