classification
Title: IDLE: Handle Shell input warnings properly.
Type: behavior Stage: patch review
Components: IDLE Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: cheryl.sabella, miss-islington, rhettinger, serhiy.storchaka, taleinat, terry.reedy
Priority: normal Keywords: patch

Created on 2019-08-11 16:32 by serhiy.storchaka, last changed 2020-05-30 05:09 by terry.reedy.

Pull Requests
URL Status Linked Edit
PR 15311 open terry.reedy, 2019-08-16 05:42
PR 15500 merged terry.reedy, 2019-08-26 04:44
PR 15504 merged miss-islington, 2019-08-26 06:28
PR 15505 merged miss-islington, 2019-08-26 06:28
Messages (17)
msg349396 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-08-11 16:32
DeprecationWarning at compile time is output multiple times to the stderr (on console from which IDLE was ran), but not to IDLE Shell window.

For example, when enter '\e':

$ ./python -m idlelib

Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e
>>> 
Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e
>>> 
Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e

And when close IDLE, additional output is printed in 3.8+:

>>> Exception ignored in: <idlelib.run.PseudoInputFile object at 0x7f9e3b4a15a0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/idlelib/run.py", line 488, in close
  File "/home/serhiy/py/cpython/Lib/idlelib/pyshell.py", line 1019, in close
  File "/home/serhiy/py/cpython/Lib/idlelib/editor.py", line 1058, in close
  File "/home/serhiy/py/cpython/Lib/idlelib/outwin.py", line 94, in maybesave
  File "/home/serhiy/py/cpython/Lib/idlelib/editor.py", line 991, in get_saved
AttributeError: 'NoneType' object has no attribute 'get_saved'
msg349441 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-12 06:58
None.get_saved is #35623.  I believe it is related to #35379, where I requested your comment on what I found so far.
---

1. Compiler warnings for shell input go to sys.__stderr, which only exists when IDLE is started from a console.  This will include SyntaxWarnings when not turned into errors.  pyshell 67+, warning_stream and idle_showwarning.  I have no idea why.  Compiler warnings for editor contents and runtime warnings go to shell.  For now, all should go to Shell. 

2. Warnings are formatted by run 42, idle_formatwarning.  Someone must have decided that "<stdin>:1: DeprecationWarning: invalid escape sequence \e" is newbie unfriendly.  When printed in an editor-derived window, the IDLE format will enable context menu 'goto file/line'.  But 4 lines is a lot and the top line only serves to imitate the 'Traceback ...' line.  I will think about this.  Perhaps

DeprecationWarning: invalid escape sequence \e
  File "<pyshell#1>", line 1
    '\e'

3. Warnings (at least this one) for shell input are printed 3 times.  I do not know why at present.  ModifiedInterpreter.runsource calls code.InteractiveInterpreter(self, source, filename (pyshell 669).  Perhaps the latter is buggy in somehow calling compile 3 times.  I will look another time.
msg349846 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-16 05:50
I am combining the trivial 'leave Shell input SyntaxWarnings alone (instead of making them SyntaxErrors)' part of #34857 with this issue, 'print Shell input (and the very rare internal IDLE Warnings occurring after Shell exists) in Shell, and changing the title accordingly.

The PR is a WIP with at least two bugs. Help wanted.
1. the triple output.  (Before the patch, when I started IDLE from command prompt, I only saw one.
2. Warnings are inserted before the text generating the warning.  I interpret this as intercepted \n causing compilation and printing of the warning before the \n is inserted at the end of the line and the iomark moved to the beginning of the next.  I don't know what moves the iomark.
msg349858 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-16 13:09
Il take a look at this. I've very recently gone through the relevant parts of the code and I should be able to figure it it efficiently.
msg349869 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-16 15:02
The triple (or double, see below) printing seems like a pure bug to be fixed.  A fix could be a separate PR to be merged 'immediately'.

But there is a complication with insertion location.  My bug claim was based on a one line statement, where 'before' looks wrong and 'after', as in REPL, right.

>>> <IDLE>
Warning (from warnings module):  # Printed 3 times, 2 deleted.
  File "<pyshell#0>", line 1
    0 is 0
SyntaxWarning: "is" with a literal. Did you mean "=="?
0 is 0
True

>>> 0 is 0  # REPL
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
True

But now consider a more realistic compound statement.

>>> if 0 is 0:  # REPL
...   print('true')
...
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
true

With IDLE, a warning is printed *before* the user enters a blank line to signal end of statement.

>>> 
Warning (from warnings module):  # Only printed TWICE!.
  File "<pyshell#2>", line 1
    if 0 is 0:
SyntaxWarning: "is" with a literal. Did you mean "=="?
if 0 is 0:
	print('true')
	|< cursor location

A user could edit the condition, add more to the suite, or hit return. In the latter 2 cases, the warnings are repeated.  In this case, the insertion seems like the best that can be done.

Perhaps the msg349845 comment about run.show_warnings collecting warnings for possible delayed display applies to pyshell.show_warnings.  Or the warning could be put in a popup, though I do not especially like the idea of that.  Or it could be logged to a separate Warnings window, and a red "Warning logged' flashed on the status bar.

I suspect that this case is why a fake prompt was previously tacked on to the end of the warning, though I also suspect these complications are why shell input warnings were relegated to a possibly non-existent console (where the fake prompt is nonsense).

Behavior is the same with '\e' and DeprecationWarning.

This example further convinces me that multiline shell input should be isolated from both prompts and intermixed outputs.  I plan to open that issue later today.
msg349884 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-16 20:00
I see none of this on latest master (0567786d26348aa7eaf0ab1b5d038fdabe409d92) on Windows 10.  Can you specify on what OS and Python branch/version this happens?
msg349896 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-17 05:40
My first post described master at the time, when SyntaxWarnings were converted to SyntexErrors and shell input warnings, in particular, Deprecation warnings were sent to the console or nowhere.  My third post describes the situation with the PR, where SyntaxWarnings are left as warnings and shell input warnings go to shell.  An improvement, but still buggy.  I just reconfirmed after updating, compiling Python, and checking out my 'warn' branch.
msg350483 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-26 04:48
I pulled the part of PR 15311 that works, stop turning Shell SyntaxWarnings into SyntaxErrors, into PR-15500, to get it into b4.  I will update the former after merging the latter.
msg350487 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-26 06:27
New changeset 1039f39c9c6edb4c185856c19316d3a4eb561c38 by Terry Jan Reedy in branch 'master':
bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
https://github.com/python/cpython/commit/1039f39c9c6edb4c185856c19316d3a4eb561c38
msg350492 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-26 06:52
New changeset 077887059a5b3d38161dfd74b160c701445a1ef0 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.8':
bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
https://github.com/python/cpython/commit/077887059a5b3d38161dfd74b160c701445a1ef0
msg350494 - (view) Author: miss-islington (miss-islington) Date: 2019-08-26 06:53
New changeset 1b15914ebb46e549ff0965c789ef03b4bac2a890 by Miss Islington (bot) in branch '3.7':
bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
https://github.com/python/cpython/commit/1b15914ebb46e549ff0965c789ef03b4bac2a890
msg369929 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-05-25 22:41
Terry,

I put this into debug and found the reason it's printing the warning three times.  In `codeop.py`, it's running `_maybe_compile` and there are three try statements:

```
def _maybe_compile(compiler, source, filename, symbol):
    # Check for source consisting of only blank lines and comments
    for line in source.split("\n"):
        line = line.strip()
        if line and line[0] != '#':
            break               # Leave it alone
    else:
        if symbol != "eval":
            source = "pass"     # Replace it with a 'pass' statement

    err = err1 = err2 = None
    code = code1 = code2 = None

    try:
        code = compiler(source, filename, symbol)
    except SyntaxError:
        pass

    try:
        code1 = compiler(source + "\n", filename, symbol)
    except SyntaxError as e:
        err1 = e

    try:
        code2 = compiler(source + "\n\n", filename, symbol)
    except SyntaxError as e:
        err2 = e

    try:
        if code:
            return code
        if not code1 and repr(err1) == repr(err2):
            raise err1
    finally:
        err1 = err2 = None
```

It also has this in the module docstring:
```
Compile three times: as is, with \n, and with \n\n appended.  If it
compiles as is, it's complete.  If it compiles with one \n appended,
we expect more.  If it doesn't compile either way, we compare the
error we get when compiling with \n or \n\n appended.  If the errors
are the same, the code is broken.  But if the errors are different, we
expect more.  Not intuitive; not even guaranteed to hold in future
releases; but this matches the compiler's behavior from Python 1.4
through 2.2, at least.
```
msg369934 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-05-26 01:05
I added a commit to the PR to show warnings once.
msg370163 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-28 03:50
codeop._maybe_compile wraps each compile in try ... except SyntaxError.  It can later reraise just once.  I think it a bug that it is not similarly careful about SyntaxWarning and DeprecationWarning to only emit a particular warning just once.

Cheryl, would you like to open a new issue and submit a patch?
msg370212 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-05-28 13:59
Thanks, Terry.  I created issue40807 for the codeop warnings.
msg370357 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-30 05:00
To review: Serhiy reported 3 'IDLE' bugs: tripled DeprecationWarning (now joined by SyntaxWarning), printing to console (if available) instead of Shell, and an exit exception.

1. Codeop generates the tripicates; #40807 will fix that.
2. PR-15311 prints warnings to Shell, but I held off  pending some fix.  3. The exception (a duplicate report) was fixed on #35623.
4. PR-15500 stopped turning a SyntaxWarning into a SyntaxError.

Additional issues, semi-independent.
5. REPL prints warning after code, IDLE before
6. REPL prints 1 line, IDLE 4, but this is related to Shell warnings going to the console.  IDLE should use 1 and highlight.
7. Should IDLE extend scope of deduplication?  Yes with 4 lines, maybe not with 1 line.
8. REPL partially deduplicates within compound statement; see #40807.
msg370358 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-30 05:09
The new items can be new issues, with 6 maybe the highest priority.  Currently
<stdin>:1: DeprecationWarning: invalid escape sequence \e
becomes
Warning (from warnings module):
  File "<pyshell#0>", line 1
    '\e'
DeprecationWarning: invalid escape sequence \e

For both Shell and Editor input, the Warning Line is not really needed.  For Shell, the fake filename is not needed, nor is the line # and line if somehow otherwise marked.
History
Date User Action Args
2020-05-30 05:09:07terry.reedysetmessages: + msg370358
2020-05-30 05:00:15terry.reedysetmessages: + msg370357
versions: - Python 3.10
2020-05-28 13:59:21cheryl.sabellasetmessages: + msg370212
2020-05-28 04:20:34terry.reedysetversions: + Python 3.10
2020-05-28 03:50:16terry.reedysetmessages: + msg370163
versions: - Python 3.10
2020-05-26 20:07:05taleinatsetversions: + Python 3.10
2020-05-26 01:05:14cheryl.sabellasetmessages: + msg369934
2020-05-25 22:41:46cheryl.sabellasetnosy: + cheryl.sabella
messages: + msg369929
2019-08-26 06:53:41miss-islingtonsetnosy: + miss-islington
messages: + msg350494
2019-08-26 06:52:49terry.reedysetmessages: + msg350492
2019-08-26 06:28:14miss-islingtonsetpull_requests: + pull_request15191
2019-08-26 06:28:08miss-islingtonsetstage: patch review
pull_requests: + pull_request15190
2019-08-26 06:27:59terry.reedysetmessages: + msg350487
2019-08-26 04:48:21terry.reedysetmessages: + msg350483
stage: patch review -> (no value)
2019-08-26 04:44:19terry.reedysetstage: patch review
pull_requests: + pull_request15189
2019-08-17 05:40:06terry.reedysetmessages: + msg349896
2019-08-16 20:00:13taleinatsetmessages: + msg349884
2019-08-16 15:02:19terry.reedysetmessages: + msg349869
2019-08-16 13:09:26taleinatsetmessages: + msg349858
2019-08-16 05:50:47terry.reedysetmessages: + msg349846
stage: patch review -> (no value)
2019-08-16 05:42:25terry.reedysetkeywords: + patch
stage: patch review
pull_requests: + pull_request15030
2019-08-16 04:43:40terry.reedysetnosy: + rhettinger, taleinat

title: IDLE: DeprecationWarning not handled properly -> IDLE: Handle Shell input warnings properly.
2019-08-12 06:58:00terry.reedysetmessages: + msg349441
2019-08-11 16:32:12serhiy.storchakacreate