Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IDLE: Handle Shell input SyntaxWarning & DeprecationWarning #82005

Open
serhiy-storchaka opened this issue Aug 11, 2019 · 19 comments
Open

IDLE: Handle Shell input SyntaxWarning & DeprecationWarning #82005

serhiy-storchaka opened this issue Aug 11, 2019 · 19 comments
Assignees
Labels
topic-IDLE type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

BPO 37824
Nosy @rhettinger, @terryjreedy, @taleinat, @serhiy-storchaka, @csabella, @miss-islington
PRs
  • gh-82005: Properly handle user input warnings in IDLE shell. #15311
  • bpo-37824: Properly handle user input warnings in IDLE shell. #15500
  • [3.8] bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500) #15504
  • [3.7] bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500) #15505
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/terryjreedy'
    closed_at = None
    created_at = <Date 2019-08-11.16:32:12.488>
    labels = ['3.8', 'expert-IDLE', 'type-bug', '3.7', '3.9']
    title = 'IDLE: Handle Shell input SyntaxWarning & DeprecationWarning'
    updated_at = <Date 2021-11-12.19:21:29.683>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2021-11-12.19:21:29.683>
    actor = 'terry.reedy'
    assignee = 'terry.reedy'
    closed = False
    closed_date = None
    closer = None
    components = ['IDLE']
    creation = <Date 2019-08-11.16:32:12.488>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 37824
    keywords = ['patch']
    message_count = 19.0
    messages = ['349396', '349441', '349846', '349858', '349869', '349884', '349896', '350483', '350487', '350492', '350494', '369929', '369934', '370163', '370212', '370357', '370358', '386622', '406234']
    nosy_count = 6.0
    nosy_names = ['rhettinger', 'terry.reedy', 'taleinat', 'serhiy.storchaka', 'cheryl.sabella', 'miss-islington']
    pr_nums = ['15311', '15500', '15504', '15505']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue37824'
    versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    @serhiy-storchaka
    Copy link
    Member Author

    DeprecationWarning at compile time is output multiple times to the stderr (on console from which IDLE was ran), but not to IDLE Shell window.

    For example, when enter '\e':

    $ ./python -m idlelib
    Warning (from warnings module):
      File "<pyshell#0>", line 1
        '\e'
    DeprecationWarning: invalid escape sequence \e
    >>> 
    Warning (from warnings module):
      File "<pyshell#0>", line 1
        '\e'
    DeprecationWarning: invalid escape sequence \e
    >>> 
    Warning (from warnings module):
      File "<pyshell#0>", line 1
        '\e'
    DeprecationWarning: invalid escape sequence \e

    And when close IDLE, additional output is printed in 3.8+:

    >>> Exception ignored in: <idlelib.run.PseudoInputFile object at 0x7f9e3b4a15a0>
    Traceback (most recent call last):
      File "/home/serhiy/py/cpython/Lib/idlelib/run.py", line 488, in close
      File "/home/serhiy/py/cpython/Lib/idlelib/pyshell.py", line 1019, in close
      File "/home/serhiy/py/cpython/Lib/idlelib/editor.py", line 1058, in close
      File "/home/serhiy/py/cpython/Lib/idlelib/outwin.py", line 94, in maybesave
      File "/home/serhiy/py/cpython/Lib/idlelib/editor.py", line 991, in get_saved
    AttributeError: 'NoneType' object has no attribute 'get_saved'

    @serhiy-storchaka serhiy-storchaka added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes labels Aug 11, 2019
    @serhiy-storchaka serhiy-storchaka added topic-IDLE type-bug An unexpected behavior, bug, or error labels Aug 11, 2019
    @terryjreedy
    Copy link
    Member

    None.get_saved is bpo-35623. I believe it is related to bpo-35379, where I requested your comment on what I found so far.
    ---

    1. Compiler warnings for shell input go to sys.__stderr, which only exists when IDLE is started from a console. This will include SyntaxWarnings when not turned into errors. pyshell 67+, warning_stream and idle_showwarning. I have no idea why. Compiler warnings for editor contents and runtime warnings go to shell. For now, all should go to Shell.

    2. Warnings are formatted by run 42, idle_formatwarning. Someone must have decided that "<stdin>:1: DeprecationWarning: invalid escape sequence \e" is newbie unfriendly. When printed in an editor-derived window, the IDLE format will enable context menu 'goto file/line'. But 4 lines is a lot and the top line only serves to imitate the 'Traceback ...' line. I will think about this. Perhaps

    DeprecationWarning: invalid escape sequence \e
    File "<pyshell#1>", line 1
    '\e'

    1. Warnings (at least this one) for shell input are printed 3 times. I do not know why at present. ModifiedInterpreter.runsource calls code.InteractiveInterpreter(self, source, filename (pyshell 669). Perhaps the latter is buggy in somehow calling compile 3 times. I will look another time.

    @terryjreedy terryjreedy changed the title IDLE: DeprecationWarning not handled properly IDLE: Handle Shell input warnings properly. Aug 16, 2019
    @terryjreedy
    Copy link
    Member

    I am combining the trivial 'leave Shell input SyntaxWarnings alone (instead of making them SyntaxErrors)' part of bpo-34857 with this issue, 'print Shell input (and the very rare internal IDLE Warnings occurring after Shell exists) in Shell, and changing the title accordingly.

    The PR is a WIP with at least two bugs. Help wanted.

    1. the triple output. (Before the patch, when I started IDLE from command prompt, I only saw one.
    2. Warnings are inserted before the text generating the warning. I interpret this as intercepted \n causing compilation and printing of the warning before the \n is inserted at the end of the line and the iomark moved to the beginning of the next. I don't know what moves the iomark.

    @taleinat
    Copy link
    Contributor

    Il take a look at this. I've very recently gone through the relevant parts of the code and I should be able to figure it it efficiently.

    @terryjreedy
    Copy link
    Member

    The triple (or double, see below) printing seems like a pure bug to be fixed. A fix could be a separate PR to be merged 'immediately'.

    But there is a complication with insertion location. My bug claim was based on a one line statement, where 'before' looks wrong and 'after', as in REPL, right.

    >>> <IDLE>
    Warning (from warnings module):  # Printed 3 times, 2 deleted.
      File "<pyshell#0>", line 1
        0 is 0
    SyntaxWarning: "is" with a literal. Did you mean "=="?
    0 is 0
    True
    
    >>> 0 is 0  # REPL
    <stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
    True

    But now consider a more realistic compound statement.

    >>> if 0 is 0:  # REPL
    ...   print('true')
    ...
    <stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
    true

    With IDLE, a warning is printed *before* the user enters a blank line to signal end of statement.

    >>> 
    Warning (from warnings module):  # Only printed TWICE!.
      File "<pyshell#2>", line 1
        if 0 is 0:
    SyntaxWarning: "is" with a literal. Did you mean "=="?
    if 0 is 0:
    	print('true')
    	|< cursor location

    A user could edit the condition, add more to the suite, or hit return. In the latter 2 cases, the warnings are repeated. In this case, the insertion seems like the best that can be done.

    Perhaps the msg349845 comment about run.show_warnings collecting warnings for possible delayed display applies to pyshell.show_warnings. Or the warning could be put in a popup, though I do not especially like the idea of that. Or it could be logged to a separate Warnings window, and a red "Warning logged' flashed on the status bar.

    I suspect that this case is why a fake prompt was previously tacked on to the end of the warning, though I also suspect these complications are why shell input warnings were relegated to a possibly non-existent console (where the fake prompt is nonsense).

    Behavior is the same with '\e' and DeprecationWarning.

    This example further convinces me that multiline shell input should be isolated from both prompts and intermixed outputs. I plan to open that issue later today.

    @taleinat
    Copy link
    Contributor

    I see none of this on latest master (0567786) on Windows 10. Can you specify on what OS and Python branch/version this happens?

    @terryjreedy
    Copy link
    Member

    My first post described master at the time, when SyntaxWarnings were converted to SyntexErrors and shell input warnings, in particular, Deprecation warnings were sent to the console or nowhere. My third post describes the situation with the PR, where SyntaxWarnings are left as warnings and shell input warnings go to shell. An improvement, but still buggy. I just reconfirmed after updating, compiling Python, and checking out my 'warn' branch.

    @terryjreedy
    Copy link
    Member

    I pulled the part of PR 15311 that works, stop turning Shell SyntaxWarnings into SyntaxErrors, into PR-15500, to get it into b4. I will update the former after merging the latter.

    @terryjreedy
    Copy link
    Member

    New changeset 1039f39 by Terry Jan Reedy in branch 'master':
    bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
    1039f39

    @terryjreedy
    Copy link
    Member

    New changeset 0778870 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.8':
    bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
    0778870

    @miss-islington
    Copy link
    Contributor

    New changeset 1b15914 by Miss Islington (bot) in branch '3.7':
    bpo-37824: Properly handle user input warnings in IDLE shell. (GH-15500)
    1b15914

    @csabella
    Copy link
    Contributor

    Terry,

    I put this into debug and found the reason it's printing the warning three times. In codeop.py, it's running _maybe_compile and there are three try statements:

    def _maybe_compile(compiler, source, filename, symbol):
        # Check for source consisting of only blank lines and comments
        for line in source.split("\n"):
            line = line.strip()
            if line and line[0] != '#':
                break               # Leave it alone
        else:
            if symbol != "eval":
                source = "pass"     # Replace it with a 'pass' statement
    
        err = err1 = err2 = None
        code = code1 = code2 = None
    
        try:
            code = compiler(source, filename, symbol)
        except SyntaxError:
            pass
    
        try:
            code1 = compiler(source + "\n", filename, symbol)
        except SyntaxError as e:
            err1 = e
    
        try:
            code2 = compiler(source + "\n\n", filename, symbol)
        except SyntaxError as e:
            err2 = e
    
        try:
            if code:
                return code
            if not code1 and repr(err1) == repr(err2):
                raise err1
        finally:
            err1 = err2 = None
    

    It also has this in the module docstring:

    Compile three times: as is, with \n, and with \n\n appended.  If it
    compiles as is, it's complete.  If it compiles with one \n appended,
    we expect more.  If it doesn't compile either way, we compare the
    error we get when compiling with \n or \n\n appended.  If the errors
    are the same, the code is broken.  But if the errors are different, we
    expect more.  Not intuitive; not even guaranteed to hold in future
    releases; but this matches the compiler's behavior from Python 1.4
    through 2.2, at least.
    

    @csabella
    Copy link
    Contributor

    I added a commit to the PR to show warnings once.

    @taleinat taleinat added the 3.10 only security fixes label May 26, 2020
    @terryjreedy
    Copy link
    Member

    codeop._maybe_compile wraps each compile in try ... except SyntaxError. It can later reraise just once. I think it a bug that it is not similarly careful about SyntaxWarning and DeprecationWarning to only emit a particular warning just once.

    Cheryl, would you like to open a new issue and submit a patch?

    @terryjreedy terryjreedy added 3.10 only security fixes and removed 3.10 only security fixes labels May 28, 2020
    @csabella
    Copy link
    Contributor

    Thanks, Terry. I created bpo-40807 for the codeop warnings.

    @terryjreedy
    Copy link
    Member

    To review: Serhiy reported 3 'IDLE' bugs: tripled DeprecationWarning (now joined by SyntaxWarning), printing to console (if available) instead of Shell, and an exit exception.

    1. Codeop generates the tripicates; bpo-40807 will fix that.
    2. PR-15311 prints warnings to Shell, but I held off pending some fix. 3. The exception (a duplicate report) was fixed on bpo-35623.
    3. PR-15500 stopped turning a SyntaxWarning into a SyntaxError.

    Additional issues, semi-independent.
    5. REPL prints warning after code, IDLE before
    6. REPL prints 1 line, IDLE 4, but this is related to Shell warnings going to the console. IDLE should use 1 and highlight.
    7. Should IDLE extend scope of deduplication? Yes with 4 lines, maybe not with 1 line.
    8. REPL partially deduplicates within compound statement; see bpo-40807.

    @terryjreedy terryjreedy removed the 3.10 only security fixes label May 30, 2020
    @terryjreedy
    Copy link
    Member

    The new items can be new issues, with 6 maybe the highest priority. Currently
    <stdin>:1: DeprecationWarning: invalid escape sequence \e
    becomes
    Warning (from warnings module):
    File "<pyshell#0>", line 1
    '\e'
    DeprecationWarning: invalid escape sequence \e

    For both Shell and Editor input, the Warning Line is not really needed. For Shell, the fake filename is not needed, nor is the line # and line if somehow otherwise marked.

    @terryjreedy
    Copy link
    Member

    bpo-43151 asked about 'is <literal>' syntax warnings in REPL. Discussion noted that IDLE Shell does note print it for "x is 'a'" (known here) and that for "if x is 'a': print('executed')" there is a syntax error and no execution. The conversion to SyntaxError was supposedly fixed by PR-15500, but it appears that the patch for bpo-40807 introduced it into codeop._maybe_compile.

    msg386621 has summary and discussion. I need to add debug prints or breakpoints into _maybe_compile to understand it better, and maybe improve it.

    Before printing syntax warnings in Shell, I want to stop expanding them from 1 to 4 lines. We could just not import the replacement from run.py, but I think the default format could be improved even with 1 or 2 lines.

    @terryjreedy terryjreedy changed the title IDLE: Handle Shell input warnings properly. IDLE: Handle Shell input SyntaxWarning & DeprecationWarning Feb 8, 2021
    @terryjreedy
    Copy link
    Member

    Might want to special case running certain console commands in Shell, such as 'pip install'.
    https://stackoverflow.com/questions/69936921/packages-not-installing-in-idle-shell
    SyntaxError: invalid syntax. Perhaps you forgot a comma?

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    topic-IDLE type-bug An unexpected behavior, bug, or error
    Projects
    Status: In Progress
    Development

    No branches or pull requests

    5 participants