Title: Codeop: Show warnings once during _maybe_compile
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: cheryl.sabella, miss-islington, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2020-05-28 13:55 by cheryl.sabella, last changed 2020-06-09 13:12 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20486 merged cheryl.sabella, 2020-05-28 13:57
PR 20639 merged miss-islington, 2020-06-04 23:40
PR 20640 closed miss-islington, 2020-06-04 23:40
PR 20641 closed miss-islington, 2020-06-04 23:41
PR 20646 merged terry.reedy, 2020-06-05 01:58
PR 20647 merged miss-islington, 2020-06-05 02:43
PR 20672 merged miss-islington, 2020-06-06 08:06
PR 20673 merged miss-islington, 2020-06-06 08:06
Messages (10)
msg370208 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2020-05-28 13:55
When calling `codeop._maybe_compile`, `compile` is run three times.  If the code being compiled causes a warning message, the warning is generated each time that `compile` is called, thus (possibly) showing the message three times.

See msg370163 and that issue for context.
msg370722 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-04 19:05
The purpose of code and codeop is to imitate interactive python in python.  Calling compile() thrice is an implementation detail.  Exposing that is a bug.  On #37824, Serhiy reported 3 'IDLE' bugs for 3.7 to 3.9: tripled DeprecationWarning (now joined by SyntaxWarning), printing to console (if available) instead of Shell, and an exit exception.  This issue is continuation of the triplicate issue, given that the bug is not in IDLE.  As a bug, I think it should still be backported.

I looked more carefully at the behavior of the standard REPL with different statement inputs.  Before executing, if the statement is complete, it may raise a compile exception, usually SyntaxError, or emit one or more a warnings, once for each instance.  For multiline statements, DeprecationWarnings are emitted immediately after the line with the deprecated code while SyntaxWarnings are emitted, if there is no syntax error, just before execution.  SyntaxWarnings for a given slice of code are not repeated.  However, at least some of these details have to be enforced in the UI code that repeatedly calls compile or _maybe_compile as lines are added, not in compile, and should not be in _maybe_compile.  Both have no memory or previous calls.  (I don't think that IDLE necessarily *must* do the same either.)

Looking at compile itself, it appears that warnings are not emitted if there is an exception.  But I am puzzled at this in 3.9 and master.

>>> if '\e' is 1:
<stdin>:1: DeprecationWarning: invalid escape sequence \e
>>> compile('if '\e' is 1:\n', '', 'single')
  File "<stdin>", line 1
SyntaxError: unexpected character after line continuation character

Something must cause "'\e'" to be a warning instead of an error.

This is enough to experiment with your patch applied.
msg370724 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-04 19:14
I am not sure if 3 compiles are always needed.  At one time, compile required a final '\n' to not raise, but that is no longer true.  And as near as I can tell, 'code without final newline\n' and 'code without final newline\n\n' always compile the same.  Certainly, Shell does not need to compile without a final newline.  But we can look at this in a separate issue.  If IDLE's requirement is different from code.II's, I believe it could pass its own compile function.
msg370738 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-04 23:40
New changeset 052d3fc0907be253cfd64b2c737a0b0aca586011 by Cheryl Sabella in branch 'master':
bpo-40807: Show warnings once from codeop._maybe_compile (#20486)
msg370739 - (view) Author: miss-islington (miss-islington) Date: 2020-06-04 23:59
New changeset 3dfe549bfa08a099b88efd030669359199f31064 by Miss Islington (bot) in branch '3.9':
bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)
msg370748 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-05 02:42
New changeset a5d6aba318ead9cc756ba750a70da41f5def3f8f by Terry Jan Reedy in branch '3.8':
[3.8] bpo-40807: Backport test_codeop change [GH-19670]
msg370750 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-05 03:00
New changeset 12d3061c7819a73d891dcce44327410eaf0e1bc2 by Miss Islington (bot) in branch '3.7':
[3.8] bpo-40807: Backport test_codeop change [GH-19670]
msg370751 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-05 03:01
This is a spinoff of #37824.

My above comments about compile are wrong.  The example nests '' within ''.  And, with freshly compiled master, I found an example where Deprecation warning, but not SyntaxWarning, accompanies SyntaxError, and which requires trailing \n.

>>> compile("if'\e' is 1: 1", '', 'single')
SyntaxError: unexpected EOF while parsing
>>> compile("if'\e' is 1: 1\n", '', 'single')
<code object ...>

>>> if'\e' is 1: 1
<stdin>:1: DeprecationWarning: invalid escape sequence \e
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?

In IDLE, DeprecationWarning is repeated, but that is another issue.

The secondary prompts seems like a bug,  The statement is complete and entering anything but a comment is a SyntaxError.  (A comment results in another ... prompt.)
msg370814 - (view) Author: miss-islington (miss-islington) Date: 2020-06-06 08:24
New changeset c067183605cf84bb1a246635f52827251d0476f8 by Miss Islington (bot) in branch '3.8':
bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)
msg371102 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-09 13:12
The 3.7 backport is needed but required a revised example.  The changeset is 4b378acb97a575892c0e372a6bb0c17da1ccdf3e.

PRs 20486, 20639, 20672, and 20673 are the main patch.  PRs 20646 and 20647 for 3.8 and 3.7 backport a test_codeop import change made elsewhere and not backported.  The alternative would have been changing our backport to the older naming convention.
Date User Action Args
2020-06-09 13:12:26terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg371102

stage: patch review -> resolved
2020-06-06 08:24:48miss-islingtonsetmessages: + msg370814
2020-06-06 08:06:50miss-islingtonsetpull_requests: + pull_request19889
2020-06-06 08:06:43miss-islingtonsetpull_requests: + pull_request19888
2020-06-05 03:01:17terry.reedysetmessages: + msg370751
2020-06-05 03:00:58terry.reedysetmessages: + msg370750
2020-06-05 02:43:05miss-islingtonsetpull_requests: + pull_request19867
2020-06-05 02:42:51terry.reedysetmessages: + msg370748
2020-06-05 01:58:29terry.reedysetpull_requests: + pull_request19866
2020-06-04 23:59:51miss-islingtonsetmessages: + msg370739
2020-06-04 23:41:00miss-islingtonsetpull_requests: + pull_request19861
2020-06-04 23:40:51miss-islingtonsetpull_requests: + pull_request19860
2020-06-04 23:40:43miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19859
2020-06-04 23:40:28terry.reedysetmessages: + msg370738
2020-06-04 19:14:19terry.reedysetmessages: + msg370724
2020-06-04 19:05:12terry.reedysetnosy: + terry.reedy, serhiy.storchaka
title: CODEOP: Show warnings once during _maybe_compile -> Codeop: Show warnings once during _maybe_compile
messages: + msg370722

versions: + Python 3.7, Python 3.8
type: enhancement -> behavior
2020-05-28 13:57:54cheryl.sabellasetkeywords: + patch
stage: patch review
pull_requests: + pull_request19735
2020-05-28 13:55:08cheryl.sabellacreate