classification
Title: IDLE needs syntax highlighting for async and await
Type: enhancement Stage: resolved
Components: IDLE Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: David E. Franco G., martin.panter, miss-islington, terry.reedy, yselivanov
Priority: normal Keywords: easy, patch

Created on 2017-03-03 03:43 by David E. Franco G., last changed 2018-05-16 01:38 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
no syntax highlighting.png David E. Franco G., 2017-03-03 03:43 sample screenshot of the current behavior
colorize_async.diff terry.reedy, 2017-03-03 07:23 review
Pull Requests
URL Status Linked Edit
PR 6846 merged terry.reedy, 2018-05-15 02:42
PR 6867 merged miss-islington, 2018-05-15 18:20
PR 6868 merged miss-islington, 2018-05-15 18:21
PR 6879 merged terry.reedy, 2018-05-15 21:11
Messages (15)
msg288849 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 03:43
Well, this is pretty self explanatory, when playing with this new features of async and await (https://docs.python.org/3.5/whatsnew/3.5.html#new-features) I found to me surprise that there is no syntax highlighting for it in the IDLE for py3.5 and also for py3.6

So I humbly ask for its addition. Thanks
msg288859 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 06:19
I presume you would like 'async' and 'await' highlighted as keywords.  However, IDLE takes its definition of 'keyword' from keyword.kwlist.  'Async' and 'await' are currently not on that list as they are not yet keywords.

>>> async = 1
>>> await = 2

According to https://www.python.org/dev/peps/pep-0492/#transition-plan, the intention was (is?) to make them keywords in 3.7.  As of Feb 11, that had not happened yet.  If and when it does, this issue will be taken care of.

I may consider adding a special context sensitive case for 3.6, if the colorizer code makes it easily possible.  Care is needed since it would be a mistake to mark them as keywords in the above statements.

The difficulty is that colorizer uses regexes and a bit of context sensitive code, while python is using a full grammar parse.  I believe that either at the beginning of a line and 'async' followed by 'for' or 'with' or 'await' not followed by certain punctuation ('.', ',', or '=') should be treated as a keyword.

Yury, does the rule above look about right?  'Await' seems trickier than 'async'.

I think it may be possible to add regexes that are not literal words to the kwlist.  If so, adding 'async +def ', 'async +for ', 'async +with ', and 'await +[^.,=]' and not worrying about 'beginning or line' or tabs  (instead of spaces) between would be easy and probably good enough.
msg288860 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-03 06:22
>According to https://www.python.org/dev/peps/pep-0492/#transition-plan, the intention was (is?) to make them keywords in 3.7.  As of Feb 11, that had not happened yet.  If and when it does, this issue will be taken care of.

I plan to do that in the next couple of weeks.

> I may consider adding a special context sensitive case for 3.6, if the colorizer code makes it easily possible.  Care is needed since it would be a mistake to mark them as keywords in the above statements.

If possible I would make 'async' and 'await' to be always highlighted. Even if a user writes <3.7 code and uses 'async' as a variable, it would still be great to let the user know that there is something wrong with this variable.
msg288868 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 07:23
Always coloring 'async' and 'await' is trivial (augment expression and add test line) and acceptable to me.

I am a git neophyte and am waiting for workflow development to settle down before working on new setup for git.  If you want to submit the change for 3.6.1, go ahead. (But leave issue open.)  Otherwise, I will get it into 3.6.2 (and 3.7).  Patch is against 3.6 idlelib.colorize, but 3.7 is identical.

The test is to run colorizer.py as main module (python -m idlelib.colorize, or F5 in IDLE editor).  Click 'Test' button and check that 'async' and 'await' have same color as other keywords.  This cannot affect anything in test suite; colorizer is only used by turtledemo, which is not tested, and which will not notice anyway.
msg288879 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-03-03 11:32
See also Issue 26264 about the “keyword” module
msg288902 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 16:25
>I presume you would like 'async' and 'await' highlighted as keywords.

Yes. 
Some others tools like IPython and the Spider IDE that come with the Anaconda package already highlighted them as keywords, even if you can use them as normal variables.

>The test is to run colorizer.py as main module (python -m >idlelib.colorize, or F5 in IDLE editor).  Click 'Test' button and check >that 'async' and 'await' have same color as other keywords.  This >cannot affect anything in test suite; colorizer is only used by >turtledemo, which is not tested, and which will not notice anyway.

super, what I have to do to put the patch so I can test it?
msg288920 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 20:20
David, the easiest thing to do would be to copy and paste the following
 + ['async', 'await']
into this line of colorizer.py (ColorDelegator.py in 3.5)
    kw = r"\b" + any("KEYWORD", keyword.kwlist) + r"\b"
about line 17, to get
    kw = r"\b" + any("KEYWORD", keyword.kwlist + ['async', 'await']) + r"\b"
msg288935 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 21:44
Ok, Done.
that work in both 3.5 and 3.6 
I also did the 

    python -m idlelib.ColorDelegator

and 

    python -m idlelib.colorizer

and they turn ok

I also open those modules in their respective idle and run them, ColorDelegator work ok but colorizer throw me this error

    Traceback (most recent call last):
      File "C:\Anaconda3\Lib\idlelib\colorizer.py", line 279, in <module>
        verbosity=2, exit=False)
      File "C:\Anaconda3\lib\unittest\main.py", line 63, in __init__
        self.module = __import__(module)
      File "C:\Anaconda3\lib\idlelib\idle_test\test_colorizer.py", line 8, in <module>
        from test.support import requires
    ImportError: bad magic number in 'test': b'\x03\xf3\r\n'
    >>>
msg288940 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 22:32
If 'import test' fails that way, then there is a problem either with Anaconda or your installation.  You might try deleting .../Lib/test/__pycache__/__init__*.pyc.
msg288945 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 23:34
I found the problem, it was a test.pyc file that was in my personal folder... once deleted it work perfect

the build-in test module should have some other name...
msg316613 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-05-15 02:48
PR 6846 adds the htest line to all versions (no news).  When merged, I will augment the keyword list and add a news blurb for 3.6.
msg316690 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-05-15 18:20
New changeset 389a48ede92bf7965889d554d2cd17b50d6e3d86 by Terry Jan Reedy in branch 'master':
bpo-29706: Test that IDLE colors async/await as keywords. (GH-6846)
https://github.com/python/cpython/commit/389a48ede92bf7965889d554d2cd17b50d6e3d86
msg316711 - (view) Author: miss-islington (miss-islington) Date: 2018-05-15 19:57
New changeset 8717cfeb6b8bebdfe13df0e9268ddd252ab5ecad by Miss Islington (bot) in branch '3.7':
bpo-29706: Test that IDLE colors async/await as keywords. (GH-6846)
https://github.com/python/cpython/commit/8717cfeb6b8bebdfe13df0e9268ddd252ab5ecad
msg316722 - (view) Author: miss-islington (miss-islington) Date: 2018-05-15 20:48
New changeset 7c59a33491b0bde639a9382ef1de2423207a8cc7 by Miss Islington (bot) in branch '3.6':
bpo-29706: Test that IDLE colors async/await as keywords. (GH-6846)
https://github.com/python/cpython/commit/7c59a33491b0bde639a9382ef1de2423207a8cc7
msg316746 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-05-16 00:10
New changeset 1b0d65fa102fae087254009f6d9903b1d4257d78 by Terry Jan Reedy in branch '3.6':
[3.6] bpo-29706: IDLE now colors async and await as keywords in 3.6. (#6879)
https://github.com/python/cpython/commit/1b0d65fa102fae087254009f6d9903b1d4257d78
History
Date User Action Args
2018-05-16 01:38:52terry.reedysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-05-16 00:10:50terry.reedysetmessages: + msg316746
2018-05-15 21:11:10terry.reedysetpull_requests: + pull_request6551
2018-05-15 20:48:16miss-islingtonsetmessages: + msg316722
2018-05-15 19:57:15miss-islingtonsetnosy: + miss-islington
messages: + msg316711
2018-05-15 18:21:48miss-islingtonsetpull_requests: + pull_request6542
2018-05-15 18:20:50miss-islingtonsetstage: needs patch -> patch review
pull_requests: + pull_request6541
2018-05-15 18:20:41terry.reedysetmessages: + msg316690
2018-05-15 02:48:32terry.reedysetstage: patch review -> needs patch
messages: + msg316613
versions: - Python 3.7
2018-05-15 02:42:29terry.reedysetstage: test needed -> patch review
pull_requests: + pull_request6526
2017-03-03 23:34:18David E. Franco G.setmessages: + msg288945
2017-03-03 22:32:49terry.reedysetmessages: + msg288940
2017-03-03 21:44:26David E. Franco G.setmessages: + msg288935
2017-03-03 20:20:36terry.reedysetmessages: + msg288920
2017-03-03 16:25:32David E. Franco G.setmessages: + msg288902
2017-03-03 11:32:31martin.pantersetnosy: + martin.panter
messages: + msg288879
2017-03-03 07:23:12terry.reedysetfiles: + colorize_async.diff
keywords: + patch
messages: + msg288868
2017-03-03 07:07:11serhiy.storchakasetkeywords: + easy
2017-03-03 06:22:28yselivanovsetmessages: + msg288860
2017-03-03 06:19:43terry.reedysetversions: + Python 3.7, - Python 3.5
nosy: + yselivanov

messages: + msg288859

type: behavior -> enhancement
stage: test needed
2017-03-03 03:43:57David E. Franco G.create