classification
Title: IDLE needs syntax highlighting for async and await
Type: enhancement Stage: test needed
Components: IDLE Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: David E. Franco G., martin.panter, terry.reedy, yselivanov
Priority: normal Keywords: easy, patch

Created on 2017-03-03 03:43 by David E. Franco G., last changed 2017-03-03 23:34 by David E. Franco G..

Files
File name Uploaded Description Edit
no syntax highlighting.png David E. Franco G., 2017-03-03 03:43 sample screenshot of the current behavior
colorize_async.diff terry.reedy, 2017-03-03 07:23 review
Messages (10)
msg288849 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 03:43
Well, this is pretty self explanatory, when playing with this new features of async and await (https://docs.python.org/3.5/whatsnew/3.5.html#new-features) I found to me surprise that there is no syntax highlighting for it in the IDLE for py3.5 and also for py3.6

So I humbly ask for its addition. Thanks
msg288859 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 06:19
I presume you would like 'async' and 'await' highlighted as keywords.  However, IDLE takes its definition of 'keyword' from keyword.kwlist.  'Async' and 'await' are currently not on that list as they are not yet keywords.

>>> async = 1
>>> await = 2

According to https://www.python.org/dev/peps/pep-0492/#transition-plan, the intention was (is?) to make them keywords in 3.7.  As of Feb 11, that had not happened yet.  If and when it does, this issue will be taken care of.

I may consider adding a special context sensitive case for 3.6, if the colorizer code makes it easily possible.  Care is needed since it would be a mistake to mark them as keywords in the above statements.

The difficulty is that colorizer uses regexes and a bit of context sensitive code, while python is using a full grammar parse.  I believe that either at the beginning of a line and 'async' followed by 'for' or 'with' or 'await' not followed by certain punctuation ('.', ',', or '=') should be treated as a keyword.

Yury, does the rule above look about right?  'Await' seems trickier than 'async'.

I think it may be possible to add regexes that are not literal words to the kwlist.  If so, adding 'async +def ', 'async +for ', 'async +with ', and 'await +[^.,=]' and not worrying about 'beginning or line' or tabs  (instead of spaces) between would be easy and probably good enough.
msg288860 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-03 06:22
>According to https://www.python.org/dev/peps/pep-0492/#transition-plan, the intention was (is?) to make them keywords in 3.7.  As of Feb 11, that had not happened yet.  If and when it does, this issue will be taken care of.

I plan to do that in the next couple of weeks.

> I may consider adding a special context sensitive case for 3.6, if the colorizer code makes it easily possible.  Care is needed since it would be a mistake to mark them as keywords in the above statements.

If possible I would make 'async' and 'await' to be always highlighted. Even if a user writes <3.7 code and uses 'async' as a variable, it would still be great to let the user know that there is something wrong with this variable.
msg288868 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 07:23
Always coloring 'async' and 'await' is trivial (augment expression and add test line) and acceptable to me.

I am a git neophyte and am waiting for workflow development to settle down before working on new setup for git.  If you want to submit the change for 3.6.1, go ahead. (But leave issue open.)  Otherwise, I will get it into 3.6.2 (and 3.7).  Patch is against 3.6 idlelib.colorize, but 3.7 is identical.

The test is to run colorizer.py as main module (python -m idlelib.colorize, or F5 in IDLE editor).  Click 'Test' button and check that 'async' and 'await' have same color as other keywords.  This cannot affect anything in test suite; colorizer is only used by turtledemo, which is not tested, and which will not notice anyway.
msg288879 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-03-03 11:32
See also Issue 26264 about the “keyword” module
msg288902 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 16:25
>I presume you would like 'async' and 'await' highlighted as keywords.

Yes. 
Some others tools like IPython and the Spider IDE that come with the Anaconda package already highlighted them as keywords, even if you can use them as normal variables.

>The test is to run colorizer.py as main module (python -m >idlelib.colorize, or F5 in IDLE editor).  Click 'Test' button and check >that 'async' and 'await' have same color as other keywords.  This >cannot affect anything in test suite; colorizer is only used by >turtledemo, which is not tested, and which will not notice anyway.

super, what I have to do to put the patch so I can test it?
msg288920 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 20:20
David, the easiest thing to do would be to copy and paste the following
 + ['async', 'await']
into this line of colorizer.py (ColorDelegator.py in 3.5)
    kw = r"\b" + any("KEYWORD", keyword.kwlist) + r"\b"
about line 17, to get
    kw = r"\b" + any("KEYWORD", keyword.kwlist + ['async', 'await']) + r"\b"
msg288935 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 21:44
Ok, Done.
that work in both 3.5 and 3.6 
I also did the 

    python -m idlelib.ColorDelegator

and 

    python -m idlelib.colorizer

and they turn ok

I also open those modules in their respective idle and run them, ColorDelegator work ok but colorizer throw me this error

    Traceback (most recent call last):
      File "C:\Anaconda3\Lib\idlelib\colorizer.py", line 279, in <module>
        verbosity=2, exit=False)
      File "C:\Anaconda3\lib\unittest\main.py", line 63, in __init__
        self.module = __import__(module)
      File "C:\Anaconda3\lib\idlelib\idle_test\test_colorizer.py", line 8, in <module>
        from test.support import requires
    ImportError: bad magic number in 'test': b'\x03\xf3\r\n'
    >>>
msg288940 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-03-03 22:32
If 'import test' fails that way, then there is a problem either with Anaconda or your installation.  You might try deleting .../Lib/test/__pycache__/__init__*.pyc.
msg288945 - (view) Author: David E. Franco G. (David E. Franco G.) Date: 2017-03-03 23:34
I found the problem, it was a test.pyc file that was in my personal folder... once deleted it work perfect

the build-in test module should have some other name...
History
Date User Action Args
2017-03-03 23:34:18David E. Franco G.setmessages: + msg288945
2017-03-03 22:32:49terry.reedysetmessages: + msg288940
2017-03-03 21:44:26David E. Franco G.setmessages: + msg288935
2017-03-03 20:20:36terry.reedysetmessages: + msg288920
2017-03-03 16:25:32David E. Franco G.setmessages: + msg288902
2017-03-03 11:32:31martin.pantersetnosy: + martin.panter
messages: + msg288879
2017-03-03 07:23:12terry.reedysetfiles: + colorize_async.diff
keywords: + patch
messages: + msg288868
2017-03-03 07:07:11serhiy.storchakasetkeywords: + easy
2017-03-03 06:22:28yselivanovsetmessages: + msg288860
2017-03-03 06:19:43terry.reedysetversions: + Python 3.7, - Python 3.5
nosy: + yselivanov

messages: + msg288859

type: behavior -> enhancement
stage: test needed
2017-03-03 03:43:57David E. Franco G.create