Author terry.reedy
Recipients epaine, kj, taleinat, terry.reedy
Date 2021-05-02.18:05:44
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1619978745.1.0.762382476495.issue44010@roundup.psfhosted.org>
In-reply-to
Content
Soft keywords are a huge nuisance for syntax highlighting as they need special case regexes and tests.

Hard keywords are matched against complete words, regardless of whether the context is syntactically valid or not.  If 'for' and 'else' were the only keywords, the keyword part of the IDLE colorizer regex would be as follows.

>>> kw = r"\b" + colorizer.any("KEYWORD", ['for', 'else']) + r"\b"
>>> kw
'\\b(?P<KEYWORD>for|else)\\b'

Both words in 'for.else' are highlighted as the tokenizer will see them as keywords.  The parser will later see the combination as an error.

The tag name in a "(?P<name>...) construct can only be used once in a regex.  Since the word-boundary context is the same for all hard keywords, the alternation can be done within one such context and all (hard) keywords get the same match tag (dict key "KEYWORD"), making it easy to give all the same highlight.

Soft keywords need different contexts to avoid false positives.  'match' and 'case' must be the first non-blank on a line and followed by ':'.  '_' must follow 'case' and space. I believe each context will have to have its own tag name, so multiple keyword tags must be mapped to 'keyword'.  

skw1 = r"^[ \t]*(?P<SKEY1>match|case)[ \t]+:"
skw2 = r"case[ \t]+(?P<SKEY1>_)\b"

Add skw1 and skw2 to the prog definition, which should use "|".join(...).

In ColorDelegator.LoadTagDefs (which should be renamed), replace

            "KEYWORD": idleConf.GetHighlight(theme, "keyword"),

with
            "KEYWORD": keydef
            "SKEY1": keydef
            "SKEY2": keydef

after first defining keydef with

        keydef = idleConf.GetHighlight(theme, "keyword")

Some new tests will be needed.
History
Date User Action Args
2021-05-02 18:05:45terry.reedysetrecipients: + terry.reedy, taleinat, epaine, kj
2021-05-02 18:05:45terry.reedysetmessageid: <1619978745.1.0.762382476495.issue44010@roundup.psfhosted.org>
2021-05-02 18:05:45terry.reedylinkissue44010 messages
2021-05-02 18:05:44terry.reedycreate