Title: IDLE needs syntax highlighting for f-strings
Type: enhancement Stage: test needed
Components: IDLE Versions: Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: peter.otten, rhettinger, terry.reedy
Priority: normal Keywords:

Created on 2017-01-16 22:20 by rhettinger, last changed 2017-01-31 18:50 by terry.reedy.

Messages (2)
msg285585 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-01-16 22:20
Follow the lead from Vim, MagicPython, and PyCharm.  Needs separate colorization to make the expression distinct from the rest of the string.  Needs close-brace matching.  Would be desirable to have autocompletion as well.
msg286547 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-01-31 18:50
This is 3 related but somewhat distinct proposals.

1. Special handling (normal syntax colorizing) of f-expressions (the grammatical term used at  

2. Brace matching within the strings.  (For normal strings, this would be nice when the string is to be used as a format string, but there is no way to know how the string will be used.)

3. Normal identifier autocompletion (as opposed to within-string file name autocompletion).

I have two types of doubts about #1

1. Should it be done?
 a. I have not yet used f-strings, so I not know if I would want uncolorized holes within them.
 b. Do beginners use f-strings?  Would they find this more useful than confusing?
 c. Is this the sort of 'advanced feature' that Guido has said should *not* be copied from other editors?

2. Can it be done sensibly within the limits of IDLE's colorizer. 

IDLE's colorizer defines a giant regex that joins regexes that match keywords, builtins, comments, and strings (and newlines for synchronization).  Each of the latter is a named capturing group that joins alternatives for that group.  Keywords and built-in names are recognized when complete.  Partial comments and strings are recognized as soon as '#' or an open quote is typed.  There is a human-verified test of colorizing that could, I believe, be turned into a unit test of the re matching.  This would be needed for approach b. below.

The compiled re is used in a ColorDelegator instance that is part of a chain of delegators tied to a text widget.  The class code is not documented and I do not understand it well enough to modify it without adding tests.  But it was not designed for easy testing.

Sidenote: There are DEBUG prints in multiple methods (but not inrecolorize_main).  Some messages can come from multiple methods.  I should add message to the r...main method and prefix all messages with an indicator of the source so the control flow is easier to follow

I see two possible approaches to separately colorizing f-expressions within an f-string.

a. Follow the example of 'def' and 'class'.  They are recognized as a special case (of builting) and when they occur, a separate 'if' clause and re is used to colorize the following name.

The problem with doing this with f-strings is that we want to recursively apply the re...main function to a short substring, and the function is not designed for that.  We also want to do this separately for each embedded f-expression.  It might work to write a reduced version of recolorize_main as recolorize_fexp.

This approach would allow for {} matching once a closing quote is typed, but not identifier autocompletion.

b. Do the special-casing by writing special regexes to recognize a null f-string (no embedded f-expression), and beginning, middle, and ending string parts of an f-string.  But I don't know if it is possible to write an re that will *only* match null f-strings.

That aside, the f-expression would then be treated normally, and autocomplete should just work.  {} matching would be harder.  Without adding new state variables, I imagine that the end quote of the invalid f"a{b" would be seen as the beginning of a new string.
Date User Action Args
2017-01-31 18:50:14terry.reedysetmessages: + msg286547
stage: needs patch -> test needed
2017-01-17 14:05:35peter.ottensetnosy: + peter.otten
2017-01-16 22:20:08rhettingercreate