classification
Title: f-strings: Add a !d conversion for ease of debugging
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: barry, dirn, eric.smith, gregory.p.smith, paul.moore, serhiy.storchaka, steven.daprano, xtreak
Priority: normal Keywords: patch

Created on 2019-05-02 11:40 by eric.smith, last changed 2019-05-07 14:10 by eric.smith. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13059 closed eric.smith, 2019-05-02 15:58
Messages (20)
msg341261 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-02 11:40
I originally propsed this here: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html, and there has also been some discussion on discourse.

The proposal is to add a !d "type conversion" to f-strings, to make for simpler "print-based debugging". The idea is that !d would be like !r, etc., except that the resulting string would be:

- the text of the expression, followed by
- an equal sign, followed by
- the value of the expression

So this code:
s = 'A string'
result = f'{s!d}'

Would set result to 's="A string"'.

Note that the text of the expression is exactly what's between the
opening brace and the !. So:
result = f'{s !d}'

Would set result to 's ="A string"'. Note the space before the equal sign.

I can't decide if I'm going to allow a format specifier. If I do, it would apply to the entire resulting string. So:
result = f'{s!d:*^20}'

Would set result to '****s="A string"****'.

But I might reserve format specs for further use, perhaps to apply to the value of the expression instead of the resulting string.

Of course this is of most value for more complex expressions. This:
v = 3
result = f'{v*9+15!d}'

Would set result to 'v*9+15=42'.

I'd expect this mostly to be used with print() or logging, but it's a general f-string feature.

I have a patch almost ready.
msg341262 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-05-02 12:05
On Thu, May 02, 2019 at 11:40:45AM +0000, Eric V. Smith wrote:
> 
> New submission from Eric V. Smith <eric@trueblade.com>:
> 
> I originally propsed this here: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html, and there has also been some discussion on discourse.
> 
> The proposal is to add a !d "type conversion" to f-strings, to make for simpler "print-based debugging". The idea is that !d would be like !r, etc., except that the resulting string would be:
> 
> - the text of the expression, followed by
> - an equal sign, followed by
> - the value of the expression

I don't want to see this added as a special case to f-strings. I think 
there are enough use-cases for having access to expressions, complete 
with source code, as first-class values to make this a general feature 
of the language and not baked into f-strings. I have a proto-PEP 
discussing this.

> I have a patch almost ready.

Please don't limit this useful feature to f-strings. "Debugging strings" 
only scratches the surface of what this could be useful for.
msg341263 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-02 12:47
I support a general mechanism. But I think that if it's much more than the 2 characters that this would take, then I don't want to use it for f-strings. The whole point here (as with all f-string features) is a concise way to get the string you're after.

But if you have some ideas, I'm willing to entertain them. I want this feature to land in 3.8, for which we're rapidly running out of time.
msg341268 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-02 15:02
for reference, the discourse thread: https://discuss.python.org/t/f-string-debug-conversion/99/14
msg341270 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-02 15:07
Steven:  We shouldn't block this immediately useful feature from going in for f-strings on waiting for some much broader first class access to expressions feature.  !d would be practical today.
msg341271 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-02 15:11
hallway conversation with Eric: neither of us have immediately good thoughts on a spelling other than !d for this.  !! or != seem esoteric and/or unparseable, using a capital letter like !D could be new and odd and might be useful to differentiate types of debug output (!d vs !D, etc) so lets not start with caps.  thus I'm not suggesting any alternative bikeshed color.
msg341273 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-05-02 15:18
!d sounds good and makes teaching easier. Adding additional symbol sort of clutters the experience in f'{name!!}'. I think it will be a good thing to have it in 3.8 since it serves one of the common things in debugging f'name = {name}' and perhaps it could be expanded out of f-strings after feedback.

Rust also introduced similar feature with https://doc.rust-lang.org/beta/std/macro.dbg.html and received very positive feedback . This will be a very exciting thing to look forward in 3.8 :)
msg341283 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-02 16:30
To implement converting printf-style string formatting into f-string expressions (see issue28307) I need to add new convertors:

'd': int(x) where x is a number, used for %d, %u, %i
'i': operator.index(i), used for %x, %o, %b
'f': float(x) where x is a number, used for %f, %g, %e

They may be private, only exposed in the AST between optimizer and code generator. But they can also be supported by Python grammar and str.format() if there is a use case for this.

If 'd' be used for other purposes, I will need to find other character for converting a number to integer (with possible truncation). Any suggestions?
msg341289 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-02 17:26
regarding issue28307 - It is not always correct to convert a %d %u %i used to render v into f'{int(v)}'.  That'd lose the TypeError when v is not an integer.

If you are looking at making internal private conversions for the purposes of that issue, I think they should be things that'll never conflict with a normal public API token that might be used in the future. (ie: don't reserve 'd' 'i' or 'f' for internal only use - use special values)  i'm being vague on purpose here as i don't know how f-strings are parsed and tokenized.
msg341298 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-05-02 18:40
+1 from me. It's something I'd find useful, and it's a natural extension of the f-string syntax.

> I can't decide if I'm going to allow a format specifier.

The only useful interpretation IMO would be for {expr!d:fmt} to expand to expr={expr:fmt}. If you're not willing to include that in the initial implementation, I'd rather see :fmt reserved for now, with the intention that it's implemented like this at a later date. Having :fmt apply to the whole string including the "expr=" bit would be basically useless to me. For a motivating example, consider f"{datetime.now()!d:%Y-%m-%d}", which is something I could easily imagine using.

Steven D'Aprano:
> I think there are enough use-cases for having access to
> expressions, complete with source code, as first-class
> values to make this a general feature of the language
> and not baked into f-strings. I have a proto-PEP
> discussing this.

I have no problem with something like this, but I don't think it precludes the proposed f-string extension. The use cases are sufficiently different that I'd expect the two features to live happily together - there's no need to block the f-string extension for a proposal like this.
msg341306 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-02 19:25
Re: format specs and what they apply to.

The problem is that you want f"{expr!d}" to expand to f"expr={repr(expr)}". And I think you want that because it's the most useful behavior for strings. Without the repr it's not very useful for strings.

But you want to have the format spec apply to the expression, not the repr of the expression. So maybe f"{expr!d:spec}" should expand to f"expr={repr(format(expr, spec))}"

So
f"{datetime.now()!d:%Y-%m-%d}"
would become:
f"datetime.now()={repr(format(datetime.now(), '%Y-%m-%d'))}"
but that gives:
"datetime.now()='2019-05-02'"

Do we want the repr of the resulting string here (the single quotes around 2019-05-02)? I think probably no (think float formatting).

So the question is: how do you get repr by default, but allow the format spec?

The only thing I've come up with is:
- f"{expr!d}" expands to f"expr={repr(expr)}", but
- f"{expr!d:spec} expands to f"expr={format(expr, spec)}"

I think this is the most useful version. But is it too complex to explain?
msg341313 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-05-02 19:41
> So the question is: how do you get repr by default, but allow the format spec?
>
> The only thing I've come up with is:
> - f"{expr!d}" expands to f"expr={repr(expr)}", but
> - f"{expr!d:spec} expands to f"expr={format(expr, spec)}"
>
> I think this is the most useful version. But is it too complex to explain?

Agreed, this is the most useful version. Not only do I not think it's
too complicated to explain, I actually think it's the obvious
behaviour, and what people would expect even without an explanation.

If asked, I'd explain it as:

    f"{expr!d:spec}" expands to "expr=<the value of expr, formatted
using spec>". If ":spec" is omitted, repr() is used.

That seems simple enough to me - the key is that we're just saying "if
:spec is omitted, we use repr".
msg341328 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-03 08:32
The most recent version of the patch implements the conditional repr/format behavior. I'm pretty happy with it.

Other than docs, I think this is done. I'm going to discuss it with a few more people at PyCon, then commit it.

There's a slight optimization I'm considering (pre-append the '=' to the expression text at compile time, instead of at runtime), but I'll do that later, if ever.
msg341331 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-03 10:25
This conversion is very special. For all other conversions {value:!c:format_spec} is equivalent to format(conv(value), format_spec), but not this conversion. It is also specific for f-strings, and is not particularly useful in format strings.

Would not be better to use more special syntax for it? For example "!=" or "!!"? Letters can be reserved for future "normal" conversions. It may be we even allowed to register new converters by name.
msg341332 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-03 11:16
!= would be my preference, but it can't work. f'{0!=1}' is already legal.

I'm not crazy about !!. I think that will be too confusing.
msg341371 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-05-04 04:23
> Steven: We shouldn't block this immediately useful feature from going 
> in for f-strings on waiting for some much broader first class access 
> to expressions feature.  !d would be practical today.

I hear you, and after giving it much more thought I agree.
msg341378 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-04 08:40
Serhiy's point about how special this is is very valid. It's so special that I can't figure out where to document it. f-strings are really only documented in Doc/reference/lexical_analysis.rst, and !d details, especially the format/repr distinction, seems like too much information for that document. But I could be wrong about that.

And since this feature can't be used in str.format(), it can't be documented in Doc/library/string.rst. In fact, it should contain a note about !d not applying.

Rather than make the documentation worse, I think I'll just open a separate issue for it when I commit this. Maybe someone else will have some ideas.
msg341379 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-04 11:52
And for those who *really* want to be able to apply a format spec to the result of the entire !d expression, you can always use nested f-strings:

>>> for x in [3.1415, 0.5772156649, 100]:
...   print(f'{f"{x!d:.1f}":*^20}')
... 
*******x=3.1********
*******x=0.6********
******x=100.0*******

Not that I recommend this, but at least it's possible.
msg341586 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-06 18:26
See issue36817 for an alternative proposal.
msg341730 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-07 14:10
After discussing this with Guido, we're going to go with a tweaked version of the '=' version of this in issue36817. So, I'm going to close this and continue the discussion there.
History
Date User Action Args
2019-05-07 14:10:46eric.smithsetstatus: open -> closed
resolution: rejected
messages: + msg341730

stage: patch review -> resolved
2019-05-06 18:26:33eric.smithsetmessages: + msg341586
2019-05-05 18:24:30eric.smithsetnosy: + barry
2019-05-04 11:52:52eric.smithsetmessages: + msg341379
2019-05-04 08:40:31eric.smithsetmessages: + msg341378
2019-05-04 04:23:06steven.dapranosetmessages: + msg341371
2019-05-03 11:16:10eric.smithsetmessages: + msg341332
2019-05-03 10:25:41serhiy.storchakasetmessages: + msg341331
2019-05-03 08:32:46eric.smithsetmessages: + msg341328
2019-05-02 19:41:42paul.mooresetmessages: + msg341313
2019-05-02 19:25:27eric.smithsetmessages: + msg341306
2019-05-02 18:40:38paul.mooresetnosy: + paul.moore
messages: + msg341298
2019-05-02 17:26:13gregory.p.smithsetmessages: + msg341289
2019-05-02 16:30:53serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg341283
2019-05-02 15:58:53eric.smithsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request12976
2019-05-02 15:18:56xtreaksetmessages: + msg341273
2019-05-02 15:11:26gregory.p.smithsetmessages: + msg341271
2019-05-02 15:07:17gregory.p.smithsetmessages: + msg341270
2019-05-02 15:02:58gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg341268
2019-05-02 14:01:12xtreaksetnosy: + xtreak
2019-05-02 13:54:10dirnsetnosy: + dirn
2019-05-02 12:47:29eric.smithsetmessages: + msg341263
2019-05-02 12:05:43steven.dapranosetnosy: + steven.daprano
messages: + msg341262
2019-05-02 11:40:45eric.smithcreate