classification
Title: Add = to f-strings for easier debugging.
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: barry, eric.smith, larry, levkivskyi, mbussonn, pablogsal, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-05-06 18:10 by eric.smith, last changed 2019-05-22 16:20 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13123 merged eric.smith, 2019-05-06 18:11
PR 13059 eric.smith, 2019-05-07 16:40
PR 13249 merged pablogsal, 2019-05-11 17:18
PR 13256 merged pablogsal, 2019-05-12 00:21
Messages (18)
msg341582 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-06 18:10
This is an alternative proposal to issue36774.

We (Eric V. Smith and Larry Hastings) propose a minor language
change.  This level of change doesn't require a PEP, so in this
post-BDFL world what we need is "a consensus among core developers".
So please vote!  Note that "+" is typed using "shift-=", and the "1"
key can be found very nearby.


Python programmers often use "printf-style" debugging.  In the
(really) bad old days this was pretty wordy:
    print "foo=", foo, "bar=", bar

f-strings make this slightly nicer to type:
    print(f"foo={foo} bar={bar}")

But you still have to repeat yourself: you have to write
out the *string* "foo", and then the *expession* "foo".
Wouldn't it be nice if you didn't have to?

f-strings are uniquely able to help with this.  Their implementation
requires them to know the original text of the expression which they
then compile.  It's not difficult for f-strings to retain the text
and prepend it; the tricky part is figuring out how to spell it.

The initial idea was to use an f-string "conversion", which we
originally spelled "!=":
    f'{foo!=}'
This spelling won't work, because f-strings permit arbitrary Python
expressions, and != of course tests for inequality.

We considered other spellings:
    !d (debug)
    !e (equals)
    !x (?)
    !! (easy to type)
We'd planned to go with !d.  In fact Eric gave a lightning talk
about this on Friday night and used this spelling.

And then!  On Saturday, the best spelling revealed itself!  Behold
the majesty of:
    {foo=}
This code:
    foo=5
    print(f"{foo=}")
would print
    foo=5

With this spelling change, we've also refined the semantics.

By default, f-strings use format() (technically they call
__format__ on the value).  But the point of this is for debugging.
But you want repr() for debugging.  When you use this on a string,
you want to see the quoted string; when you use this on a datetime
object, you want to see the datetime repr, not the default
formatted string.

Second, this is now composable with conversions.  So you can use
    {foo=!s}
to use str() instead of repr() on the value.

Relatedly, we've added a new conversion: "!f" means "use format()",
which you could never explicitly specify before.  For example, to only
format pi to two decimal places:
    f"{math.pi=!f:.2f}" => "3.14"

Finally, and this is the best part: what if you want whitespace around
the equals sign?  Well, to the left is no problem; whitespace is preserved
from the original text inside the curly braces:
    f"{ chr(65) =}" => " chr(65) ='A'"
But we also explicitly permit, and preserve, whitespace *after* the
equals sign:
    f"{chr(65) = }" => "chr(65) = 'A'"

What's particularly elegant is that we simply preserve all the
characters up to the final delimiter.  The equals sign sets a flag
but doesn't stop flushing through text.  So this:
       vvvvvvvvvv
    f"{chr(65) = }"
is *exactly* the same as this:
       vvvvvvvvvv
      "chr(65) = 'A'"

Please vote!


Eric and /arry
msg341583 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-06 18:21
I'll assume you can resolve any weird corner cases, in which case +1
msg341668 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-05-07 01:39
+1 from me (as a big fan of print-debugging).
msg341682 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-07 05:05
I like this!

Except that I think that !f is not needed. You can use repr by default only when no format spec is specified, and add explicit !r if you want to use repr with the format spec. If you want to format the value without repr and the format spec -- specify the empty format spec: f"{foo=:}".
msg341691 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-07 07:32
> Except that I think that !f is not needed. You can use repr by default only when no format spec is specified, and add explicit !r if you want to use repr with the format spec. If you want to format the value without repr and the format spec -- specify the empty format spec: f"{foo=:}".

I had this working in issue36774, but it seems like a little too much magic. It also prevents you from formatting the result of the repr, which works in f-strings without the =.

Say you wanted a fixed width output. You need to apply a format to the value of the repr:

>>> nums = [1/3, 1.0, 10.0, math.pi]
>>> for n in nums:
...   print(f'*{n=}*')
... 
*n=0.3333333333333333*
*n=1.0*
*n=10.0*
*n=3.141592653589793*

>>> for n in nums:
...   print(f'*{n=:30}*')
... 
*n=0.3333333333333333            *
*n=1.0                           *
*n=10.0                          *
*n=3.141592653589793             *

If the presence of a format spec meant automatically apply the format to the value being printed, this wouldn't be possible.
msg341694 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-07 07:52
You can use f'*{n=!r:30}*' if you want to format the result of the repr.

In you example the format spec is applied to both the value and the literal representation of the expression. Is it an error? I do not think this is an expected behavior. If you want to apply it to both the literal expression and its value you can use the nested f-string: f"*{f'{n=}':30}*".

There is not too much more magic here: if both converter and format specifier are omitted use !r because it is a common special case. I think it is better than the other difference in the default converter used for debugging and normal formatting.
msg341696 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-07 08:17
> In you example the format spec is applied to both the value and the literal representation of the expression. Is it an error? I do not think this is an expected behavior.

No, you're misreading it. I admit that my example wasn't great. Try this one:

>>> for n in nums:
...   print(f'*{n=:+<30}*')
... 
*n=0.3333333333333333++++++++++++*
*n=1.0+++++++++++++++++++++++++++*
*n=10.0++++++++++++++++++++++++++*
*n=3.141592653589793+++++++++++++*

> If you want to apply it to both the literal expression and its value you can use the nested f-string: f"*{f'{n=}':30}*".

Correct. There's a similar discussion in issue36774.
msg341722 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2019-05-07 13:29
> I think that !f is not needed. You can use repr by default only when
> no format spec is specified, and add explicit !r if you want to use
> repr with the format spec.

Actually that's how !d worked.  We changed the behavior because it was too "magical".  We need to keep the f-strings format spec simple so it was easier to remember.  I for one already have difficulty remembering how f-string formatting works, I don't want to make add even more complications.

In the current proposal, the special syntax must be specified in a particular order, and the order is easy to remember because information always flows from left-to-right. The "=" must come before the "!" and/or the ":", and the "!" must come before the ":".  Like so:

   f'{foo
         =
          !s
            :20}'

Modification information strictly flows from left to right:

* The = changes the "conversion function" to repr, but then you can override the conversion function with !.

* The : format spec runs __format__ on the stuff to its left; if you're using the "format" conversion function, it applies the spec directly, otherwise it calls format with that spec to the output (the string) you got from the conversion function.


If we made the default conversion function when using = dependent on the presence or absence of the format spec, now we have information flowing to the left, all the way from the end to the beginning.  Eric and I agree: this is too magical and too hard to remember.  We want to keep it simple.

(True story: Eric had the !d implementation already done and ready for checkin.  When he changed it to this = syntax he actually mostly *threw out* code, because this way is simpler and more regular.  Hopefully you're thinking "well THAT sounds nice!"--we agree.)
msg341732 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-07 14:18
After discussing this with Guido and Larry, we're going to go with the "implicit format mode", as outlined by Serhiy, and drop the !f feature.

So the rules are:
{x=} -> "x="+repr(x)
{x=:.2f} -> "x="+format(x, ".2f")
{x=:} -> "x="+format(x, "")
{x=:!s:20} -> "x="+format(str(x), "20")
{x=:!r:20} -> "x="+format(repr(x), "20")

I think the 95% case will be {x=}, the 99%+ case will be {x=:2f} case. So I'm happy with this outcome. All functionality you had available with !f is still available, but with slightly different spellings. The most common cases now have the shortest spellings.
msg341774 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-07 16:26
The most recent version of the PR 0ec4daead4d5b3c42613bd210a51da60932035f3 has the behavior without !f and auto-selecting what used to be !f. It's ready for a final review before I commit it.
msg341940 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-05-08 20:28
New changeset 9a4135e939bc223f592045a38e0f927ba170da32 by Eric V. Smith in branch 'master':
bpo-36817: Add f-string debugging using '='. (GH-13123)
https://github.com/python/cpython/commit/9a4135e939bc223f592045a38e0f927ba170da32
msg342208 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 16:55
Commit 9a4135e939bc223f592045a38e0f927ba170da32 introduced a reference leak:

https://buildbot.python.org/all/#/builders/80/builds/587/steps/3/logs/stdio


Bisect results for test_future:

0:00:00 load avg: 10.04 [1/1] test_future
beginning 9 repetitions
123456789
.........
test_future leaked [24, 24, 24, 24] references, sum=96
test_future leaked [24, 24, 24, 24] memory blocks, sum=96
test_future failed

== Tests result: FAILURE ==

1 test failed:
    test_future

Total duration: 300 ms
Tests result: FAILURE
9a4135e939bc223f592045a38e0f927ba170da32 is the first bad commit
commit 9a4135e939bc223f592045a38e0f927ba170da32
Author: Eric V. Smith <ericvsmith@users.noreply.github.com>
Date:   Wed May 8 16:28:48 2019 -0400

    bpo-36817: Add f-string debugging using '='. (GH-13123)

    If a "=" is specified a the end of an f-string expression, the f-string will evaluate to the text of the expression, followed by '=', followed by the repr of the value of the expression.

:040000 040000 303c86dc65bb09cade6b8f2a0fa3f97715f1793b 7ddfc1c6c5f86bf6d0a38a64ff1415c7ca55a5fe M      Doc
:040000 040000 88a8dc9ffb46d652d086168e18e6a1265e0bfb92 847032e70ebd74ad7e828f8013acc7bdc4570779 M      Include
:040000 040000 119e11a4f2fa3966ca300d06dd44c46dcbae11a8 ed8c093aecd82a6b886d83218681df886f653206 M      Lib
:040000 040000 9c8d303bc3b468a06c39389520795bd34cfc7534 36b368a8e4ddd3f470c99b68b00a4a3a30c8c2b1 M      Misc
:040000 040000 ac4c322bf5eb2685c31b375bb8a7d49a7b6a9ed9 8e311dcfcafb1260eb5c617783318088a029b067 M      Parser
:040000 040000 e9db8405b51b7f72015f70556ce611c371450731 9c3da4ade5f48b3a2e842d845cafa58ba21089a3 M      Python
bisect run success
msg342211 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 17:19
Opened PR13249 to fix the leak
msg342220 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-11 19:54
New changeset 5833e94d8615ea18b14e4830ecdb868aec81b378 by Pablo Galindo in branch 'master':
bpo-36817: Fix reference leak for expr_text in f-string = parsing (GH-13249)
https://github.com/python/cpython/commit/5833e94d8615ea18b14e4830ecdb868aec81b378
msg342231 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-12 00:43
New changeset 26f55c29f2316648939ad12a9d3730a2118da2ea by Pablo Galindo in branch 'master':
bpo-36817: Do not decrement reference for expr_text on fstring = parsing failure (GH-13256)
https://github.com/python/cpython/commit/26f55c29f2316648939ad12a9d3730a2118da2ea
msg343106 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2019-05-21 22:10
I'm not quite sure I completely understand how this is implemented and all the possibilities;  – so I would appreciate reviews on the issue (and patch) to handle this in ast-unparse. 

See https://bugs.python.org/issue37003

Thanks,
msg343210 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-22 16:19
The reason the main CI did not catch this is that test_tools is only executed on a (random) subset of all the files if I remember correctly because when executed on all files it massively increases the time of the CI.
msg343211 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-05-22 16:20
Anecdotally, this happened as well when in the implementation of PEP572
History
Date User Action Args
2019-05-22 16:20:16pablogsalsetmessages: + msg343211
2019-05-22 16:19:54pablogsalsetmessages: + msg343210
2019-05-21 22:10:23mbussonnsetnosy: + mbussonn
messages: + msg343106
2019-05-12 00:43:07pablogsalsetmessages: + msg342231
2019-05-12 00:21:31pablogsalsetpull_requests: + pull_request13168
2019-05-11 19:54:49pablogsalsetstatus: open -> closed
resolution: fixed
2019-05-11 19:54:39pablogsalsetmessages: + msg342220
2019-05-11 17:19:17pablogsalsetmessages: + msg342211
stage: patch review -> resolved
2019-05-11 17:18:33pablogsalsetstage: resolved -> patch review
pull_requests: + pull_request13160
2019-05-11 16:55:29pablogsalsetstatus: closed -> open

nosy: + pablogsal
messages: + msg342208

resolution: fixed -> (no value)
2019-05-08 20:33:10eric.smithsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-05-08 20:28:54eric.smithsetmessages: + msg341940
2019-05-07 16:40:03eric.smithsetpull_requests: + pull_request13079
2019-05-07 16:26:13eric.smithsetmessages: + msg341774
2019-05-07 14:18:03eric.smithsetmessages: + msg341732
2019-05-07 13:29:02larrysetmessages: + msg341722
2019-05-07 08:17:50eric.smithsetmessages: + msg341696
2019-05-07 07:52:01serhiy.storchakasetmessages: + msg341694
2019-05-07 07:32:42eric.smithsetmessages: + msg341691
2019-05-07 05:05:05serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg341682
2019-05-07 01:39:04levkivskyisetnosy: + levkivskyi
messages: + msg341668
2019-05-06 18:24:31eric.smithsettype: enhancement
2019-05-06 18:21:39barrysettype: enhancement -> (no value)
messages: + msg341583
2019-05-06 18:19:51eric.smithsettype: enhancement
2019-05-06 18:17:48barrysetnosy: + barry
2019-05-06 18:11:51eric.smithsetkeywords: + patch
pull_requests: + pull_request13035
2019-05-06 18:10:56eric.smithcreate