Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add = to f-strings for easier debugging. #80998

Closed
ericvsmith opened this issue May 6, 2019 · 18 comments
Closed

Add = to f-strings for easier debugging. #80998

ericvsmith opened this issue May 6, 2019 · 18 comments
Assignees
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@ericvsmith
Copy link
Member

BPO 36817
Nosy @warsaw, @larryhastings, @ericvsmith, @serhiy-storchaka, @ilevkivskyi, @Carreau, @pablogsal
PRs
  • bpo-36817: Add f-string debugging using '='. #13123
  • bpo-36774: Add !d to f-strings to make debugging easier. #13059
  • bpo-36817: Fix reference leak for expr_text in f-string = parsing #13249
  • bpo-36817: Do not decrement reference for expr_text on fstring = parsing failure #13256
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ericvsmith'
    closed_at = <Date 2019-05-11.19:54:49.046>
    created_at = <Date 2019-05-06.18:10:56.540>
    labels = ['interpreter-core', 'type-feature', '3.8']
    title = 'Add = to f-strings for easier debugging.'
    updated_at = <Date 2019-05-22.16:20:16.071>
    user = 'https://github.com/ericvsmith'

    bugs.python.org fields:

    activity = <Date 2019-05-22.16:20:16.071>
    actor = 'pablogsal'
    assignee = 'eric.smith'
    closed = True
    closed_date = <Date 2019-05-11.19:54:49.046>
    closer = 'pablogsal'
    components = ['Interpreter Core']
    creation = <Date 2019-05-06.18:10:56.540>
    creator = 'eric.smith'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36817
    keywords = ['patch']
    message_count = 18.0
    messages = ['341582', '341583', '341668', '341682', '341691', '341694', '341696', '341722', '341732', '341774', '341940', '342208', '342211', '342220', '342231', '343106', '343210', '343211']
    nosy_count = 7.0
    nosy_names = ['barry', 'larry', 'eric.smith', 'serhiy.storchaka', 'levkivskyi', 'mbussonn', 'pablogsal']
    pr_nums = ['13123', '13059', '13249', '13256']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue36817'
    versions = ['Python 3.8']

    @ericvsmith
    Copy link
    Member Author

    This is an alternative proposal to bpo-36774.

    We (Eric V. Smith and Larry Hastings) propose a minor language
    change. This level of change doesn't require a PEP, so in this
    post-BDFL world what we need is "a consensus among core developers".
    So please vote! Note that "+" is typed using "shift-=", and the "1"
    key can be found very nearby.

    Python programmers often use "printf-style" debugging. In the
    (really) bad old days this was pretty wordy:
    print "foo=", foo, "bar=", bar

    f-strings make this slightly nicer to type:
    print(f"foo={foo} bar={bar}")

    But you still have to repeat yourself: you have to write
    out the *string* "foo", and then the *expession* "foo".
    Wouldn't it be nice if you didn't have to?

    f-strings are uniquely able to help with this. Their implementation
    requires them to know the original text of the expression which they
    then compile. It's not difficult for f-strings to retain the text
    and prepend it; the tricky part is figuring out how to spell it.

    The initial idea was to use an f-string "conversion", which we
    originally spelled "!=":
    f'{foo!=}'
    This spelling won't work, because f-strings permit arbitrary Python
    expressions, and != of course tests for inequality.

    We considered other spellings:
    !d (debug)
    !e (equals)
    !x (?)
    !! (easy to type)
    We'd planned to go with !d. In fact Eric gave a lightning talk
    about this on Friday night and used this spelling.

    And then! On Saturday, the best spelling revealed itself! Behold
    the majesty of:
    {foo=}
    This code:
    foo=5
    print(f"{foo=}")
    would print
    foo=5

    With this spelling change, we've also refined the semantics.

    By default, f-strings use format() (technically they call
    __format__ on the value). But the point of this is for debugging.
    But you want repr() for debugging. When you use this on a string,
    you want to see the quoted string; when you use this on a datetime
    object, you want to see the datetime repr, not the default
    formatted string.

    Second, this is now composable with conversions. So you can use
    {foo=!s}
    to use str() instead of repr() on the value.

    Relatedly, we've added a new conversion: "!f" means "use format()",
    which you could never explicitly specify before. For example, to only
    format pi to two decimal places:
    f"{math.pi=!f:.2f}" => "3.14"

    Finally, and this is the best part: what if you want whitespace around
    the equals sign? Well, to the left is no problem; whitespace is preserved
    from the original text inside the curly braces:
    f"{ chr(65) =}" => " chr(65) ='A'"
    But we also explicitly permit, and preserve, whitespace *after* the
    equals sign:
    f"{chr(65) = }" => "chr(65) = 'A'"

    What's particularly elegant is that we simply preserve all the
    characters up to the final delimiter. The equals sign sets a flag
    but doesn't stop flushing through text. So this:
    vvvvvvvvvv
    f"{chr(65) = }"
    is *exactly* the same as this:
    vvvvvvvvvv
    "chr(65) = 'A'"

    Please vote!

    Eric and /arry

    @ericvsmith ericvsmith added the 3.8 only security fixes label May 6, 2019
    @ericvsmith ericvsmith self-assigned this May 6, 2019
    @ericvsmith ericvsmith added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement labels May 6, 2019
    @warsaw
    Copy link
    Member

    warsaw commented May 6, 2019

    I'll assume you can resolve any weird corner cases, in which case +1

    @warsaw warsaw removed the type-feature A feature request or enhancement label May 6, 2019
    @ericvsmith ericvsmith added the type-feature A feature request or enhancement label May 6, 2019
    @ilevkivskyi
    Copy link
    Member

    +1 from me (as a big fan of print-debugging).

    @serhiy-storchaka
    Copy link
    Member

    I like this!

    Except that I think that !f is not needed. You can use repr by default only when no format spec is specified, and add explicit !r if you want to use repr with the format spec. If you want to format the value without repr and the format spec -- specify the empty format spec: f"{foo=:}".

    @ericvsmith
    Copy link
    Member Author

    Except that I think that !f is not needed. You can use repr by default only when no format spec is specified, and add explicit !r if you want to use repr with the format spec. If you want to format the value without repr and the format spec -- specify the empty format spec: f"{foo=:}".

    I had this working in bpo-36774, but it seems like a little too much magic. It also prevents you from formatting the result of the repr, which works in f-strings without the =.

    Say you wanted a fixed width output. You need to apply a format to the value of the repr:

    >>> nums = [1/3, 1.0, 10.0, math.pi]
    >>> for n in nums:
    ...   print(f'*{n=}*')
    ... 
    *n=0.3333333333333333*
    *n=1.0*
    *n=10.0*
    *n=3.141592653589793*
    
    >>> for n in nums:
    ...   print(f'*{n=:30}*')
    ... 
    *n=0.3333333333333333            *
    *n=1.0                           *
    *n=10.0                          *
    *n=3.141592653589793             *

    If the presence of a format spec meant automatically apply the format to the value being printed, this wouldn't be possible.

    @serhiy-storchaka
    Copy link
    Member

    You can use f'*{n=!r:30}*' if you want to format the result of the repr.

    In you example the format spec is applied to both the value and the literal representation of the expression. Is it an error? I do not think this is an expected behavior. If you want to apply it to both the literal expression and its value you can use the nested f-string: f"*{f'{n=}':30}*".

    There is not too much more magic here: if both converter and format specifier are omitted use !r because it is a common special case. I think it is better than the other difference in the default converter used for debugging and normal formatting.

    @ericvsmith
    Copy link
    Member Author

    In you example the format spec is applied to both the value and the literal representation of the expression. Is it an error? I do not think this is an expected behavior.

    No, you're misreading it. I admit that my example wasn't great. Try this one:

    >>> for n in nums:
    ...   print(f'*{n=:+<30}*')
    ... 
    *n=0.3333333333333333++++++++++++*
    *n=1.0+++++++++++++++++++++++++++*
    *n=10.0++++++++++++++++++++++++++*
    *n=3.141592653589793+++++++++++++*

    If you want to apply it to both the literal expression and its value you can use the nested f-string: f"*{f'{n=}':30}*".

    Correct. There's a similar discussion in bpo-36774.

    @larryhastings
    Copy link
    Contributor

    I think that !f is not needed. You can use repr by default only when
    no format spec is specified, and add explicit !r if you want to use
    repr with the format spec.

    Actually that's how !d worked. We changed the behavior because it was too "magical". We need to keep the f-strings format spec simple so it was easier to remember. I for one already have difficulty remembering how f-string formatting works, I don't want to make add even more complications.

    In the current proposal, the special syntax must be specified in a particular order, and the order is easy to remember because information always flows from left-to-right. The "=" must come before the "!" and/or the ":", and the "!" must come before the ":". Like so:

    f'{foo
    =
    !s
    :20}'

    Modification information strictly flows from left to right:

    • The = changes the "conversion function" to repr, but then you can override the conversion function with !.

    • The : format spec runs __format__ on the stuff to its left; if you're using the "format" conversion function, it applies the spec directly, otherwise it calls format with that spec to the output (the string) you got from the conversion function.

    If we made the default conversion function when using = dependent on the presence or absence of the format spec, now we have information flowing to the left, all the way from the end to the beginning. Eric and I agree: this is too magical and too hard to remember. We want to keep it simple.

    (True story: Eric had the !d implementation already done and ready for checkin. When he changed it to this = syntax he actually mostly *threw out* code, because this way is simpler and more regular. Hopefully you're thinking "well THAT sounds nice!"--we agree.)

    @ericvsmith
    Copy link
    Member Author

    After discussing this with Guido and Larry, we're going to go with the "implicit format mode", as outlined by Serhiy, and drop the !f feature.

    So the rules are:
    {x=} -> "x="+repr(x)
    {x=:.2f} -> "x="+format(x, ".2f")
    {x=:} -> "x="+format(x, "")
    {x=:!s:20} -> "x="+format(str(x), "20")
    {x=:!r:20} -> "x="+format(repr(x), "20")

    I think the 95% case will be {x=}, the 99%+ case will be {x=:2f} case. So I'm happy with this outcome. All functionality you had available with !f is still available, but with slightly different spellings. The most common cases now have the shortest spellings.

    @ericvsmith
    Copy link
    Member Author

    The most recent version of the PR 0ec4dae has the behavior without !f and auto-selecting what used to be !f. It's ready for a final review before I commit it.

    @ericvsmith
    Copy link
    Member Author

    New changeset 9a4135e by Eric V. Smith in branch 'master':
    bpo-36817: Add f-string debugging using '='. (GH-13123)
    9a4135e

    @pablogsal
    Copy link
    Member

    Commit 9a4135e introduced a reference leak:

    https://buildbot.python.org/all/#/builders/80/builds/587/steps/3/logs/stdio

    Bisect results for test_future:

    0:00:00 load avg: 10.04 [1/1] test_future
    beginning 9 repetitions
    123456789
    .........
    test_future leaked [24, 24, 24, 24] references, sum=96
    test_future leaked [24, 24, 24, 24] memory blocks, sum=96
    test_future failed

    == Tests result: FAILURE ==

    1 test failed:
    test_future

    Total duration: 300 ms
    Tests result: FAILURE
    9a4135e is the first bad commit
    commit 9a4135e
    Author: Eric V. Smith <ericvsmith@users.noreply.github.com>
    Date: Wed May 8 16:28:48 2019 -0400

    bpo-36817: Add f-string debugging using '='. (GH-13123)
    
    If a "=" is specified a the end of an f-string expression, the f-string will evaluate to the text of the expression, followed by '=', followed by the repr of the value of the expression.
    

    :040000 040000 303c86dc65bb09cade6b8f2a0fa3f97715f1793b 7ddfc1c6c5f86bf6d0a38a64ff1415c7ca55a5fe M Doc
    :040000 040000 88a8dc9ffb46d652d086168e18e6a1265e0bfb92 847032e70ebd74ad7e828f8013acc7bdc4570779 M Include
    :040000 040000 119e11a4f2fa3966ca300d06dd44c46dcbae11a8 ed8c093aecd82a6b886d83218681df886f653206 M Lib
    :040000 040000 9c8d303bc3b468a06c39389520795bd34cfc7534 36b368a8e4ddd3f470c99b68b00a4a3a30c8c2b1 M Misc
    :040000 040000 ac4c322bf5eb2685c31b375bb8a7d49a7b6a9ed9 8e311dcfcafb1260eb5c617783318088a029b067 M Parser
    :040000 040000 e9db8405b51b7f72015f70556ce611c371450731 9c3da4ade5f48b3a2e842d845cafa58ba21089a3 M Python
    bisect run success

    @pablogsal pablogsal reopened this May 11, 2019
    @pablogsal
    Copy link
    Member

    Opened PR13249 to fix the leak

    @pablogsal
    Copy link
    Member

    New changeset 5833e94 by Pablo Galindo in branch 'master':
    bpo-36817: Fix reference leak for expr_text in f-string = parsing (GH-13249)
    5833e94

    @pablogsal
    Copy link
    Member

    New changeset 26f55c2 by Pablo Galindo in branch 'master':
    bpo-36817: Do not decrement reference for expr_text on fstring = parsing failure (GH-13256)
    26f55c2

    @Carreau
    Copy link
    Mannequin

    Carreau mannequin commented May 21, 2019

    I'm not quite sure I completely understand how this is implemented and all the possibilities; – so I would appreciate reviews on the issue (and patch) to handle this in ast-unparse.

    See https://bugs.python.org/issue37003

    Thanks,

    @pablogsal
    Copy link
    Member

    The reason the main CI did not catch this is that test_tools is only executed on a (random) subset of all the files if I remember correctly because when executed on all files it massively increases the time of the CI.

    @pablogsal
    Copy link
    Member

    Anecdotally, this happened as well when in the implementation of PEP-572

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    ilius added a commit to ilius/starcal that referenced this issue Sep 18, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants