Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

f-strings: Add a !d conversion for ease of debugging #80955

Closed
ericvsmith opened this issue May 2, 2019 · 20 comments
Closed

f-strings: Add a !d conversion for ease of debugging #80955

ericvsmith opened this issue May 2, 2019 · 20 comments
Assignees
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@ericvsmith
Copy link
Member

BPO 36774
Nosy @warsaw, @gpshead, @pfmoore, @ericvsmith, @stevendaprano, @dirn, @serhiy-storchaka, @tirkarthi
PRs
  • bpo-36774: Add !d to f-strings to make debugging easier. #13059
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ericvsmith'
    closed_at = <Date 2019-05-07.14:10:46.734>
    created_at = <Date 2019-05-02.11:40:45.910>
    labels = ['interpreter-core', 'type-feature', '3.8']
    title = 'f-strings: Add a !d conversion for ease of debugging'
    updated_at = <Date 2019-05-07.14:10:46.733>
    user = 'https://github.com/ericvsmith'

    bugs.python.org fields:

    activity = <Date 2019-05-07.14:10:46.733>
    actor = 'eric.smith'
    assignee = 'eric.smith'
    closed = True
    closed_date = <Date 2019-05-07.14:10:46.734>
    closer = 'eric.smith'
    components = ['Interpreter Core']
    creation = <Date 2019-05-02.11:40:45.910>
    creator = 'eric.smith'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36774
    keywords = ['patch']
    message_count = 20.0
    messages = ['341261', '341262', '341263', '341268', '341270', '341271', '341273', '341283', '341289', '341298', '341306', '341313', '341328', '341331', '341332', '341371', '341378', '341379', '341586', '341730']
    nosy_count = 8.0
    nosy_names = ['barry', 'gregory.p.smith', 'paul.moore', 'eric.smith', 'steven.daprano', 'dirn', 'serhiy.storchaka', 'xtreak']
    pr_nums = ['13059']
    priority = 'normal'
    resolution = 'rejected'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue36774'
    versions = ['Python 3.8']

    @ericvsmith
    Copy link
    Member Author

    I originally propsed this here: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html, and there has also been some discussion on discourse.

    The proposal is to add a !d "type conversion" to f-strings, to make for simpler "print-based debugging". The idea is that !d would be like !r, etc., except that the resulting string would be:

    • the text of the expression, followed by
    • an equal sign, followed by
    • the value of the expression

    So this code:
    s = 'A string'
    result = f'{s!d}'

    Would set result to 's="A string"'.

    Note that the text of the expression is exactly what's between the
    opening brace and the !. So:
    result = f'{s !d}'

    Would set result to 's ="A string"'. Note the space before the equal sign.

    I can't decide if I'm going to allow a format specifier. If I do, it would apply to the entire resulting string. So:
    result = f'{s!d:*^20}'

    Would set result to 's="A string"'.

    But I might reserve format specs for further use, perhaps to apply to the value of the expression instead of the resulting string.

    Of course this is of most value for more complex expressions. This:
    v = 3
    result = f'{v*9+15!d}'

    Would set result to 'v*9+15=42'.

    I'd expect this mostly to be used with print() or logging, but it's a general f-string feature.

    I have a patch almost ready.

    @ericvsmith ericvsmith added the 3.8 only security fixes label May 2, 2019
    @ericvsmith ericvsmith self-assigned this May 2, 2019
    @ericvsmith ericvsmith added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement labels May 2, 2019
    @stevendaprano
    Copy link
    Member

    On Thu, May 02, 2019 at 11:40:45AM +0000, Eric V. Smith wrote:

    New submission from Eric V. Smith <eric@trueblade.com>:

    I originally propsed this here: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html, and there has also been some discussion on discourse.

    The proposal is to add a !d "type conversion" to f-strings, to make for simpler "print-based debugging". The idea is that !d would be like !r, etc., except that the resulting string would be:

    • the text of the expression, followed by
    • an equal sign, followed by
    • the value of the expression

    I don't want to see this added as a special case to f-strings. I think
    there are enough use-cases for having access to expressions, complete
    with source code, as first-class values to make this a general feature
    of the language and not baked into f-strings. I have a proto-PEP
    discussing this.

    I have a patch almost ready.

    Please don't limit this useful feature to f-strings. "Debugging strings"
    only scratches the surface of what this could be useful for.

    @ericvsmith
    Copy link
    Member Author

    I support a general mechanism. But I think that if it's much more than the 2 characters that this would take, then I don't want to use it for f-strings. The whole point here (as with all f-string features) is a concise way to get the string you're after.

    But if you have some ideas, I'm willing to entertain them. I want this feature to land in 3.8, for which we're rapidly running out of time.

    @gpshead
    Copy link
    Member

    gpshead commented May 2, 2019

    for reference, the discourse thread: https://discuss.python.org/t/f-string-debug-conversion/99/14

    @gpshead
    Copy link
    Member

    gpshead commented May 2, 2019

    Steven: We shouldn't block this immediately useful feature from going in for f-strings on waiting for some much broader first class access to expressions feature. !d would be practical today.

    @gpshead
    Copy link
    Member

    gpshead commented May 2, 2019

    hallway conversation with Eric: neither of us have immediately good thoughts on a spelling other than !d for this. !! or != seem esoteric and/or unparseable, using a capital letter like !D could be new and odd and might be useful to differentiate types of debug output (!d vs !D, etc) so lets not start with caps. thus I'm not suggesting any alternative bikeshed color.

    @tirkarthi
    Copy link
    Member

    !d sounds good and makes teaching easier. Adding additional symbol sort of clutters the experience in f'{name!!}'. I think it will be a good thing to have it in 3.8 since it serves one of the common things in debugging f'name = {name}' and perhaps it could be expanded out of f-strings after feedback.

    Rust also introduced similar feature with https://doc.rust-lang.org/beta/std/macro.dbg.html and received very positive feedback . This will be a very exciting thing to look forward in 3.8 :)

    @serhiy-storchaka
    Copy link
    Member

    To implement converting printf-style string formatting into f-string expressions (see bpo-28307) I need to add new convertors:

    'd': int(x) where x is a number, used for %d, %u, %i
    'i': operator.index(i), used for %x, %o, %b
    'f': float(x) where x is a number, used for %f, %g, %e

    They may be private, only exposed in the AST between optimizer and code generator. But they can also be supported by Python grammar and str.format() if there is a use case for this.

    If 'd' be used for other purposes, I will need to find other character for converting a number to integer (with possible truncation). Any suggestions?

    @gpshead
    Copy link
    Member

    gpshead commented May 2, 2019

    regarding bpo-28307 - It is not always correct to convert a %d %u %i used to render v into f'{int(v)}'. That'd lose the TypeError when v is not an integer.

    If you are looking at making internal private conversions for the purposes of that issue, I think they should be things that'll never conflict with a normal public API token that might be used in the future. (ie: don't reserve 'd' 'i' or 'f' for internal only use - use special values) i'm being vague on purpose here as i don't know how f-strings are parsed and tokenized.

    @pfmoore
    Copy link
    Member

    pfmoore commented May 2, 2019

    +1 from me. It's something I'd find useful, and it's a natural extension of the f-string syntax.

    I can't decide if I'm going to allow a format specifier.

    The only useful interpretation IMO would be for {expr!d:fmt} to expand to expr={expr:fmt}. If you're not willing to include that in the initial implementation, I'd rather see :fmt reserved for now, with the intention that it's implemented like this at a later date. Having :fmt apply to the whole string including the "expr=" bit would be basically useless to me. For a motivating example, consider f"{datetime.now()!d:%Y-%m-%d}", which is something I could easily imagine using.

    Steven D'Aprano:

    I think there are enough use-cases for having access to
    expressions, complete with source code, as first-class
    values to make this a general feature of the language
    and not baked into f-strings. I have a proto-PEP
    discussing this.

    I have no problem with something like this, but I don't think it precludes the proposed f-string extension. The use cases are sufficiently different that I'd expect the two features to live happily together - there's no need to block the f-string extension for a proposal like this.

    @ericvsmith
    Copy link
    Member Author

    Re: format specs and what they apply to.

    The problem is that you want f"{expr!d}" to expand to f"expr={repr(expr)}". And I think you want that because it's the most useful behavior for strings. Without the repr it's not very useful for strings.

    But you want to have the format spec apply to the expression, not the repr of the expression. So maybe f"{expr!d:spec}" should expand to f"expr={repr(format(expr, spec))}"

    So
    f"{datetime.now()!d:%Y-%m-%d}"
    would become:
    f"datetime.now()={repr(format(datetime.now(), '%Y-%m-%d'))}"
    but that gives:
    "datetime.now()='2019-05-02'"

    Do we want the repr of the resulting string here (the single quotes around 2019-05-02)? I think probably no (think float formatting).

    So the question is: how do you get repr by default, but allow the format spec?

    The only thing I've come up with is:

    • f"{expr!d}" expands to f"expr={repr(expr)}", but
    • f"{expr!d:spec} expands to f"expr={format(expr, spec)}"

    I think this is the most useful version. But is it too complex to explain?

    @pfmoore
    Copy link
    Member

    pfmoore commented May 2, 2019

    So the question is: how do you get repr by default, but allow the format spec?

    The only thing I've come up with is:

    • f"{expr!d}" expands to f"expr={repr(expr)}", but
    • f"{expr!d:spec} expands to f"expr={format(expr, spec)}"

    I think this is the most useful version. But is it too complex to explain?

    Agreed, this is the most useful version. Not only do I not think it's
    too complicated to explain, I actually think it's the obvious
    behaviour, and what people would expect even without an explanation.

    If asked, I'd explain it as:

    f"{expr!d:spec}" expands to "expr=\<the value of expr, formatted
    

    using spec>". If ":spec" is omitted, repr() is used.

    That seems simple enough to me - the key is that we're just saying "if
    :spec is omitted, we use repr".

    @ericvsmith
    Copy link
    Member Author

    The most recent version of the patch implements the conditional repr/format behavior. I'm pretty happy with it.

    Other than docs, I think this is done. I'm going to discuss it with a few more people at PyCon, then commit it.

    There's a slight optimization I'm considering (pre-append the '=' to the expression text at compile time, instead of at runtime), but I'll do that later, if ever.

    @serhiy-storchaka
    Copy link
    Member

    This conversion is very special. For all other conversions {value:!c:format_spec} is equivalent to format(conv(value), format_spec), but not this conversion. It is also specific for f-strings, and is not particularly useful in format strings.

    Would not be better to use more special syntax for it? For example "!=" or "!!"? Letters can be reserved for future "normal" conversions. It may be we even allowed to register new converters by name.

    @ericvsmith
    Copy link
    Member Author

    != would be my preference, but it can't work. f'{0!=1}' is already legal.

    I'm not crazy about !!. I think that will be too confusing.

    @stevendaprano
    Copy link
    Member

    Steven: We shouldn't block this immediately useful feature from going
    in for f-strings on waiting for some much broader first class access
    to expressions feature. !d would be practical today.

    I hear you, and after giving it much more thought I agree.

    @ericvsmith
    Copy link
    Member Author

    Serhiy's point about how special this is is very valid. It's so special that I can't figure out where to document it. f-strings are really only documented in Doc/reference/lexical_analysis.rst, and !d details, especially the format/repr distinction, seems like too much information for that document. But I could be wrong about that.

    And since this feature can't be used in str.format(), it can't be documented in Doc/library/string.rst. In fact, it should contain a note about !d not applying.

    Rather than make the documentation worse, I think I'll just open a separate issue for it when I commit this. Maybe someone else will have some ideas.

    @ericvsmith
    Copy link
    Member Author

    And for those who *really* want to be able to apply a format spec to the result of the entire !d expression, you can always use nested f-strings:

    >>> for x in [3.1415, 0.5772156649, 100]:
    ...   print(f'{f"{x!d:.1f}":*^20}')
    ... 
    *******x=3.1********
    *******x=0.6********
    ******x=100.0*******

    Not that I recommend this, but at least it's possible.

    @ericvsmith
    Copy link
    Member Author

    See bpo-36817 for an alternative proposal.

    @ericvsmith
    Copy link
    Member Author

    After discussing this with Guido, we're going to go with a tweaked version of the '=' version of this in bpo-36817. So, I'm going to close this and continue the discussion there.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants