classification
Title: brace escapes are not working in formatted string literal format specifications
Type: behavior Stage:
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: crwilcox, eric.smith, jitterman, veky, zach.ware
Priority: normal Keywords: patch

Created on 2020-02-10 16:48 by jitterman, last changed 2020-02-13 06:21 by jitterman.

Files
File name Uploaded Description Edit
bugreport.py jitterman, 2020-02-10 16:48 an example that demonstrates the problem
bugreport.py jitterman, 2020-02-10 16:50 a hopefully simple example that demonstrates the problem
test_case.patch crwilcox, 2020-02-11 17:47 test case that demonstrates the defect
Messages (13)
msg361702 - (view) Author: JitterMan (jitterman) Date: 2020-02-10 16:48
It appears as if escaping the braces by doubling them up is not working properly if the braces are in a format specification within a f-string.

>>> print(f'Email:\n    {C:{{v.name}} {{v.email}}|\n    }')                         
Traceback (most recent call last):
  File "bugreport.py", line 95, in <module>
    print(f'Email:\n    {C:{{v.name}} {{v.email}}|\n    }')
NameError: name 'v' is not defined

The escaping works as expected when the string's format method is used.
msg361703 - (view) Author: JitterMan (jitterman) Date: 2020-02-10 16:50
It appears as if escaping the braces by doubling them up is not working properly if the braces are in a format specification within a f-string.

>>> print(f'Email:\n    {C:{{v.name}} {{v.email}}|\n    }')                         
Traceback (most recent call last):
  File "bugreport.py", line 95, in <module>
    print(f'Email:\n    {C:{{v.name}} {{v.email}}|\n    }')
NameError: name 'v' is not defined

The escaping works as expected when the string's format method is used.
msg361704 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-02-10 17:00
What result are you expecting here?
msg361707 - (view) Author: Chris Wilcox (crwilcox) * Date: 2020-02-10 18:55
The attached code implements `__format__` on the `Collections` class. In case 1, the template passed to `__format__` is "{v.name}: {v.email}|". In case 2, a name error will occur while processing the f string and v will not be found as no object 'v' exists in locals or globals.

In reviewing PEP 0498, https://www.python.org/dev/peps/pep-0498/, I think the difference is in what object is being formatted.

In case 1 of the attached code, the collection is being formatted. In case 2 where f-strings are used, 'v' is being formatted. Because v doesn't exist in this context, it fails. I found this in the PEP and I think it is what is going on here.

```
Note that __format__() is not called directly on each value. The actual code uses the equivalent of type(value).__format__(value, format_spec), or format(value, format_spec). See the documentation of the builtin format() function for more details.

```
msg361751 - (view) Author: JitterMan (jitterman) Date: 2020-02-10 23:44
My expectation is that doubling up the braces acts to escape them, meaning that characters between the braces is treated as simple text and passed to the __format__ method as is. The only processing that should occur on the format specification is to convert the double braces to single braces. The fact that an error occurs saying that 'v' is not defined before the __format__ method is ever called indicates that the contents of the braces are being evaluated as an expression, which fails because v is not defined in the outer scope.  Thus the f-string seems to be ignoring the escaping of the braces, but it only does so in the format specifier.
msg361774 - (view) Author: Chris Wilcox (crwilcox) * Date: 2020-02-11 01:29
Double curly braces do not indicate to not process the inner content. They indicate to include a literal curly brace. That said, I think there may be something not quite correct.

I came up with an example based on the example in the format specifiers section of the PEP.

From the PEP.
```
>>> width = 10
>>> precision = 4
>>> value = decimal.Decimal('12.34567')
>>> f'result: {value:{width}.{precision}}'
'result:      12.35'
```
The template in this instance is "10.4"

If we leave the sample the same, but don't wrap width or precision in single curly braces,
```
>>> f'result: {value:width.precision}'
```
I would expect the template "width.precision".

Further, I would expect
```
>>> f'result: {value:{{width}}.{{precision}}}'
```
to have a template of "{width}.{precision}". This is not the case.


Here is some code that should demonstrate this.
```
class Decimal:
    def __init__(self, value):
        pass
    def __format__(self, template):
        return template
width = 10
precision = 4
value = Decimal('12.34567')
print("Expect Template to be '10.4' (TRUE)")
print(f'result0: {value:{width}.{precision}}')
print("Expect Template to be 'width.precision' (TRUE)")
print(f'result1: {value:width.precision}')
print("Expect Template to be '{width}.{precision}' (FALSE)")
print(f'result2: {value:{{width}}.{{precision}}}') # ACTUAL: {10}.{4}
```
msg361787 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-02-11 07:18
I always use datetime for testing such things. Its __format__() returns its argument, as long as you don't have '%' in it.

I do agree that it's odd that the doubled braces cause the expression to be evaluated, but are still kept as single braces in the resulting string. I'll have to investigate, although I'm not sure if we can change it at this point. Surely someone, somewhere is relying on this behavior.

I'll assume that what the OP wants in my example below is a format specifier of "x{x}x":

>>> x = 42
>>> import datetime
>>> now = datetime.datetime.now()

>>> f'{now:x{{x}}x}'
'x{42}x'

Given the current behavior, I'm not sure it's possible to get 'x{x}x'. Sometimes nested f-strings will help, but I don't think that will work here. The OP might need to use a separate expression to calculate the format specifier, which I realize isn't a very satisfying solution.

>>> spec = 'x{x}x'
>>> f'{now:{spec}}'
'x{x}x'
msg361826 - (view) Author: Chris Wilcox (crwilcox) * Date: 2020-02-11 17:47
Thanks Eric. That is very handy.

I had made a test case earlier on a branch. Attached as a patch here if helpful.

I haven't tried to fix this yet, but would be interested if it is something that makes sense to address.
msg361832 - (view) Author: JitterMan (jitterman) Date: 2020-02-11 21:56
I believe it is worth fixing as it clears up some rather glaring inconsistencies␣
and enables a useful capability. Specifically,

1. Formatted string literals and the string format method are currently 
   inconsistent in the way that they handle double braces in the format 
   specifier.

    >>> x = 42
    >>> import datetime
    >>> now = datetime.datetime.now()

    >>> f'{now:x{{x}}x}'
    'x{42}x'

    >>> '{:x{{x}}x}'.format(now)
    'x{x}x'

2. Formatted string literals currently seem inconsistent in the way they handle 
   handle doubled braces.

   In the base string doubling the braces escapes them.

    >>> f'x{{x}}x'
    'x{x}x'

   In the replacement expression doubling the braces escapes them.
    >>> f'{f"x{{x}}x"}'
    'x{x}x'

   In the format specifier doubling the braces does not escape them.
    >>> f'{now:x{{x}}x}'
    'x{42}x'

3. Currently there is no way I know of escape the braces in the format 
   specifier.

4. Allowing the braces to be escaped in the format specifier allows the user to 
   defer the interpretation of the of a format specifier so that it is evaluated 
   by a format function inside the object rather than being evaluated in the 
   current context.  That seems like a generally useful feature.
msg361834 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-02-11 22:03
But you can't just change it without breaking the code of anyone who's relying on the current behavior. If we could say "no one relies on that", that's would let us move forward with such a breaking change. But I don't think we can make that determination. And we're generally conservative with such breaking changes.

I think the current behavior doesn't make much sense, although I haven't completely analyzed the issue.
msg361835 - (view) Author: JitterMan (jitterman) Date: 2020-02-11 22:32
Okay, I get it. Someone might be using two braces in the format specifier because they found that it is a way to both evaluate a sub-expression and get braces in the formatted result. I was thinking that they would just use three braces, but that does not appear to work, though I cannot understand the resulting error message.

    >>> x = 42
    >>> import datetime
    >>> now = datetime.datetime.now()

    >>> f'{now:x{x}x}'
    'x42x'

    >>> f'{now:x{{x}}x}'
    'x{42}x'

    >>> f'{now:x{{{x}}}x}'
    Traceback (most recent call last):
    ...
    TypeError: unhashable type: 'set'

I think you are right. This particular ship may have already sailed away.
msg361933 - (view) Author: Vedran Čačić (veky) * Date: 2020-02-13 04:42
I can't help with the issue itself (though I must say that I philosophically don't believe in "sailed off ships" -- in the limit, Python must be the best it can be, though it can take decades to get there), but I can help you with the error message. It simply tries to take {{x}} as your v (expression to be formatted). That is, a set containing a set containing x. But since sets themselves are not hashable (being mutable), they cannot be contained in sets (that's why frozensets were invented).

>>> {{4}}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
msg361937 - (view) Author: JitterMan (jitterman) Date: 2020-02-13 06:21
Ah, that make sense. Thanks!
History
Date User Action Args
2020-02-13 06:21:00jittermansetmessages: + msg361937
2020-02-13 04:42:07vekysetnosy: + veky
messages: + msg361933
2020-02-11 22:32:05jittermansetmessages: + msg361835
2020-02-11 22:04:03eric.smithsetassignee: eric.smith
2020-02-11 22:03:57eric.smithsetmessages: + msg361834
2020-02-11 21:56:56jittermansetmessages: + msg361832
2020-02-11 17:47:09crwilcoxsetfiles: + test_case.patch
keywords: + patch
messages: + msg361826
2020-02-11 07:18:24eric.smithsetmessages: + msg361787
2020-02-11 04:34:50zach.waresetcomponents: - 2to3 (2.x to 3.x conversion tool)
2020-02-11 04:34:28zach.waresetnosy: + eric.smith
2020-02-11 01:29:50crwilcoxsetmessages: + msg361774
2020-02-10 23:44:44jittermansetmessages: + msg361751
2020-02-10 18:55:45crwilcoxsetnosy: + crwilcox
messages: + msg361707
2020-02-10 17:00:55zach.waresetnosy: + zach.ware
messages: + msg361704
2020-02-10 16:50:35jittermansetfiles: + bugreport.py

messages: + msg361703
2020-02-10 16:48:04jittermancreate