classification
Title: Lazy Literal String Interpolation (PEP-498-based fl-strings)
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: arcivanov, barry, eric.smith, mbdevpl, serhiy.storchaka
Priority: normal Keywords:

Created on 2018-02-26 10:19 by arcivanov, last changed 2019-12-18 06:25 by mbdevpl.

Messages (9)
msg312909 - (view) Author: Arcadiy Ivanov (arcivanov) Date: 2018-02-26 10:19
I'd like to start a discussion on/gauge interest for introducing an enhancement to PEP-498 in a form of delayed/lazy/lambda f-string.

The proposed change is, conceptually, minor and would not differ from PEP-498 syntactically at all except for string prefix.

E.g. 

     extra = f'{extra},waiters:{len(self._waiters)}'

becomes

     extra = fl'{extra},waiters:{len(self._waiters)}'

The proposal would produce a lambda-like object x('format'), where x.__str__() == f'format'.

This should come extremely useful in all cases where delayed or conditional string formatting and concatenation are desired, such as in cases of logging.

As an example, currently, the 

    logger.debug("Format %s string", value) 

cannot be used with an f-string as follows 

    logger.debug(f"Format {value} string") 

without an unconditional evaluation of all parameters due to current compilation prior to logging checking whether it's even enabled for debug:

>>> b = 1
>>> def a(x):
...     return f"Foo {x} bar {b}"
... 
>>> dis.dis(a)
  2           0 LOAD_CONST               1 ('Foo ')
              2 LOAD_FAST                0 (x)
              4 FORMAT_VALUE             0
              6 LOAD_CONST               2 (' bar ')
              8 LOAD_GLOBAL              0 (b)
             10 FORMAT_VALUE             0
             12 BUILD_STRING             4
             14 RETURN_VALUE

Additional great optimizations may be rendered by introducing an a fl-string is a case where

    foo = "foo"
    s1 = fl"S1 value ${foo}"
    s2 = fl"S2 value of ${s1}"
    print(s2)

may produce only one BUILD_STRING instruction, potentially dramatically increasing performance in heavy string-concat based applications. 

Even when a compiler will not be able to statically prove that a particular value is in fact an fl-string, an interpreter-level check or a JIT-based Python implementation may be able to optimize such concats ad-nauseam (with trap-based deopt), allowing to collapse an extremely long chains of formats into a single BUILD_STRING op without any intermediary string allocation/concatenation (very useful in cases such as web servers, templating engines etc).

I'll be happy to produce patches against 3.7/3.8 for this if a general concept is found useful/desirable by the community.
msg312910 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-26 10:27
class FL:
    def __init__(self, func):
        self.func = func
    def __str__(self):
        return self.func()

extra = FL(lambda: f'{extra},waiters:{len(self._waiters)}')
msg312911 - (view) Author: Arcadiy Ivanov (arcivanov) Date: 2018-02-26 10:42
As an example this is the current state of affairs:

>>> def x():
...     foo = "foo"
...     s1 = f"S1 value {foo}"
...     s2 = f"S2 value {s1}"
...     print(s2)
... 
>>> dis.dis(x)
  2           0 LOAD_CONST               1 ('foo')
              2 STORE_FAST               0 (foo)

  3           4 LOAD_CONST               2 ('S1 value ')
              6 LOAD_FAST                0 (foo)
              8 FORMAT_VALUE             0
             10 BUILD_STRING             2
             12 STORE_FAST               1 (s1)

  4          14 LOAD_CONST               3 ('S2 value ')
             16 LOAD_FAST                1 (s1)
             18 FORMAT_VALUE             0
             20 BUILD_STRING             2
             22 STORE_FAST               2 (s2)

  5          24 LOAD_GLOBAL              0 (print)
             26 LOAD_FAST                2 (s2)
             28 CALL_FUNCTION            1
             30 POP_TOP
             32 LOAD_CONST               0 (None)
             34 RETURN_VALUE

whereas an fl-string representation of:

>>> def x():
...     foo = "foo"
...     s1 = fl"S1 value {foo}"
...     s2 = fl"S2 value {s1}"
...     print(s2)
... 

would behave close to:

>>> def x():
...     foo = "foo"
...     s1 = lambda: f"S1 value {foo}"
...     s2 = lambda: f"S2 value {s1()}"
...     print(s2())
... 
>>> dis.dis(x)
  2           0 LOAD_CONST               1 ('foo')
              2 STORE_DEREF              0 (foo)

  3           4 LOAD_CLOSURE             0 (foo)
              6 BUILD_TUPLE              1
              8 LOAD_CONST               2 (<code object <lambda> at 0x7ff2ef1abd20, file "<stdin>", line 3>)
             10 LOAD_CONST               3 ('x.<locals>.<lambda>')
             12 MAKE_FUNCTION            8
             14 STORE_DEREF              1 (s1)

  4          16 LOAD_CLOSURE             1 (s1)
             18 BUILD_TUPLE              1
             20 LOAD_CONST               4 (<code object <lambda> at 0x7ff2ef1abae0, file "<stdin>", line 4>)
             22 LOAD_CONST               3 ('x.<locals>.<lambda>')
             24 MAKE_FUNCTION            8
             26 STORE_FAST               0 (s2)

  5          28 LOAD_GLOBAL              0 (print)
             30 LOAD_FAST                0 (s2)
             32 CALL_FUNCTION            0
             34 CALL_FUNCTION            1
             36 POP_TOP
             38 LOAD_CONST               0 (None)
             40 RETURN_VALUE

Where LOAD_CONST, LOAD_CONST, MAKE_FUNCTION sequences are replaced with BUILD_LAZY_STRING (name's provisional) producing the same lambda-like formatter function that cooperates with FORMAT_VALUE and BUILD_STRING/BUILD_LAZY_STRING ops.
msg312912 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-02-26 10:51
See also PEP 501.
msg312913 - (view) Author: Arcadiy Ivanov (arcivanov) Date: 2018-02-26 10:54
@serhiy.storchaka Of course a similar pattern can be implemented via a class (or even without one as I've shown below). 

But you can clearly notice that in your example:

1) There are tons of boilerplate (as in mine with lambdas).
2) It's going to be slower than a core implementation with specialized objects.
3) It can't be made to cooperate with a FORMAT_VALUE/BUILD_STRING - intermediate strings will still be produced.
msg312914 - (view) Author: Arcadiy Ivanov (arcivanov) Date: 2018-02-26 10:59
@eric.smith

Thanks! I was looking for such a general-purpose proposal but could not find it. 

Although general-purpose mechanism that would allow pluggable constructs like `sh`, `html`, `sql` and the like is awesome and very desirable (especially sh in preference of Popen madness), string formatting/concatenation is IMO a fundamental concept (hence PEP-498 is in core with specialized opcodes instead of being deferred to be a general purpose pluggable PEP-501 implementation).
msg312918 - (view) Author: Arcadiy Ivanov (arcivanov) Date: 2018-02-26 11:24
> Although general-purpose mechanism that would allow pluggable constructs like `sh`, `html`, `sql`

Strike that one, I didn't read into PEP-0501 deep enough before replying. 

Yes, `i"format"` is what I'm talking about.
msg312922 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-02-26 12:20
Arcadiy: Somehow you're dropping Serhiy and me from the nosy list. I've re-added us.
msg312923 - (view) Author: Arcadiy Ivanov (arcivanov) Date: 2018-02-26 12:46
Sorry about that! I'll be sure to refresh next time before posting a reply.
History
Date User Action Args
2019-12-18 06:25:31mbdevplsetnosy: + mbdevpl
2018-02-26 16:10:28barrysetnosy: + barry
2018-02-26 12:46:52arcivanovsetmessages: + msg312923
2018-02-26 12:20:52eric.smithsetnosy: + eric.smith, serhiy.storchaka
messages: + msg312922
2018-02-26 11:24:30arcivanovsetmessages: + msg312918
2018-02-26 10:59:46arcivanovsetmessages: + msg312914
2018-02-26 10:54:18arcivanovsetnosy: - eric.smith, serhiy.storchaka
messages: + msg312913
2018-02-26 10:51:24eric.smithsetnosy: + eric.smith, serhiy.storchaka
messages: + msg312912
2018-02-26 10:42:52arcivanovsetnosy: - serhiy.storchaka
messages: + msg312911
components: + Interpreter Core, - Library (Lib)
2018-02-26 10:27:49serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg312910
2018-02-26 10:19:19arcivanovcreate