classification
Title: Move constant folding to AST level
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: A constant folding optimization pass for the AST
View: 1346238
Assigned To: Nosy List: abarnert, benjamin.peterson, brett.cannon, georg.brandl, ncoghlan, serhiy.storchaka, vstinner, yselivanov
Priority: normal Keywords:

Created on 2016-02-05 17:34 by serhiy.storchaka, last changed 2016-02-07 18:10 by serhiy.storchaka. This issue is now closed.

Messages (6)
msg259680 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-05 17:34
For using more efficient bytecode (either specialized 8-bit opcodes or 16-bit opcodes) we need to move some optimizations from bytecode level to AST level, since LOAD_CONST variants could have variable size. Now with the Constant node this should be easy.
msg259692 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-05 22:12
*If* the PEP 511 is accepted, it will be super easy to start with an
implementation in pure Python.

But I think that we should benchmark the overhead of the Python API of
PEP 511, because we need to convert all AST internal objects to Python
objects, run the AST optimizer, and then again convert Python objects
to internal AST objects.

For a long running process, the time to compile a .py file doesn't
matter. For a short script, it matters. At least, we need to compile
the script itself.

By the way, would it be insane to *not* optimize the script when
running "python script.py"?

> Now with the Constant node this should be easy.

ast.Constant is *not* emited by the compiler to not break backward
compatibility. I *know* that there is no stable API on AST, but I
noticed some issues when working on my AST project. For example, pip
doesn't work because an internal library uses AST and the code doesn't
handle ast.Constant (it's probably super easy to fix it).

I'm open to change the compiler to emit ast.Constant directly, but
maybe only in "optimized mode" (ex: python -O?). ast.Constant for
tuples and frozensets has a limitation: it doesn't store the location
of items (line number, column offset).
msg259694 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-02-05 22:35
> ast.Constant is *not* emited by the compiler to not break backward
compatibility. I *know* that there is no stable API on AST, but I
noticed some issues when working on my AST project. For example, pip
doesn't work because an internal library uses AST and the code doesn't
handle ast.Constant (it's probably super easy to fix it).

Yeah, it's probably very easy to fix.  And if pip stops working on 3.6-alpha it will be fixed.  We don't give any guarantees on AST backwards compatibility.  In fact, we even tell that the AST might change in every release in the docs "The abstract syntax itself might change with each Python release".


> I'm open to change the compiler to emit ast.Constant directly, but
maybe only in "optimized mode" (ex: python -O?). 

I'm -1 on using -O flag since some people would prefer to have optimizations *and* have their assert statements.

In fact I'm -1 on using any kind of flag to trigger optimizations on or off.  Those flags will only drive confusion.  If the optimizations are any good, people will simply start using the flag all the time.
msg259696 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-05 22:40
Yury Selivanov added the comment:
> In fact I'm -1 on using any kind of flag to trigger optimizations on or off.  Those flags will only drive confusion.  If the optimizations are any good, people will simply start using the flag all the time.

Yeah, Guido calls this a missed optimization opportunity.

My only concern is:

> ast.Constant for tuples and frozensets has a limitation: it doesn't store the location of items (line number, column offset).

Is it a problem in practice? I didn't have time to investigate this.

Since the PEP 511 is still under discussion, I prefer to focus on the
PEP itself than this specific implementation detail ;-)
msg259699 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-05 23:01
> we need to move some optimizations from bytecode level to AST level

You may be aware that there is a long list of similar (old) issues proposing something similar:
https://www.python.org/dev/peps/pep-0511/#usage-1-ast-optimizer

My PEP 511 tries to propose a generic API to support all proposed optimizations.

--

I didn't try yet to do something like importlib to compile an AST optimizer written in Python into a frozen module and then use it in Python. It would allow to implement an AST optimizer in pure Python to ease its maintenance *and* have it builtin into CPython, rather than having to play with AST in C (less fun).

My plan for the PEP 511 is more to start to experiment multiple AST optimizers and bytecode optimizers outside CPython, wait until the code becomes more stable, and later discuss if one optimizer deserves to become part of Python stdlib.

"Writing an optimizer or a preprocessor is out of the scope of this PEP."

--

But maybe you would prefer to have a single and simple AST optimizer builtin Python without new additional API?

I don't want to block your effort on experimentation :-) I just see a nice opportunity to sell you the PEP 511 ;-)
msg259795 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-07 18:10
Issue1346238 has a patch.
History
Date User Action Args
2016-02-07 18:10:55serhiy.storchakasetstatus: open -> closed
superseder: A constant folding optimization pass for the AST
messages: + msg259795

resolution: duplicate
stage: needs patch -> resolved
2016-02-05 23:01:49vstinnersetmessages: + msg259699
2016-02-05 22:40:22vstinnersetmessages: + msg259696
2016-02-05 22:35:33yselivanovsetmessages: + msg259694
2016-02-05 22:12:05vstinnersetmessages: + msg259692
2016-02-05 17:34:07serhiy.storchakacreate