classification
Title: PEP 511: Add ast.Constant to allow AST optimizer to emit constants
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: 25843 Superseder:
Assigned To: Nosy List: benjamin.peterson, brett.cannon, haypo, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-01-18 17:04 by haypo, last changed 2016-01-27 11:20 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
constant.patch haypo, 2016-01-18 17:04 review
constant-2.patch haypo, 2016-01-22 13:05 review
constant-3.patch haypo, 2016-01-25 14:35 review
Messages (16)
msg258528 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-18 17:04
Currently, the Python parser emits ast.Tuple AST nodes which are compiled to multiple LOAD_xxx instructions followed a final BUILD_TUPLE instruction. The Python peephole optimizer detect LOAD_CONST x n + BUILD_TUPLE instructions to replace it directly with a tuple constant.

IHMO it's better to implement this optimization early at the AST level in an AST optimizer. The PEP 511 proposes to add a new ast.Constant AST node for that.

With this new AST node, the AST optimizer can emit tuple constants, but also any kind of constant like frozenset. For example, it's also possible to optimize "x in {1, 2}" to "x in frozenset({1, 2})" where frozenset({1, 2}) is a constant (don't call frozenset type at runtime). (Again, this optimization is already implemented in the peephole optimizer, it's just an example.)

Attached patch adds the new ast.Constant type but update also code consuming AST to handle this new kind of node:

* add the node: Parser/Python.asdl, Parser/asdl.py, Parser/asdl_c.py
* generated changes: Include/asdl.h, Include/Python-ast.h, Python/Python-ast.c
* accept the node kind: Python/symtable.c, Python/ast.c
* accept Constant instead of Num or Str: Lib/ast.py, Python/future.c, Tools/parser/unparse.py
* emit constants to bytecode: Python/compile.c

I didn't change the compiler to emit directly ast.Constant nodes to reduce the impact on the backward compatibility. This change can be done. An AST optimizer is responsible to convert NameConstant (False, True, None), Num (int, float, complex), Str, Bytes, Tuple to Constant. Example with fatoptimizer:
https://github.com/haypo/fatoptimizer/blob/2d794f511fe23ccde320725c6d12ce5ce8ffbdfe/fatoptimizer/convert_const.py

ast.Constant also simplifies AST optimizers: no need to check each time if a node is constant or not.

Adding a new kind of node was already proposed in the old issue #11549: the patch adds ast.Lit (it was proposed to rename it to ast.Literal).
msg258531 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-18 17:21
To support emiting constants from ast.Constant, we will also need the fix for the issue #25843. Currently, the compile merges constants (0, 0) and (0.0, 0.0) because they are equal, but item types are different.
msg258536 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-18 17:44
Would it make sense to tag the type of the constant in the node somehow?

We also make no backwards-compatibility guarantees about the AST, so if it simplifies things to switch entirely to Constant from Num, etc. then I said do it.
msg258543 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-18 20:20
Brett Cannon: "Would it make sense to tag the type of the constant in the node somehow?"

It's easy to get the type of a constant: type(node.value). I like using isinstance().

Brett Cannon: "We also make no backwards-compatibility guarantees about the AST, so if it simplifies things to switch entirely to Constant from Num, etc. then I said do it."

Oh, I forgot an important point from the PEP: "[ast.Constant] does not contain line number and column offset informations on tuple or frozenset items." I don't know if it's an issue or not.

I prefer to move step by step. As I wrote, we can decide to use directly ast.Constant later, even before the Python 3.6 release.
msg258621 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-19 21:30
First I also wanted to add ast.Literal to literals: list, set, dict, etc. But it doesn't work, we loose the item order: set and dict are unordered. An optimizer must not change the order in which items are created. At least, not by default.

I'm talking about an hypothetical ast.Literal type which would take a Python object (list, set, etc.) Current ast.Set contains an ordered list of items, ast.Dict uses two ordered lists for keys and values.
msg258759 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-21 17:02
> We also make no backwards-compatibility guarantees about the AST, so if it simplifies things to switch entirely to Constant from Num, etc. then I said do it.

Replacing Num/Str/... with Constant causes bug in markerlib (used by pip, it breaks indirectly the venv module of the stdlib) and Chameleon benchmark. I only found these projects by mistake. I'm quite sure that much more code rely on the AST "API" even if it's unstable.

I prefer to not break the API for free :-)
msg258799 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-22 13:05
Patch version 2:

- rework the whole patch
- add unit tests
- fix AST validation: the code was completly wrong in patch 1, I don't understand how it worked :-p Validate also correctly nested tuple and nested frozenset.
- add a comment to explain why obj2ast_constant() doesn't have to Py_INCREF() singletons
- handle also Ellipsis
- revert changes on set_context(). It seems like set_context() is only called with code emited directly by the compiler (not by compile(custom_ast_tree, ...)). If someone finds a way to call set_context() with an ast.Constant, we can support this case later.
msg258865 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-23 12:04
@Serhiy: Would you mind reviewing constant-2.patch? I reviewed my own patch and added some comments, I found a refleak ;-)
msg258869 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-23 13:16
New changeset 5de1bd759f3b by Victor Stinner in branch 'default':
Issue #26146: marshal.loads() now uses the empty frozenset singleton
https://hg.python.org/cpython/rev/5de1bd759f3b
msg258870 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-23 13:18
> New changeset 5de1bd759f3b by Victor Stinner in branch 'default':
> Issue #26146: marshal.loads() now uses the empty frozenset singleton
> https://hg.python.org/cpython/rev/5de1bd759f3b

This change is not directly related to this issue.

It's required by test_singleton_empty_frozenset() of test_set when an AST transformer replaces frozenset(), frozenset([]), etc. calls with an empty frozenset constant.
msg258901 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-25 14:35
Patch version 3: fix a reference leak in validate_constant().
msg258936 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-26 00:11
New changeset 27e5437f442c by Victor Stinner in branch 'default':
Add ast.Constant
https://hg.python.org/cpython/rev/27e5437f442c
msg258937 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-01-26 00:19
I pushed an enhanced version of constant-3.patch:

* fix the compiler: don't emit LOAD_CONST for ast.Constant if the value is a string (str) or a number (int, float, complex), as done previously for ast.Str and ast.Num

* add unit tests on: the compiler (ensure that LOAD_CONST is emitted for constants), ast.get_docstring() and ast.literal_eval()
msg258982 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-26 23:11
New changeset c5df914e73ad by Victor Stinner in branch 'default':
Fix a refleak in validate_constant()
https://hg.python.org/cpython/rev/c5df914e73ad
msg258983 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-26 23:39
New changeset 04444d0d0a05 by Victor Stinner in branch 'default':
Issue #26146: remove useless code
https://hg.python.org/cpython/rev/04444d0d0a05

New changeset 58086f0a953a by Victor Stinner in branch 'default':
Issue #26146: enhance ast.Constant error message
https://hg.python.org/cpython/rev/58086f0a953a
msg259017 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-27 11:20
New changeset 16f60cd918e0 by Victor Stinner in branch 'default':
PEP 511
https://hg.python.org/peps/rev/16f60cd918e0
History
Date User Action Args
2016-01-27 11:20:23python-devsetmessages: + msg259017
2016-01-26 23:39:49python-devsetmessages: + msg258983
2016-01-26 23:11:56python-devsetmessages: + msg258982
2016-01-26 00:19:15hayposetstatus: open -> closed
resolution: fixed
messages: + msg258937

components: + Interpreter Core
2016-01-26 00:11:42python-devsetmessages: + msg258936
2016-01-25 14:35:27hayposetfiles: + constant-3.patch

messages: + msg258901
2016-01-23 13:18:33hayposetmessages: + msg258870
2016-01-23 13:16:47python-devsetnosy: + python-dev
messages: + msg258869
2016-01-23 12:04:34hayposetmessages: + msg258865
2016-01-22 13:05:11hayposetfiles: + constant-2.patch

messages: + msg258799
2016-01-21 17:02:57hayposetmessages: + msg258759
2016-01-21 17:00:14hayposetdependencies: + code_richcompare() don't use constant type when comparing code constants
2016-01-19 21:30:08hayposetmessages: + msg258621
2016-01-18 20:20:45hayposetmessages: + msg258543
2016-01-18 17:44:22brett.cannonsetmessages: + msg258536
2016-01-18 17:21:37hayposetmessages: + msg258531
2016-01-18 17:04:51haypocreate