classification
Title: Expose ast.unparse in the ast module
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: pablogsal Nosy List: BTaskaya, Batuhan Taskaya, brandtbucher, brett.cannon, levkivskyi, miss-islington, pablogsal
Priority: normal Keywords: patch

Created on 2019-11-20 22:34 by pablogsal, last changed 2020-01-07 20:42 by vstinner.

Pull Requests
URL Status Linked Edit
PR 17302 merged pablogsal, 2019-11-20 22:44
PR 17376 merged pablogsal, 2019-11-25 11:29
PR 17377 open BTaskaya, 2019-11-25 14:25
PR 17612 merged BTaskaya, 2019-12-15 11:12
PR 17613 merged BTaskaya, 2019-12-15 12:25
PR 17687 merged pablogsal, 2019-12-23 16:18
PR 17738 merged pablogsal, 2019-12-29 18:48
PR 17739 merged pablogsal, 2019-12-29 19:48
PR 17760 open BTaskaya, 2019-12-30 20:37
PR 17797 open BTaskaya, 2020-01-02 17:21
PR 17798 merged BTaskaya, 2020-01-02 17:45
PR 17892 open BTaskaya, 2020-01-07 11:04
Messages (16)
msg357107 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-20 22:34
As discussed in https://mail.python.org/archives/list/python-dev@python.org/thread/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/ I propose to expose the unparse.py tool as part of the standard library in the ast module.

The exposed function will have the interface:

ast.unparse(ast_obj, *, option1, option2,...)

and will return a string with the unparsed version of ast_obj.

The unparse tool will need some cosmetic improvements that will be tracked separately in this issue.
msg357111 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-20 23:01
After PR17302 is merged we need to fix the following cosmetic issues indicated by Victor:

(*) unparse adds many useless parentheses. The algorithm seems naive.
For example, it adds "()" to "class _AddedDllDirectory():". It also
adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
whereas the AST node was at "top level": it isn't a sub-expression.
Maybe this algortihm should be made smarter.

(*) newlines in docstring are rendered as two characters: "\" + "n"
(escaped newline: \n), rather than a newline character. I would expect
a newline, it's more common that \n... But it may "break" inline
doctests rendering... Maybe it should be an option (render newlines as
newline character or escape them?), or maybe even let the user choose
how to render them using a callback (that's related to the "pluggable
formatter" question).

(*) Indentation is hardcoded to 4 spaces. What if I want 2 spaces or
something different? Should it become an option?

(*) Float infinity is replaces with 1e309. Again, maybe someone wants
to render this differently? It sounds like an arbitrary choice (which
"works" as expected).
msg357138 - (view) Author: Batuhan Taskaya (Batuhan Taskaya) Date: 2019-11-21 07:48
After PR 17302 is accepted, I'll work on refactorings including a precedence algorithm to find when to parentheses.
msg357176 - (view) Author: Batuhan (BTaskaya) * Date: 2019-11-21 15:08
@gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.
msg357258 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-22 09:21
> @gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.

Good catch, definitely do that!
msg357417 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-24 23:02
New changeset 27fc3b6f3fc49a36d3f962caac5c5495696d12ed by Pablo Galindo in branch 'master':
bpo-38870: Expose a function to unparse an ast object in the ast module (GH-17302)
https://github.com/python/cpython/commit/27fc3b6f3fc49a36d3f962caac5c5495696d12ed
msg357438 - (view) Author: miss-islington (miss-islington) Date: 2019-11-25 11:49
New changeset ded8888fbc33011dd39b7b1c86a5adfacc4943f3 by Miss Islington (bot) (Pablo Galindo) in branch 'master':
bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)
https://github.com/python/cpython/commit/ded8888fbc33011dd39b7b1c86a5adfacc4943f3
msg358478 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-16 12:27
New changeset a322f50c369e2e4138266c88e32ef83af95b2da6 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Remove dead code related with argument unparsing (GH-17613)
https://github.com/python/cpython/commit/a322f50c369e2e4138266c88e32ef83af95b2da6
msg358504 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-16 18:25
I created bpo-39069: "Move ast.unparse() function to a different module".
msg358824 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-23 16:11
New changeset 4b3b1226e86df6cd45e921c8f2ad23c3639c43b2 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Refactor delimiting with context managers in ast.unparse (GH-17612)
https://github.com/python/cpython/commit/4b3b1226e86df6cd45e921c8f2ad23c3639c43b2
msg358826 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-23 16:42
New changeset d69cbeb99d5fd0d5464e937202cca6a2024d1bcf by Pablo Galindo in branch 'master':
Revert "bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)" (GH-17687)
https://github.com/python/cpython/commit/d69cbeb99d5fd0d5464e937202cca6a2024d1bcf
msg359001 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-29 19:21
New changeset 23a226bf3ae7b462084e899d007d12d9fe398ac5 by Pablo Galindo in branch 'master':
bpo-38870: Run always tests that heavily use grammar features in test_unparse (GH-17738)
https://github.com/python/cpython/commit/23a226bf3ae7b462084e899d007d12d9fe398ac5
msg359207 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-02 18:20
New changeset 7b35bef9787cd80ed1e12124f759b4be03c849db by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Throw ValueError on invalid yield from usage (GH-17798)
https://github.com/python/cpython/commit/7b35bef9787cd80ed1e12124f759b4be03c849db
msg359500 - (view) Author: Batuhan (BTaskaya) * Date: 2020-01-07 10:27
ExtSlice nodes without second value doesn't roundtrip properly

source: x[1:2,]
         Expr(
             value=Subscript(
                 value=Name(id='x', ctx=Load()),
-                slice=ExtSlice(
-                    dims=[
-                        Slice(
-                            lower=Constant(value=1, kind=None),
-                            upper=Constant(value=2, kind=None),
-                            step=None)]),
+                slice=Slice(
+                    lower=Constant(value=1, kind=None),
+                    upper=Constant(value=2, kind=None),
+                    step=None),
                 ctx=Load()))],
     type_ignores=[])

(I have a fix for unifying both tuple, constant tuple and extslice dims traversing)
msg359502 - (view) Author: Batuhan (BTaskaya) * Date: 2020-01-07 11:25
We might need to tweak the documentation @pablogsal, 

> Unparse an ast.AST object and generate a string with code that would produce an equivalent ast.AST object if parsed back with ast.parse().

If I interpret `equivalent` correctly, this explanation is false under cases like constant nodes has an immutable container value (which they can). 

>>> def wrap(expr):
...     return ast.Module(body=[ast.Expr(expr)], type_ignores=[])
... 
>>> constant_tuple = wrap(ast.Constant(value=(1, 2, 3), kind=None))
>>> normalpy_tuple = ast.parse("(1, 2, 3)")
>>> ast.unparse(constant_tuple) == ast.unparse(normalpy_tuple)
True
>>> ast.dump(ast.parse(ast.unparse(constant_tuple))) != ast.dump(constant_tuple)
True
>>> ast.dump(ast.parse(ast.unparse(constant_tuple))) == ast.dump(normalpy_tuple)
True

This isn't a bug in the code because there is no way we can generate a string that can produce a constant tuple. But this makes the docstring false, at least in certain cases.
msg359508 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-07 12:45
>  We might need to tweak the documentation @pablogsal, 

Maybe we can say that is 'as close as possible' if you pass an arbitrary AST object (if only happens with Constants we could documment exactly that). Let me think about this and I will make PR updating the docs.
History
Date User Action Args
2020-01-07 20:42:16vstinnersetnosy: - vstinner
2020-01-07 12:45:47pablogsalsetmessages: + msg359508
2020-01-07 11:25:14BTaskayasetmessages: + msg359502
2020-01-07 11:04:43BTaskayasetpull_requests: + pull_request17302
2020-01-07 10:27:23BTaskayasetmessages: + msg359500
2020-01-02 18:20:33pablogsalsetmessages: + msg359207
2020-01-02 17:45:55BTaskayasetpull_requests: + pull_request17233
2020-01-02 17:21:52BTaskayasetpull_requests: + pull_request17232
2019-12-30 20:37:27BTaskayasetpull_requests: + pull_request17196
2019-12-29 19:48:20pablogsalsetpull_requests: + pull_request17182
2019-12-29 19:21:06pablogsalsetmessages: + msg359001
2019-12-29 18:48:37pablogsalsetpull_requests: + pull_request17181
2019-12-23 16:42:54pablogsalsetmessages: + msg358826
2019-12-23 16:30:10gvanrossumsetnosy: - gvanrossum
2019-12-23 16:18:18pablogsalsetpull_requests: + pull_request17143
2019-12-23 16:11:07pablogsalsetmessages: + msg358824
2019-12-16 18:25:26vstinnersetnosy: + vstinner
messages: + msg358504
2019-12-16 12:27:03pablogsalsetmessages: + msg358478
2019-12-15 12:25:28BTaskayasetpull_requests: + pull_request17085
2019-12-15 11:12:47BTaskayasetpull_requests: + pull_request17084
2019-11-25 14:25:53BTaskayasetpull_requests: + pull_request16859
2019-11-25 11:49:21miss-islingtonsetnosy: + miss-islington
messages: + msg357438
2019-11-25 11:29:23pablogsalsetpull_requests: + pull_request16858
2019-11-24 23:02:46pablogsalsetmessages: + msg357417
2019-11-22 09:21:49gvanrossumsetmessages: + msg357258
2019-11-21 21:00:42brett.cannonsetnosy: + brett.cannon
2019-11-21 15:08:05BTaskayasetnosy: + gvanrossum
messages: + msg357176
2019-11-21 07:48:21Batuhan Taskayasetnosy: + Batuhan Taskaya
messages: + msg357138
2019-11-20 23:01:49pablogsalsetmessages: + msg357111
2019-11-20 22:55:42levkivskyisetnosy: + levkivskyi
2019-11-20 22:55:23brandtbuchersetnosy: + brandtbucher
2019-11-20 22:44:42pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request16794
2019-11-20 22:34:29pablogsalcreate