classification
Title: Expose ast.unparse in the ast module
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pablogsal Nosy List: 1v3m, BTaskaya, Batuhan Taskaya, brandtbucher, levkivskyi, miss-islington, pablogsal
Priority: normal Keywords: patch

Created on 2019-11-20 22:34 by pablogsal, last changed 2020-10-23 18:24 by BTaskaya. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17302 merged pablogsal, 2019-11-20 22:44
PR 17376 merged pablogsal, 2019-11-25 11:29
PR 17377 merged BTaskaya, 2019-11-25 14:25
PR 17612 merged BTaskaya, 2019-12-15 11:12
PR 17613 merged BTaskaya, 2019-12-15 12:25
PR 17687 merged pablogsal, 2019-12-23 16:18
PR 17738 merged pablogsal, 2019-12-29 18:48
PR 17739 merged pablogsal, 2019-12-29 19:48
PR 17760 merged BTaskaya, 2019-12-30 20:37
PR 17797 merged BTaskaya, 2020-01-02 17:21
PR 17798 merged BTaskaya, 2020-01-02 17:45
PR 17892 merged BTaskaya, 2020-01-07 11:04
PR 18768 merged BTaskaya, 2020-03-03 16:00
PR 19016 merged BTaskaya, 2020-03-15 12:51
PR 19636 merged BTaskaya, 2020-04-21 22:16
PR 20133 merged BTaskaya, 2020-05-16 20:07
PR 20134 merged BTaskaya, 2020-05-16 20:51
PR 20141 merged pablogsal, 2020-05-17 00:05
PR 20152 merged BTaskaya, 2020-05-17 15:18
PR 20166 merged 1v3m, 2020-05-18 00:02
PR 20240 merged BTaskaya, 2020-05-19 21:53
PR 20244 merged miss-islington, 2020-05-19 22:14
PR 21053 merged BTaskaya, 2020-06-22 10:41
PR 21191 merged pablogsal, 2020-06-28 01:12
Messages (33)
msg357107 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-20 22:34
As discussed in https://mail.python.org/archives/list/python-dev@python.org/thread/JAQDBMC23HW2PQ27HQNJ7G244T423IDD/ I propose to expose the unparse.py tool as part of the standard library in the ast module.

The exposed function will have the interface:

ast.unparse(ast_obj, *, option1, option2,...)

and will return a string with the unparsed version of ast_obj.

The unparse tool will need some cosmetic improvements that will be tracked separately in this issue.
msg357111 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-20 23:01
After PR17302 is merged we need to fix the following cosmetic issues indicated by Victor:

(*) unparse adds many useless parentheses. The algorithm seems naive.
For example, it adds "()" to "class _AddedDllDirectory():". It also
adds parenthesis around yield, like "(yield (top, dirs, nondirs))",
whereas the AST node was at "top level": it isn't a sub-expression.
Maybe this algortihm should be made smarter.

(*) newlines in docstring are rendered as two characters: "\" + "n"
(escaped newline: \n), rather than a newline character. I would expect
a newline, it's more common that \n... But it may "break" inline
doctests rendering... Maybe it should be an option (render newlines as
newline character or escape them?), or maybe even let the user choose
how to render them using a callback (that's related to the "pluggable
formatter" question).

(*) Indentation is hardcoded to 4 spaces. What if I want 2 spaces or
something different? Should it become an option?

(*) Float infinity is replaces with 1e309. Again, maybe someone wants
to render this differently? It sounds like an arbitrary choice (which
"works" as expected).
msg357138 - (view) Author: Batuhan Taskaya (Batuhan Taskaya) Date: 2019-11-21 07:48
After PR 17302 is accepted, I'll work on refactorings including a precedence algorithm to find when to parentheses.
msg357176 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2019-11-21 15:08
@gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.
msg357258 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-22 09:21
> @gvanrossum are you OK with adding type comments support? Current version loses type comment information so if typed_ast parses this, they wont be the same in AST representation.

Good catch, definitely do that!
msg357417 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-11-24 23:02
New changeset 27fc3b6f3fc49a36d3f962caac5c5495696d12ed by Pablo Galindo in branch 'master':
bpo-38870: Expose a function to unparse an ast object in the ast module (GH-17302)
https://github.com/python/cpython/commit/27fc3b6f3fc49a36d3f962caac5c5495696d12ed
msg357438 - (view) Author: miss-islington (miss-islington) Date: 2019-11-25 11:49
New changeset ded8888fbc33011dd39b7b1c86a5adfacc4943f3 by Miss Islington (bot) (Pablo Galindo) in branch 'master':
bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)
https://github.com/python/cpython/commit/ded8888fbc33011dd39b7b1c86a5adfacc4943f3
msg358478 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-16 12:27
New changeset a322f50c369e2e4138266c88e32ef83af95b2da6 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Remove dead code related with argument unparsing (GH-17613)
https://github.com/python/cpython/commit/a322f50c369e2e4138266c88e32ef83af95b2da6
msg358504 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-16 18:25
I created bpo-39069: "Move ast.unparse() function to a different module".
msg358824 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-23 16:11
New changeset 4b3b1226e86df6cd45e921c8f2ad23c3639c43b2 by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Refactor delimiting with context managers in ast.unparse (GH-17612)
https://github.com/python/cpython/commit/4b3b1226e86df6cd45e921c8f2ad23c3639c43b2
msg358826 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-23 16:42
New changeset d69cbeb99d5fd0d5464e937202cca6a2024d1bcf by Pablo Galindo in branch 'master':
Revert "bpo-38870: Remove dependency on contextlib to avoid performance regression on import (GH-17376)" (GH-17687)
https://github.com/python/cpython/commit/d69cbeb99d5fd0d5464e937202cca6a2024d1bcf
msg359001 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-29 19:21
New changeset 23a226bf3ae7b462084e899d007d12d9fe398ac5 by Pablo Galindo in branch 'master':
bpo-38870: Run always tests that heavily use grammar features in test_unparse (GH-17738)
https://github.com/python/cpython/commit/23a226bf3ae7b462084e899d007d12d9fe398ac5
msg359207 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-02 18:20
New changeset 7b35bef9787cd80ed1e12124f759b4be03c849db by Pablo Galindo (Batuhan Taşkaya) in branch 'master':
bpo-38870: Throw ValueError on invalid yield from usage (GH-17798)
https://github.com/python/cpython/commit/7b35bef9787cd80ed1e12124f759b4be03c849db
msg359500 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-01-07 10:27
ExtSlice nodes without second value doesn't roundtrip properly

source: x[1:2,]
         Expr(
             value=Subscript(
                 value=Name(id='x', ctx=Load()),
-                slice=ExtSlice(
-                    dims=[
-                        Slice(
-                            lower=Constant(value=1, kind=None),
-                            upper=Constant(value=2, kind=None),
-                            step=None)]),
+                slice=Slice(
+                    lower=Constant(value=1, kind=None),
+                    upper=Constant(value=2, kind=None),
+                    step=None),
                 ctx=Load()))],
     type_ignores=[])

(I have a fix for unifying both tuple, constant tuple and extslice dims traversing)
msg359502 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-01-07 11:25
We might need to tweak the documentation @pablogsal, 

> Unparse an ast.AST object and generate a string with code that would produce an equivalent ast.AST object if parsed back with ast.parse().

If I interpret `equivalent` correctly, this explanation is false under cases like constant nodes has an immutable container value (which they can). 

>>> def wrap(expr):
...     return ast.Module(body=[ast.Expr(expr)], type_ignores=[])
... 
>>> constant_tuple = wrap(ast.Constant(value=(1, 2, 3), kind=None))
>>> normalpy_tuple = ast.parse("(1, 2, 3)")
>>> ast.unparse(constant_tuple) == ast.unparse(normalpy_tuple)
True
>>> ast.dump(ast.parse(ast.unparse(constant_tuple))) != ast.dump(constant_tuple)
True
>>> ast.dump(ast.parse(ast.unparse(constant_tuple))) == ast.dump(normalpy_tuple)
True

This isn't a bug in the code because there is no way we can generate a string that can produce a constant tuple. But this makes the docstring false, at least in certain cases.
msg359508 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-07 12:45
>  We might need to tweak the documentation @pablogsal, 

Maybe we can say that is 'as close as possible' if you pass an arbitrary AST object (if only happens with Constants we could documment exactly that). Let me think about this and I will make PR updating the docs.
msg363087 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-03-01 20:12
New changeset 397b96f6d7a89f778ebc0591e32216a8183fe667 by Batuhan Taşkaya in branch 'master':
bpo-38870: Implement a precedence algorithm in ast.unparse (GH-17377)
https://github.com/python/cpython/commit/397b96f6d7a89f778ebc0591e32216a8183fe667
msg363196 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-03-02 18:59
New changeset 89aa4694fc8c6d190325ef8ed6ce6a6b8efb3e50 by Batuhan Taşkaya in branch 'master':
bpo-38870: Add docstring support to ast.unparse (GH-17760)
https://github.com/python/cpython/commit/89aa4694fc8c6d190325ef8ed6ce6a6b8efb3e50
msg363768 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-03-09 20:27
New changeset e7cab7f780ac253999512ee86374fc3454342811 by Batuhan Taşkaya in branch 'master':
bpo-38870: Simplify sequence interleaves in ast.unparse (GH-17892)
https://github.com/python/cpython/commit/e7cab7f780ac253999512ee86374fc3454342811
msg364257 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-03-15 19:57
New changeset 5b66ec166b81c8a77286da2c0d17be3579c3069a by Batuhan Taşkaya in branch 'master':
bpo-38870: Implement support for ast.FunctionType in ast.unparse (GH-19016)
https://github.com/python/cpython/commit/5b66ec166b81c8a77286da2c0d17be3579c3069a
msg367985 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-03 17:11
New changeset 493bf1cc316b0b5bd90779ecd1132878c881669e by Batuhan Taskaya in branch 'master':
bpo-38870: Don't start generated output with newlines in ast.unparse (GH-19636)
https://github.com/python/cpython/commit/493bf1cc316b0b5bd90779ecd1132878c881669e
msg369069 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-16 21:46
New changeset ce4a753dcb3eef3d68e892a6515490b1aa219651 by Batuhan Taskaya in branch 'master':
bpo-38870: Do not separate factor prefixes in ast.unparse (GH-20133)
https://github.com/python/cpython/commit/ce4a753dcb3eef3d68e892a6515490b1aa219651
msg369070 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-16 21:53
New changeset 25160cdc4775a1ddb4e37c8bf5a6e31ad9c146ed by Batuhan Taskaya in branch 'master':
bpo-38870: Don't put unnecessary parentheses on class declarations in ast.parse (GH-20134)
https://github.com/python/cpython/commit/25160cdc4775a1ddb4e37c8bf5a6e31ad9c146ed
msg369076 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-16 22:49
New changeset e966af7cff78e14e1d289db587433504b4b53533 by Batuhan Taskaya in branch 'master':
bpo-38870: Correctly handle empty docstrings in ast.unparse (GH-18768)
https://github.com/python/cpython/commit/e966af7cff78e14e1d289db587433504b4b53533
msg369078 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-16 23:04
New changeset dff92bb31f7db1a80ac431811f8108bd0ef9be43 by Batuhan Taskaya in branch 'master':
bpo-38870: Implement round tripping support for typed AST in ast.unparse (GH-17797)
https://github.com/python/cpython/commit/dff92bb31f7db1a80ac431811f8108bd0ef9be43
msg369088 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-17 02:54
New changeset 6341fc7257d89d798675ad6e425f7eb0b6f2b4bb by Pablo Galindo in branch 'master':
bpo-38870: Use subTest in test_unparse for better error reporting (GH-20141)
https://github.com/python/cpython/commit/6341fc7257d89d798675ad6e425f7eb0b6f2b4bb
msg369286 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-18 18:41
New changeset d71a6492dbd5434dfa6a0ad95e3ad98aa690887a by CyberSaxosTiGER in branch 'master':
bpo-38870: correctly escape unprintable characters on ast.unparse (GH-20166)
https://github.com/python/cpython/commit/d71a6492dbd5434dfa6a0ad95e3ad98aa690887a
msg369289 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-18 20:48
New changeset c102a148256b00b7d48c51a1a97df19042e603de by Batuhan Taskaya in branch 'master':
bpo-38870: Don't omit parenthesis when unparsing a slice in ast.unparse
https://github.com/python/cpython/commit/c102a148256b00b7d48c51a1a97df19042e603de
msg369398 - (view) Author: miss-islington (miss-islington) Date: 2020-05-19 22:14
New changeset dd74b6fde31aff9aa46c4fc2a830c569764e1b63 by Batuhan Taskaya in branch 'master':
bpo-38870: invalid escape sequence (GH-20240)
https://github.com/python/cpython/commit/dd74b6fde31aff9aa46c4fc2a830c569764e1b63
msg369402 - (view) Author: miss-islington (miss-islington) Date: 2020-05-19 22:33
New changeset 059279d8706074489144f6ba8ccc8723e0b85007 by Miss Islington (bot) in branch '3.9':
bpo-38870: invalid escape sequence (GH-20240)
https://github.com/python/cpython/commit/059279d8706074489144f6ba8ccc8723e0b85007
msg372488 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-28 01:11
New changeset 8df1016e2ef8c0a9f4d15bf7894c284295c99d9f by Batuhan Taskaya in branch 'master':
bpo-38870: Extend subject of ast.unparse warnings (GH-21053)
https://github.com/python/cpython/commit/8df1016e2ef8c0a9f4d15bf7894c284295c99d9f
msg372490 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-28 01:22
New changeset 6e39999a246a019a356e1befcdfd3d54a52cb7b5 by Pablo Galindo in branch '3.9':
[3.9] bpo-38870: Extend subject of ast.unparse warnings (GH-21053) (GH-21191)
https://github.com/python/cpython/commit/6e39999a246a019a356e1befcdfd3d54a52cb7b5
msg379462 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-10-23 18:21
Well, I'm happy to say that 3.9.0 is finally released with the ast.unparse interface. After tens of PRs, I think it is time for us to move on. For all future bugs, please create a new issue and nosy me. Thanks everyone who has participated this journey!
History
Date User Action Args
2020-10-23 18:24:39BTaskayasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-10-23 18:21:36BTaskayasetmessages: + msg379462
2020-06-28 01:22:33pablogsalsetmessages: + msg372490
2020-06-28 01:12:59pablogsalsetpull_requests: + pull_request20345
2020-06-28 01:11:46pablogsalsetmessages: + msg372488
2020-06-22 10:41:23BTaskayasetpull_requests: + pull_request20224
2020-05-20 22:32:37brett.cannonsetnosy: - brett.cannon
2020-05-19 22:33:51miss-islingtonsetmessages: + msg369402
2020-05-19 22:14:27miss-islingtonsetpull_requests: + pull_request19532
2020-05-19 22:14:20miss-islingtonsetmessages: + msg369398
2020-05-19 21:53:50BTaskayasetpull_requests: + pull_request19528
2020-05-18 20:48:56pablogsalsetmessages: + msg369289
2020-05-18 18:41:47pablogsalsetmessages: + msg369286
2020-05-18 00:02:071v3msetnosy: + 1v3m
pull_requests: + pull_request19465
2020-05-17 15:18:44BTaskayasetpull_requests: + pull_request19455
2020-05-17 02:54:01pablogsalsetmessages: + msg369088
2020-05-17 00:05:10pablogsalsetpull_requests: + pull_request19446
2020-05-16 23:04:18pablogsalsetmessages: + msg369078
2020-05-16 22:49:11pablogsalsetmessages: + msg369076
2020-05-16 21:53:29pablogsalsetmessages: + msg369070
2020-05-16 21:46:15pablogsalsetmessages: + msg369069
2020-05-16 20:51:35BTaskayasetpull_requests: + pull_request19439
2020-05-16 20:07:08BTaskayasetpull_requests: + pull_request19438
2020-05-03 17:11:59pablogsalsetmessages: + msg367985
2020-04-21 22:16:14BTaskayasetpull_requests: + pull_request18961
2020-03-15 19:57:00pablogsalsetmessages: + msg364257
2020-03-15 12:51:13BTaskayasetpull_requests: + pull_request18365
2020-03-09 20:27:11pablogsalsetmessages: + msg363768
2020-03-03 16:00:19BTaskayasetpull_requests: + pull_request18126
2020-03-02 18:59:25pablogsalsetmessages: + msg363196
2020-03-01 20:12:20pablogsalsetmessages: + msg363087
2020-01-07 20:42:16vstinnersetnosy: - vstinner
2020-01-07 12:45:47pablogsalsetmessages: + msg359508
2020-01-07 11:25:14BTaskayasetmessages: + msg359502
2020-01-07 11:04:43BTaskayasetpull_requests: + pull_request17302
2020-01-07 10:27:23BTaskayasetmessages: + msg359500
2020-01-02 18:20:33pablogsalsetmessages: + msg359207
2020-01-02 17:45:55BTaskayasetpull_requests: + pull_request17233
2020-01-02 17:21:52BTaskayasetpull_requests: + pull_request17232
2019-12-30 20:37:27BTaskayasetpull_requests: + pull_request17196
2019-12-29 19:48:20pablogsalsetpull_requests: + pull_request17182
2019-12-29 19:21:06pablogsalsetmessages: + msg359001
2019-12-29 18:48:37pablogsalsetpull_requests: + pull_request17181
2019-12-23 16:42:54pablogsalsetmessages: + msg358826
2019-12-23 16:30:10gvanrossumsetnosy: - gvanrossum
2019-12-23 16:18:18pablogsalsetpull_requests: + pull_request17143
2019-12-23 16:11:07pablogsalsetmessages: + msg358824
2019-12-16 18:25:26vstinnersetnosy: + vstinner
messages: + msg358504
2019-12-16 12:27:03pablogsalsetmessages: + msg358478
2019-12-15 12:25:28BTaskayasetpull_requests: + pull_request17085
2019-12-15 11:12:47BTaskayasetpull_requests: + pull_request17084
2019-11-25 14:25:53BTaskayasetpull_requests: + pull_request16859
2019-11-25 11:49:21miss-islingtonsetnosy: + miss-islington
messages: + msg357438
2019-11-25 11:29:23pablogsalsetpull_requests: + pull_request16858
2019-11-24 23:02:46pablogsalsetmessages: + msg357417
2019-11-22 09:21:49gvanrossumsetmessages: + msg357258
2019-11-21 21:00:42brett.cannonsetnosy: + brett.cannon
2019-11-21 15:08:05BTaskayasetnosy: + gvanrossum
messages: + msg357176
2019-11-21 07:48:21Batuhan Taskayasetnosy: + Batuhan Taskaya
messages: + msg357138
2019-11-20 23:01:49pablogsalsetmessages: + msg357111
2019-11-20 22:55:42levkivskyisetnosy: + levkivskyi
2019-11-20 22:55:23brandtbuchersetnosy: + brandtbucher
2019-11-20 22:44:42pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request16794
2019-11-20 22:34:29pablogsalcreate