This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ast.unparse can't roundtrip some f-strings
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: BTaskaya, berker.peksag, eric.smith, hauntsaninja, loganasherjones, miss-islington, pablogsal, remi.lapeyre, serhiy.storchaka, simon.percivall
Priority: low Keywords: patch

Created on 2016-09-07 17:18 by eric.smith, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19612 merged hauntsaninja, 2020-04-20 01:06
PR 23430 merged miss-islington, 2020-11-20 21:16
Messages (12)
msg274841 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-09-07 17:18
The problem relates to how f-strings are re-created.

For the input file foo.txt containing this one line:
f'''{"'"}'''

Run:
./python Tools/parser/test_unparse.py foo.txt

Gives this output:
f'{"\'"}'

This result is not a valid f-string, since it contains a backslash inside the expression part of the f-string.

The input string is a valid f-string:
>>> f'''{"'"}'''
"'"
msg341502 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-06 13:54
There is the same problem with annotations.

>>> from __future__ import annotations
>>> def f() -> X[f'''{"'"}''']: pass
... 
>>> print(f.__annotations__['return'])
X[f'{"\'"}']

You can look at https://github.com/berkerpeksag/astor. Maybe this problem is solved in that project.
msg341512 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-05-06 14:42
The issue is also present in Astor:



Python 3.7.3 (default, Mar 27 2019, 09:23:15) 
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> tree = ast.parse("""
... f'''{"'"}'''
... """)
>>> import astor
>>> astor.to_source(tree)
'f"""{"\'"}"""\n'
>>> f"""{"\'"}"""
  File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash


The issue comes too from calling repr to format the inner string as it does not try to minimize the number of \.
msg341633 - (view) Author: Logan Jones (loganasherjones) * Date: 2019-05-06 19:56
After speaking with Lukasz about this, it seems like the unparser is using the normal unicode repr to determine what should be returned. The default unicode repr will escape quotes if necessary. This is not allowed for f-strings and is the root cause of the problem. 
 
One way to solve this is to add a flag to the unicode_repr function to determine whether or not we need to allow triple quotes in the output of the repr. By default this will be false and will use backslashes, but the ast_unparse will use true for this.
msg366481 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-04-15 02:44
Now that `ast.unparse` is being added to stdlib, I thought I'd bump this thread. The third party library `astunparse` (which attempts to support multiple Python versions while staying very close to Tools/parser/unparse.py) has an implementation that works for complex cases.
If that sounds good, I can submit a PR and inform the original author (astunparse is license is based on PSF's, so should be okay if the original author doesn't respond).

https://github.com/simonpercivall/astunparse/blob/master/lib/astunparse/unparser.py#L461
msg366796 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-04-19 18:21
> If that sounds good, I can submit a PR and inform the original author

I can't say anything before seeing the approach but fixing the problem definitely sounds good.
msg366800 - (view) Author: Simon Percivall (simon.percivall) Date: 2020-04-19 19:13
Any and all code from astunparse is certainly available for inclusion. Go ahead.
msg374724 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-08-03 05:54
Just bumping this issue, as per dev guide, since https://github.com/python/cpython/pull/19612 has been ready for about two months. Would be grateful for review :-)
msg378530 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-10-12 22:03
Bumping again!
msg381507 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-11-20 21:16
New changeset a993e901ebe60c38d46ecb31f771d0b4a206828c by Shantanu in branch 'master':
bpo-28002: Roundtrip f-strings with ast.unparse better (#19612)
https://github.com/python/cpython/commit/a993e901ebe60c38d46ecb31f771d0b4a206828c
msg381509 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-11-20 21:47
New changeset 3763cc1dbdb930f67b443ceed7c44e4feb883b42 by Miss Islington (bot) in branch '3.9':
bpo-28002: Roundtrip f-strings with ast.unparse better (GH-19612) (GH-23430)
https://github.com/python/cpython/commit/3763cc1dbdb930f67b443ceed7c44e4feb883b42
msg381510 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-11-20 21:53
Thanks for your efforts Shantanu!
History
Date User Action Args
2022-04-11 14:58:35adminsetgithub: 72189
2020-11-20 21:55:43eric.smithsetresolution: not a bug -> fixed
2020-11-20 21:53:35BTaskayasetmessages: + msg381510
2020-11-20 21:53:16BTaskayasetstatus: open -> closed
resolution: not a bug
stage: patch review -> resolved
2020-11-20 21:47:00BTaskayasetmessages: + msg381509
2020-11-20 21:16:56miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request22321
2020-11-20 21:16:49BTaskayasetmessages: + msg381507
2020-10-12 22:03:47hauntsaninjasetmessages: + msg378530
2020-08-03 05:54:27hauntsaninjasetmessages: + msg374724
2020-04-20 01:06:03hauntsaninjasetkeywords: + patch
stage: patch review
pull_requests: + pull_request18946
2020-04-19 19:13:44simon.percivallsetnosy: + simon.percivall
messages: + msg366800
2020-04-19 18:21:51BTaskayasetmessages: + msg366796
2020-04-15 07:33:13pablogsalsetnosy: + pablogsal
2020-04-15 02:44:13hauntsaninjasetnosy: + hauntsaninja
messages: + msg366481
2020-01-08 16:18:44BTaskayasetversions: + Python 3.9, - Python 3.7, Python 3.8
components: + Library (Lib), - Demos and Tools
title: Some f-strings do not round trip through Tools/parser/test_unparse.py -> ast.unparse can't roundtrip some f-strings
2020-01-08 16:18:12BTaskayasetnosy: + BTaskaya
2019-05-06 19:56:22loganasherjonessetnosy: + loganasherjones
messages: + msg341633
2019-05-06 14:42:37remi.lapeyresetnosy: + remi.lapeyre
messages: + msg341512
2019-05-06 13:54:35serhiy.storchakasetnosy: + berker.peksag, serhiy.storchaka
messages: + msg341502
2019-05-06 12:14:21cheryl.sabellasetassignee: eric.smith
versions: + Python 3.7, Python 3.8, - Python 3.6
2016-09-07 21:52:40eric.smithsettitle: f-strings do not round trip through Tools/parser/test_unparse.py -> Some f-strings do not round trip through Tools/parser/test_unparse.py
2016-09-07 17:18:18eric.smithcreate