classification
Title: ast.unparse can't roundtrip some f-strings
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: BTaskaya, berker.peksag, eric.smith, hauntsaninja, loganasherjones, pablogsal, remi.lapeyre, serhiy.storchaka, simon.percivall
Priority: low Keywords: patch

Created on 2016-09-07 17:18 by eric.smith, last changed 2020-10-12 22:03 by hauntsaninja.

Pull Requests
URL Status Linked Edit
PR 19612 open hauntsaninja, 2020-04-20 01:06
Messages (9)
msg274841 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-09-07 17:18
The problem relates to how f-strings are re-created.

For the input file foo.txt containing this one line:
f'''{"'"}'''

Run:
./python Tools/parser/test_unparse.py foo.txt

Gives this output:
f'{"\'"}'

This result is not a valid f-string, since it contains a backslash inside the expression part of the f-string.

The input string is a valid f-string:
>>> f'''{"'"}'''
"'"
msg341502 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-06 13:54
There is the same problem with annotations.

>>> from __future__ import annotations
>>> def f() -> X[f'''{"'"}''']: pass
... 
>>> print(f.__annotations__['return'])
X[f'{"\'"}']

You can look at https://github.com/berkerpeksag/astor. Maybe this problem is solved in that project.
msg341512 - (view) Author: RĂ©mi Lapeyre (remi.lapeyre) * Date: 2019-05-06 14:42
The issue is also present in Astor:



Python 3.7.3 (default, Mar 27 2019, 09:23:15) 
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> tree = ast.parse("""
... f'''{"'"}'''
... """)
>>> import astor
>>> astor.to_source(tree)
'f"""{"\'"}"""\n'
>>> f"""{"\'"}"""
  File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash


The issue comes too from calling repr to format the inner string as it does not try to minimize the number of \.
msg341633 - (view) Author: Logan Jones (loganasherjones) * Date: 2019-05-06 19:56
After speaking with Lukasz about this, it seems like the unparser is using the normal unicode repr to determine what should be returned. The default unicode repr will escape quotes if necessary. This is not allowed for f-strings and is the root cause of the problem. 
 
One way to solve this is to add a flag to the unicode_repr function to determine whether or not we need to allow triple quotes in the output of the repr. By default this will be false and will use backslashes, but the ast_unparse will use true for this.
msg366481 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-04-15 02:44
Now that `ast.unparse` is being added to stdlib, I thought I'd bump this thread. The third party library `astunparse` (which attempts to support multiple Python versions while staying very close to Tools/parser/unparse.py) has an implementation that works for complex cases.
If that sounds good, I can submit a PR and inform the original author (astunparse is license is based on PSF's, so should be okay if the original author doesn't respond).

https://github.com/simonpercivall/astunparse/blob/master/lib/astunparse/unparser.py#L461
msg366796 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-04-19 18:21
> If that sounds good, I can submit a PR and inform the original author

I can't say anything before seeing the approach but fixing the problem definitely sounds good.
msg366800 - (view) Author: Simon Percivall (simon.percivall) Date: 2020-04-19 19:13
Any and all code from astunparse is certainly available for inclusion. Go ahead.
msg374724 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-08-03 05:54
Just bumping this issue, as per dev guide, since https://github.com/python/cpython/pull/19612 has been ready for about two months. Would be grateful for review :-)
msg378530 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-10-12 22:03
Bumping again!
History
Date User Action Args
2020-10-12 22:03:47hauntsaninjasetmessages: + msg378530
2020-08-03 05:54:27hauntsaninjasetmessages: + msg374724
2020-04-20 01:06:03hauntsaninjasetkeywords: + patch
stage: patch review
pull_requests: + pull_request18946
2020-04-19 19:13:44simon.percivallsetnosy: + simon.percivall
messages: + msg366800
2020-04-19 18:21:51BTaskayasetmessages: + msg366796
2020-04-15 07:33:13pablogsalsetnosy: + pablogsal
2020-04-15 02:44:13hauntsaninjasetnosy: + hauntsaninja
messages: + msg366481
2020-01-08 16:18:44BTaskayasetversions: + Python 3.9, - Python 3.7, Python 3.8
components: + Library (Lib), - Demos and Tools
title: Some f-strings do not round trip through Tools/parser/test_unparse.py -> ast.unparse can't roundtrip some f-strings
2020-01-08 16:18:12BTaskayasetnosy: + BTaskaya
2019-05-06 19:56:22loganasherjonessetnosy: + loganasherjones
messages: + msg341633
2019-05-06 14:42:37remi.lapeyresetnosy: + remi.lapeyre
messages: + msg341512
2019-05-06 13:54:35serhiy.storchakasetnosy: + berker.peksag, serhiy.storchaka
messages: + msg341502
2019-05-06 12:14:21cheryl.sabellasetassignee: eric.smith
versions: + Python 3.7, Python 3.8, - Python 3.6
2016-09-07 21:52:40eric.smithsettitle: f-strings do not round trip through Tools/parser/test_unparse.py -> Some f-strings do not round trip through Tools/parser/test_unparse.py
2016-09-07 17:18:18eric.smithcreate