msg274841 - (view) |
Author: Eric V. Smith (eric.smith) *  |
Date: 2016-09-07 17:18 |
The problem relates to how f-strings are re-created.
For the input file foo.txt containing this one line:
f'''{"'"}'''
Run:
./python Tools/parser/test_unparse.py foo.txt
Gives this output:
f'{"\'"}'
This result is not a valid f-string, since it contains a backslash inside the expression part of the f-string.
The input string is a valid f-string:
>>> f'''{"'"}'''
"'"
|
msg341502 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2019-05-06 13:54 |
There is the same problem with annotations.
>>> from __future__ import annotations
>>> def f() -> X[f'''{"'"}''']: pass
...
>>> print(f.__annotations__['return'])
X[f'{"\'"}']
You can look at https://github.com/berkerpeksag/astor. Maybe this problem is solved in that project.
|
msg341512 - (view) |
Author: Rémi Lapeyre (remi.lapeyre) * |
Date: 2019-05-06 14:42 |
The issue is also present in Astor:
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> tree = ast.parse("""
... f'''{"'"}'''
... """)
>>> import astor
>>> astor.to_source(tree)
'f"""{"\'"}"""\n'
>>> f"""{"\'"}"""
File "<stdin>", line 1
SyntaxError: f-string expression part cannot include a backslash
The issue comes too from calling repr to format the inner string as it does not try to minimize the number of \.
|
msg341633 - (view) |
Author: Logan Jones (loganasherjones) * |
Date: 2019-05-06 19:56 |
After speaking with Lukasz about this, it seems like the unparser is using the normal unicode repr to determine what should be returned. The default unicode repr will escape quotes if necessary. This is not allowed for f-strings and is the root cause of the problem.
One way to solve this is to add a flag to the unicode_repr function to determine whether or not we need to allow triple quotes in the output of the repr. By default this will be false and will use backslashes, but the ast_unparse will use true for this.
|
msg366481 - (view) |
Author: Shantanu (hauntsaninja) * |
Date: 2020-04-15 02:44 |
Now that `ast.unparse` is being added to stdlib, I thought I'd bump this thread. The third party library `astunparse` (which attempts to support multiple Python versions while staying very close to Tools/parser/unparse.py) has an implementation that works for complex cases.
If that sounds good, I can submit a PR and inform the original author (astunparse is license is based on PSF's, so should be okay if the original author doesn't respond).
https://github.com/simonpercivall/astunparse/blob/master/lib/astunparse/unparser.py#L461
|
msg366796 - (view) |
Author: Batuhan Taskaya (BTaskaya) *  |
Date: 2020-04-19 18:21 |
> If that sounds good, I can submit a PR and inform the original author
I can't say anything before seeing the approach but fixing the problem definitely sounds good.
|
msg366800 - (view) |
Author: Simon Percivall (simon.percivall) |
Date: 2020-04-19 19:13 |
Any and all code from astunparse is certainly available for inclusion. Go ahead.
|
msg374724 - (view) |
Author: Shantanu (hauntsaninja) * |
Date: 2020-08-03 05:54 |
Just bumping this issue, as per dev guide, since https://github.com/python/cpython/pull/19612 has been ready for about two months. Would be grateful for review :-)
|
msg378530 - (view) |
Author: Shantanu (hauntsaninja) * |
Date: 2020-10-12 22:03 |
Bumping again!
|
msg381507 - (view) |
Author: Batuhan Taskaya (BTaskaya) *  |
Date: 2020-11-20 21:16 |
New changeset a993e901ebe60c38d46ecb31f771d0b4a206828c by Shantanu in branch 'master':
bpo-28002: Roundtrip f-strings with ast.unparse better (#19612)
https://github.com/python/cpython/commit/a993e901ebe60c38d46ecb31f771d0b4a206828c
|
msg381509 - (view) |
Author: Batuhan Taskaya (BTaskaya) *  |
Date: 2020-11-20 21:47 |
New changeset 3763cc1dbdb930f67b443ceed7c44e4feb883b42 by Miss Islington (bot) in branch '3.9':
bpo-28002: Roundtrip f-strings with ast.unparse better (GH-19612) (GH-23430)
https://github.com/python/cpython/commit/3763cc1dbdb930f67b443ceed7c44e4feb883b42
|
msg381510 - (view) |
Author: Batuhan Taskaya (BTaskaya) *  |
Date: 2020-11-20 21:53 |
Thanks for your efforts Shantanu!
|
|
Date |
User |
Action |
Args |
2022-04-11 14:58:35 | admin | set | github: 72189 |
2020-11-20 21:55:43 | eric.smith | set | resolution: not a bug -> fixed |
2020-11-20 21:53:35 | BTaskaya | set | messages:
+ msg381510 |
2020-11-20 21:53:16 | BTaskaya | set | status: open -> closed resolution: not a bug stage: patch review -> resolved |
2020-11-20 21:47:00 | BTaskaya | set | messages:
+ msg381509 |
2020-11-20 21:16:56 | miss-islington | set | nosy:
+ miss-islington pull_requests:
+ pull_request22321
|
2020-11-20 21:16:49 | BTaskaya | set | messages:
+ msg381507 |
2020-10-12 22:03:47 | hauntsaninja | set | messages:
+ msg378530 |
2020-08-03 05:54:27 | hauntsaninja | set | messages:
+ msg374724 |
2020-04-20 01:06:03 | hauntsaninja | set | keywords:
+ patch stage: patch review pull_requests:
+ pull_request18946 |
2020-04-19 19:13:44 | simon.percivall | set | nosy:
+ simon.percivall messages:
+ msg366800
|
2020-04-19 18:21:51 | BTaskaya | set | messages:
+ msg366796 |
2020-04-15 07:33:13 | pablogsal | set | nosy:
+ pablogsal
|
2020-04-15 02:44:13 | hauntsaninja | set | nosy:
+ hauntsaninja messages:
+ msg366481
|
2020-01-08 16:18:44 | BTaskaya | set | versions:
+ Python 3.9, - Python 3.7, Python 3.8 components:
+ Library (Lib), - Demos and Tools title: Some f-strings do not round trip through Tools/parser/test_unparse.py -> ast.unparse can't roundtrip some f-strings |
2020-01-08 16:18:12 | BTaskaya | set | nosy:
+ BTaskaya
|
2019-05-06 19:56:22 | loganasherjones | set | nosy:
+ loganasherjones messages:
+ msg341633
|
2019-05-06 14:42:37 | remi.lapeyre | set | nosy:
+ remi.lapeyre messages:
+ msg341512
|
2019-05-06 13:54:35 | serhiy.storchaka | set | nosy:
+ berker.peksag, serhiy.storchaka messages:
+ msg341502
|
2019-05-06 12:14:21 | cheryl.sabella | set | assignee: eric.smith versions:
+ Python 3.7, Python 3.8, - Python 3.6 |
2016-09-07 21:52:40 | eric.smith | set | title: f-strings do not round trip through Tools/parser/test_unparse.py -> Some f-strings do not round trip through Tools/parser/test_unparse.py |
2016-09-07 17:18:18 | eric.smith | create | |