This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: lib2to3 cannot parse rf''
Type: crash Stage: resolved
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: 2to3 doesn't parse all valid string literals
View: 33266
Assigned To: Nosy List: benjamin.peterson, majuscule, ned.deily, xtreak
Priority: normal Keywords:

Created on 2019-02-11 16:39 by majuscule, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (4)
msg335238 - (view) Author: Dylan Lloyd (majuscule) Date: 2019-02-11 16:39
```
#!/usr/bin/env python
# -*- coding: utf-8 -*-

regex_format = rf''
```

```
yapf poc.py
```

```
Traceback (most recent call last):
  File "lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 107, in parse_string
    return self.parse_tokens(tokens, debug)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="''", context=('', (4, 17))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/yapf", line 10, in <module>
    sys.exit(run_main())
  File "lib/python3.6/site-packages/yapf/__init__.py", line 326, in run_main
    sys.exit(main(sys.argv))
  File "lib/python3.6/site-packages/yapf/__init__.py", line 213, in main
    verbose=args.verbose)
  File "lib/python3.6/site-packages/yapf/__init__.py", line 263, in FormatFiles
    in_place, print_diff, verify, verbose)
  File "lib/python3.6/site-packages/yapf/__init__.py", line 289, in _FormatFile
    logger=logging.warning)
  File "lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 91, in FormatFile
    verify=verify)
  File "lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 107, in parse_string
    return self.parse_tokens(tokens, debug)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="''", context=('', (4, 17))
```
msg335240 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-02-11 17:02
Seems related : issue33266
msg335241 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-02-11 17:18
The reported example works fine on master and 3.7 . The related issue's fix was not backported to 3.6 . 3.6 is in security fixes only mode. So upgrading to 3.7 will help and I propose closing this as a duplicate of issue33266 .

➜  cpython git:(master) ✗ cat foo.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

regex_format = rf''
➜  cpython git:(master) ✗ ./python.exe ./Tools/scripts/2to3 foo.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: No changes to foo.py
RefactoringTool: Files that need to be modified:
RefactoringTool: foo.py
➜  cpython git:(master) ✗ python3.7 ./Tools/scripts/2to3 foo.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: No changes to foo.py
RefactoringTool: Files that need to be modified:
RefactoringTool: foo.py
➜  cpython git:(master) ✗ python3.6 ./Tools/scripts/2to3 foo.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't parse foo.py: ParseError: bad input: type=3, value="''", context=('', (4, 17))
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't parse foo.py: ParseError: bad input: type=3, value="''", context=('', (4, 17))
msg335250 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-02-11 18:41
@xtreak, I concur.  While it would have better to originally fix this in the release where f strings first appeared, the window to do that has closed.  Sorry!
History
Date User Action Args
2022-04-11 14:59:11adminsetgithub: 80149
2019-02-11 18:41:07ned.deilysetstatus: open -> closed

superseder: 2to3 doesn't parse all valid string literals

nosy: + ned.deily
messages: + msg335250
resolution: duplicate
stage: resolved
2019-02-11 17:18:39xtreaksetnosy: + benjamin.peterson
messages: + msg335241
2019-02-11 17:02:42xtreaksetnosy: + xtreak
messages: + msg335240
2019-02-11 16:39:57majusculecreate