Title: lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6
Type: behavior Stage: resolved
Components: Versions: Python 3.7, Python 3.6
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: benjamin.peterson, eli.bendersky, gregory.p.smith, jab, josephgordon, lukasz.langa, ned.deily, serhiy.storchaka
Priority: normal Keywords: easy, patch

Created on 2015-04-09 13:18 by eli.bendersky, last changed 2017-05-23 05:23 by lukasz.langa. This issue is now closed.

File name Uploaded Description Edit
issue23894.patch josephgordon, 2015-12-22 08:02 review
Pull Requests
URL Status Linked Edit
PR 593 closed lukasz.langa, 2017-03-10 09:08
PR 1724 merged lukasz.langa, 2017-05-22 18:46
PR 1730 merged lukasz.langa, 2017-05-22 22:57
PR 1733 merged lukasz.langa, 2017-05-22 23:16
PR 1737 merged lukasz.langa, 2017-05-22 23:37
Messages (13)
msg240322 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2015-04-09 13:18
lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc')

This is because pgen2/ doesn't list rb'' as a viable prefix for a string, even though according to it is
msg240444 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-10 18:23
Should it?

>>> rb'abc'
  File "<stdin>", line 1
SyntaxError: invalid syntax

According to "rb" is not valid string prefix.
msg240445 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2015-04-10 18:44
Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that.

rb'...' is valid in Python 3:

$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'abc'
msg256831 - (view) Author: Joseph Gordon (josephgordon) Date: 2015-12-22 08:02
With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings.
msg289343 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-03-10 08:29
I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1.

Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think?
msg289415 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-10 22:44
Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final.  Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release.  I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced.  The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests.
msg289423 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-03-11 00:08
Alright, we'll merge this for 3.6.2 then!
msg291321 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-08 06:38
Should this issue be closed now?
msg291572 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-04-12 20:58
No Serhiy, I need a new patch with tests, etc.
msg294196 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-05-22 22:19
New changeset 0c4aca54dcf0c54f299c78aa71fe8f48ff04f9d9 by Łukasz Langa in branch 'master':
Make rb'' strings work in lib2to3 (#1724)
msg294202 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-05-22 23:35
New changeset 1398b1bc7d80df5bde17041e7ec0a3bdbf54b19e by Łukasz Langa in branch '3.6':
[3.6] Make rb'' strings work in lib2to3 (GH-1724) (#1730)
msg294203 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-05-22 23:35
New changeset 1b9530c536664276ce866ae602ce04adce0810e1 by Łukasz Langa in branch 'master':
bpo-23894: make lib2to3 recognize f-strings (#1733)
msg294223 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-05-23 05:23
New changeset e8412e684ed741df246e8430f4911b31b0d8be1f by Łukasz Langa in branch '3.6':
[3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) (#1737)
Date User Action Args
2017-05-23 05:23:31lukasz.langasetmessages: + msg294223
2017-05-23 00:04:34lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-05-22 23:37:52lukasz.langasetpull_requests: + pull_request1827
2017-05-22 23:35:50lukasz.langasetmessages: + msg294203
2017-05-22 23:35:17lukasz.langasetmessages: + msg294202
2017-05-22 23:16:38lukasz.langasetpull_requests: + pull_request1824
2017-05-22 22:57:44lukasz.langasetpull_requests: + pull_request1820
2017-05-22 22:19:11lukasz.langasetmessages: + msg294196
2017-05-22 18:46:08lukasz.langasetpull_requests: + pull_request1811
2017-04-12 20:58:41lukasz.langasetmessages: + msg291572
2017-04-08 06:38:07serhiy.storchakasetmessages: + msg291321
2017-04-07 22:06:58gregory.p.smithsetnosy: + gregory.p.smith
2017-03-11 00:08:57lukasz.langasetmessages: + msg289423
2017-03-10 22:44:04ned.deilysetmessages: + msg289415
2017-03-10 09:31:42lukasz.langasetstage: needs patch -> patch review
2017-03-10 09:31:34lukasz.langasetassignee: lukasz.langa
2017-03-10 09:08:15lukasz.langasetpull_requests: + pull_request489
2017-03-10 08:29:17lukasz.langasetnosy: + ned.deily, lukasz.langa
title: lib2to3 doesn't recognize rb'...' as a raw byte string in Python 3 -> lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6
messages: + msg289343

versions: + Python 3.7, - Python 3.5
2015-12-22 08:02:59josephgordonsetfiles: + issue23894.patch
keywords: + patch
messages: + msg256831
2015-12-14 02:28:47josephgordonsetnosy: + josephgordon
2015-04-10 18:44:20eli.benderskysetmessages: + msg240445
2015-04-10 18:23:46serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg240444
2015-04-09 15:05:14jabsetnosy: + jab
2015-04-09 13:49:58eli.benderskysettype: behavior
stage: needs patch
2015-04-09 13:18:15eli.benderskycreate