Title: lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6
Type: behavior Stage: patch review
Components: Versions: Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: benjamin.peterson, eli.bendersky, jab, josephgordon, lukasz.langa, ned.deily, serhiy.storchaka
Priority: normal Keywords: easy, patch

Created on 2015-04-09 13:18 by eli.bendersky, last changed 2017-03-11 00:08 by lukasz.langa.

File name Uploaded Description Edit
issue23894.patch josephgordon, 2015-12-22 08:02 review
Pull Requests
URL Status Linked Edit
PR 593 closed lukasz.langa, 2017-03-10 09:08
Messages (7)
msg240322 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2015-04-09 13:18
lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc')

This is because pgen2/ doesn't list rb'' as a viable prefix for a string, even though according to it is
msg240444 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-10 18:23
Should it?

>>> rb'abc'
  File "<stdin>", line 1
SyntaxError: invalid syntax

According to "rb" is not valid string prefix.
msg240445 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2015-04-10 18:44
Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that.

rb'...' is valid in Python 3:

$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'abc'
msg256831 - (view) Author: Joseph Gordon (josephgordon) Date: 2015-12-22 08:02
With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings.
msg289343 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-03-10 08:29
I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1.

Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think?
msg289415 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-03-10 22:44
Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final.  Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release.  I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced.  The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests.
msg289423 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-03-11 00:08
Alright, we'll merge this for 3.6.2 then!
Date User Action Args
2017-03-11 00:08:57lukasz.langasetmessages: + msg289423
2017-03-10 22:44:04ned.deilysetmessages: + msg289415
2017-03-10 09:31:42lukasz.langasetstage: needs patch -> patch review
2017-03-10 09:31:34lukasz.langasetassignee: lukasz.langa
2017-03-10 09:08:15lukasz.langasetpull_requests: + pull_request489
2017-03-10 08:29:17lukasz.langasetnosy: + ned.deily, lukasz.langa
title: lib2to3 doesn't recognize rb'...' as a raw byte string in Python 3 -> lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6
messages: + msg289343

versions: + Python 3.7, - Python 3.5
2015-12-22 08:02:59josephgordonsetfiles: + issue23894.patch
keywords: + patch
messages: + msg256831
2015-12-14 02:28:47josephgordonsetnosy: + josephgordon
2015-04-10 18:44:20eli.benderskysetmessages: + msg240445
2015-04-10 18:23:46serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg240444
2015-04-09 15:05:14jabsetnosy: + jab
2015-04-09 13:49:58eli.benderskysettype: behavior
stage: needs patch
2015-04-09 13:18:15eli.benderskycreate