lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc')

This is because pgen2/tokenize.py doesn't list rb'' as a viable prefix for a string, even though according to https://docs.python.org/3/reference/lexical_analysis.html it is

Should it?

>>> rb'abc'
  File "<stdin>", line 1
    rb'abc'
          ^
SyntaxError: invalid syntax

According to https://docs.python.org/2/reference/lexical_analysis.html#string-literals "rb" is not valid string prefix.

Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that.

rb'...' is valid in Python 3:

$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'abc'
b'abc'
>>>

With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings.

I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1.

Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think?

Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final.  Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release.  I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced.  The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests.

Alright, we'll merge this for 3.6.2 then!

Should this issue be closed now?

No Serhiy, I need a new patch with tests, etc.

New changeset 0c4aca54dcf0c54f299c78aa71fe8f48ff04f9d9 by Łukasz Langa in branch 'master':
Make rb'' strings work in lib2to3 (#1724)
https://github.com/python/cpython/commit/0c4aca54dcf0c54f299c78aa71fe8f48ff04f9d9

New changeset 1398b1bc7d80df5bde17041e7ec0a3bdbf54b19e by Łukasz Langa in branch '3.6':
[3.6] Make rb'' strings work in lib2to3 (GH-1724) (#1730)
https://github.com/python/cpython/commit/1398b1bc7d80df5bde17041e7ec0a3bdbf54b19e

New changeset 1b9530c536664276ce866ae602ce04adce0810e1 by Łukasz Langa in branch 'master':
bpo-23894: make lib2to3 recognize f-strings (#1733)
https://github.com/python/cpython/commit/1b9530c536664276ce866ae602ce04adce0810e1

New changeset e8412e684ed741df246e8430f4911b31b0d8be1f by Łukasz Langa in branch '3.6':
[3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) (#1737)
https://github.com/python/cpython/commit/e8412e684ed741df246e8430f4911b31b0d8be1f

Misc/NEWS entries for these changes?

New changeset b7c59f0d05bde0d4dd2d7bffc1b8af72c25d3f71 by Ned Deily in branch '3.6':
bpo-23894: add Misc/NEWS entry. (#2255)
https://github.com/python/cpython/commit/b7c59f0d05bde0d4dd2d7bffc1b8af72c25d3f71

New changeset ceb817dcfe8bf99725de033cce16a6f4cbced6ba by Ned Deily in branch 'master':
bpo-23894: add Misc/NEWS entry. (#2256)
https://github.com/python/cpython/commit/ceb817dcfe8bf99725de033cce16a6f4cbced6ba