Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6 #68082

Closed
elibendersky mannequin opened this issue Apr 9, 2015 · 16 comments
Closed

lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6 #68082

elibendersky mannequin opened this issue Apr 9, 2015 · 16 comments
Assignees
@ambv
Labels
3.7 (EOL) end of life easy type-bug An unexpected behavior, bug, or error

Comments

@elibendersky
Copy link
Mannequin

elibendersky mannequin commented Apr 9, 2015

BPO 23894
Nosy @gpshead, @benjaminp, @ned-deily, @jab, @ambv, @serhiy-storchaka
PRs
  • bpo-23894: lib2to3 doesn't recognize rb'...' and f'...' #593
  • bpo-23894: Make rb'' strings work in lib2to3 #1724
  • [3.6] bpo-23894: Make rb'' strings work in lib2to3 (GH-1724) #1730
  • bpo-23894: make lib2to3 recognize f-strings #1733
  • [3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) #1737
  • bpo-23894: add Misc/NEWS entry. #2255
  • bpo-23894: add Misc/NEWS entry. #2256
    		•
    Files
  • issue23894.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ambv'
closed_at = <Date 2017-05-23.00:04:34.799>
created_at = <Date 2015-04-09.13:18:15.097>
labels = ['easy', 'type-bug', '3.7']
title = "lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6"
updated_at = <Date 2017-06-17.02:43:33.286>
user = 'https://bugs.python.org/elibendersky'

    bugs.python.org fields:

    activity = <Date 2017-06-17.02:43:33.286>
actor = 'ned.deily'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2017-05-23.00:04:34.799>
closer = 'lukasz.langa'
components = []
creation = <Date 2015-04-09.13:18:15.097>
creator = 'eli.bendersky'
dependencies = []
files = ['41382']
hgrepos = []
issue_num = 23894
keywords = ['patch', 'easy']
message_count = 16.0
messages = ['240322', '240444', '240445', '256831', '289343', '289415', '289423', '291321', '291572', '294196', '294202', '294203', '294223', '295836', '296227', '296228']
nosy_count = 8.0
nosy_names = ['gregory.p.smith', 'benjamin.peterson', 'ned.deily', 'eli.bendersky', 'jab', 'lukasz.langa', 'serhiy.storchaka', 'josephgordon']
pr_nums = ['593', '1724', '1730', '1733', '1737', '2255', '2256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23894'
versions = ['Python 3.6', 'Python 3.7']
    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Apr 9, 2015

    lib2to3 tokenizes br'abc' as a single STRING token, but rb'abc' as two separate tokens (NAME "rb" and STRING 'abc')

    This is because pgen2/tokenize.py doesn't list rb'' as a viable prefix for a string, even though according to https://docs.python.org/3/reference/lexical_analysis.html it is

    @elibendersky elibendersky mannequin added easy type-bug An unexpected behavior, bug, or error labels Apr 9, 2015
    @serhiy-storchaka
    Copy link
    Member

    Should it?

    >>> rb'abc'
  File "<stdin>", line 1
    rb'abc'
          ^
SyntaxError: invalid syntax

    According to https://docs.python.org/2/reference/lexical_analysis.html#string-literals "rb" is not valid string prefix.

    @elibendersky
    Copy link
    Mannequin Author

    elibendersky mannequin commented Apr 10, 2015

    Serhiy, AFAIK lib2to3 has been repurposed to parse Python 3 as well - it has certainly gained quite a bit of support for that.

    rb'...' is valid in Python 3:

    $ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> rb'abc'
b'abc'
>>>

    @josephgordon
    Copy link
    Mannequin

    josephgordon mannequin commented Dec 22, 2015

    With the changes in the uploaded patch it looks like strings of the form rb'...' are now recognized as raw byte strings.

    @ambv
    Copy link
    Contributor

    ambv commented Mar 10, 2017

    I don't think we'll fix it for 3.5 anymore but it's definitely worth doing so for 3.6.1.

    Ned, I have a patch, will upload momentarily. I'd like this to go in 3.6.1 since without it fixed tools depending on lib2to3 will crash seeing f-strings. This includes YAPF, a pretty widely used auto-formatter for Python code. The fix is relatively trivial. What do you think?

    @ambv ambv added the 3.7 (EOL) end of life label Mar 10, 2017
    @ambv ambv changed the title lib2to3 doesn't recognize rb'...' as a raw byte string in Python 3 lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6 Mar 10, 2017
    @ambv ambv self-assigned this Mar 10, 2017
    @ned-deily
    Copy link
    Member

    Sorry, the announced deadline for bugfixes for 3.6.1 has already passed; at this point, only showstopper problems with 3.6.1rc1 are on the table for the final. Trivial impact isn't a criterion for post-rc1 changes and there's a good reason for that: we ask everyone in the community to test a release candidate with the expectation that this is the final release. I don't think it's fair to add more unrelated changes and risk that it will break something and/or cause additional release candidates to be produced. The original problem has been open for nearly two years now and, while "f" string support adds to the importance of the tokenizer getting updated, this can wait 3 more months for 3.6.2 with a proper review cycle and with tests.

    @ambv
    Copy link
    Contributor

    ambv commented Mar 11, 2017

    Alright, we'll merge this for 3.6.2 then!

    @serhiy-storchaka
    Copy link
    Member

    Should this issue be closed now?

    @ambv
    Copy link
    Contributor

    ambv commented Apr 12, 2017

    No Serhiy, I need a new patch with tests, etc.

    @ambv
    Copy link
    Contributor

    ambv commented May 22, 2017

    New changeset 0c4aca5 by Łukasz Langa in branch 'master':
    Make rb'' strings work in lib2to3 (bpo-1724)
    0c4aca5

    @ambv
    Copy link
    Contributor

    ambv commented May 22, 2017

    New changeset 1398b1b by Łukasz Langa in branch '3.6':
    [3.6] Make rb'' strings work in lib2to3 (GH-1724) (bpo-1730)
    1398b1b

    @ambv
    Copy link
    Contributor

    ambv commented May 22, 2017

    New changeset 1b9530c by Łukasz Langa in branch 'master':
    bpo-23894: make lib2to3 recognize f-strings (bpo-1733)
    1b9530c

    @ambv ambv closed this as completed May 23, 2017
    @ambv
    Copy link
    Contributor

    ambv commented May 23, 2017

    New changeset e8412e6 by Łukasz Langa in branch '3.6':
    [3.6] bpo-23894: make lib2to3 recognize f-strings (GH-1733) (bpo-1737)
    e8412e6

    @ned-deily
    Copy link
    Member

    Misc/NEWS entries for these changes?

    @ned-deily
    Copy link
    Member

    New changeset b7c59f0 by Ned Deily in branch '3.6':
    bpo-23894: add Misc/NEWS entry. (bpo-2255)
    b7c59f0

    @ned-deily
    Copy link
    Member

    New changeset ceb817d by Ned Deily in branch 'master':
    bpo-23894: add Misc/NEWS entry. (bpo-2256)
    ceb817d

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    hroncok pushed a commit to hroncok/cpython that referenced this issue Jan 17, 2024
    @encukou
    00001-pydocnogui.patch
    253b416 
    Removes the "-g" option from "pydoc", for some reason; I believe
(dmalcolm 2010-01-29) that this was introduced in this change:
- fix pydoc (python#68082)
in 2.2.1-12 as a response to the -g option needing TkInter installed
(Red Hat Linux 8)
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @ambv ambv

    Labels
    3.7 (EOL) end of life easy type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @ambv @serhiy-storchaka @ned-deily