Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nested multi-line expression will lead to "compile()" fails #87084

Closed
xxm mannequin opened this issue Jan 13, 2021 · 9 comments
Closed

Nested multi-line expression will lead to "compile()" fails #87084

xxm mannequin opened this issue Jan 13, 2021 · 9 comments
Labels
3.10 only security fixes 3.11 only security fixes build The build process and cross-build interpreter-core (Objects, Python, Grammar, and Parser dirs)

Comments

@xxm
Copy link
Mannequin

xxm mannequin commented Jan 13, 2021

BPO 42918
Nosy @gvanrossum, @ambv, @lysnikolaou, @hongweipeng, @pablogsal, @miss-islington
PRs
  • bpo-42918: Improve build-in function compile() in mode 'single' #29934
  • [3.10] bpo-42918: Improve build-in function compile() in mode 'single' (GH-29934) #30040
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-12-27.16:17:23.694>
    created_at = <Date 2021-01-13.06:01:57.716>
    labels = ['interpreter-core', 'build', '3.10', '3.11']
    title = 'Nested multi-line expression will lead to "compile()" fails'
    updated_at = <Date 2021-12-27.16:17:23.690>
    user = 'https://bugs.python.org/xxm'

    bugs.python.org fields:

    activity = <Date 2021-12-27.16:17:23.690>
    actor = 'lukasz.langa'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-12-27.16:17:23.694>
    closer = 'lukasz.langa'
    components = ['Interpreter Core']
    creation = <Date 2021-01-13.06:01:57.716>
    creator = 'xxm'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 42918
    keywords = ['patch']
    message_count = 9.0
    messages = ['384995', '384999', '385000', '385001', '385003', '386542', '408269', '409231', '409232']
    nosy_count = 7.0
    nosy_names = ['gvanrossum', 'lukasz.langa', 'lys.nikolaou', 'hongweipeng', 'pablogsal', 'miss-islington', 'xxm']
    pr_nums = ['29934', '30040']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue42918'
    versions = ['Python 3.10', 'Python 3.11']

    @xxm
    Copy link
    Mannequin Author

    xxm mannequin commented Jan 13, 2021

    For build-in function compile() in mode 'single', only single statement can be well compiled. If we compile multiple statements, the parser will raise a syntaxError. Seeing the following two programs, In program 1, 2 statement are compiled. So the parser raises a Syntax error. It's the expected output. However, if we insert a nested multi-line assignment in this code(see program 2), 3 statements are compiled. We expect the parser also raise a Sytax error. But it' not.

    Program 1 ( with expected results)
    ===================================

    code1 = """
    a = 1
    b = 2
    """
    c = compile(code1, "", "single")

    ===================================

    Traceback (most recent call last):
      File "/home/xxm/Desktop/nameChanging/report/test1.py", line 641, in <module>
        c = compile(code1, "", "single")
      File "", line 2
        a = 1
             ^
    SyntaxError: multiple statements found while compiling a single statement

    Program 2 (with unexpected results)
    =================================

    code2 = """
    c ='''
    d=1
    '''
    a = 1
    b = 2
    """
    c = compile(code2, "", "single")

    =================================

    Expected out for program 2: Raise a syntaxError too, But it's not.

    > python -V
    Python 3.10.0a2
    >uname -a
    Linux xxm-System-Product-Name 4.15.0-64-generic #73~16.04.1-Ubuntu SMP Fri Sep 13 09:56:18 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

    @xxm xxm mannequin added 3.10 only security fixes build The build process and cross-build interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jan 13, 2021
    @gvanrossum
    Copy link
    Member

    What does the bytecode for c in program 2 look like?

    Also, how do you find all these bugs? Are you developing a new fuzzier?

    @xxm
    Copy link
    Mannequin Author

    xxm mannequin commented Jan 13, 2021

    The bytecode of c is as following:

              0 LOAD_CONST               0 (1)
              2 STORE_NAME               0 (d)
              4 LOAD_CONST               1 (None)
              6 RETURN_VALUE
    

    ===============================================

    Yes, my team is working on developing a new fuzzier for the CPython.

    Also, we are trying to apply it on the newest version of CPython.

    We will carefully analyze all bugs before reporting them on the tracker.

    Hope our work does not bother you too much.

    @gvanrossum
    Copy link
    Member

    It’s good work! Is it open source?

    @xxm
    Copy link
    Mannequin Author

    xxm mannequin commented Jan 13, 2021

    For sure! As soon as we validate this technique, we will open source it on GitHub.

    @lysnikolaou
    Copy link
    Contributor

    Thanks a lot for the report, Xinmeng! This is a bug in how we're checking for bad single statements in bad_single_statement and it's not trivial to fix due to the string shenanigans we're doing there in newline_in_string, which does not account for many edge cases. I'll try to dig a bit deeper during the weekend.

    @ambv
    Copy link
    Contributor

    ambv commented Dec 10, 2021

    New changeset 28179aa by Weipeng Hong in branch 'main':
    bpo-42918: Improve build-in function compile() in mode 'single' (GH-29934)
    28179aa

    @ambv
    Copy link
    Contributor

    ambv commented Dec 27, 2021

    New changeset 576e38f by Miss Islington (bot) in branch '3.10':
    bpo-42918: Improve built-in function compile() in mode 'single' (GH-29934) (GH-30040)
    576e38f

    @ambv
    Copy link
    Contributor

    ambv commented Dec 27, 2021

    Thanks! ✨ 🍰 ✨

    @ambv ambv added the 3.11 only security fixes label Dec 27, 2021
    @ambv ambv closed this as completed Dec 27, 2021
    @ambv ambv added the 3.11 only security fixes label Dec 27, 2021
    @ambv ambv closed this as completed Dec 27, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes 3.11 only security fixes build The build process and cross-build interpreter-core (Objects, Python, Grammar, and Parser dirs)
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants