This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Heap buffer overflow in the parser
Type: crash Stage:
Components: Interpreter Core Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pablogsal Nosy List: blarsen, gvanrossum, lukasz.langa, pablogsal
Priority: release blocker Keywords: patch

Created on 2020-09-03 01:18 by blarsen, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22077 merged pablogsal, 2020-09-03 10:48
PR 22079 merged pablogsal, 2020-09-03 14:32
Messages (19)
msg376270 - (view) Author: Brad Larsen (blarsen) * Date: 2020-09-03 01:18
It looks like commit 4a97b1517a6b5ff22e2984b677a680b07ff0ce11 introduced a heap buffer overflow:

    commit 4a97b1517a6b5ff22e2984b677a680b07ff0ce11 (HEAD -> master, origin/master, origin/HEAD)
    Author: Pablo Galindo <Pablogsal@gmail.com>
    Date:   Wed Sep 2 17:44:19 2020 +0100
    
        bpo-41690: Use a loop to collect args in the parser instead of recursion (GH-22053)
    
        This program can segfault the parser by stack overflow:
    
        ```
        import ast
    
        code = "f(" + ",".join(['a' for _ in range(100000)]) + ")"
        print("Ready!")
        ast.parse(code)
        ```
    
        the reason is that the rule for arguments has a simple recursion when collecting args:
    
        args[expr_ty]:
            [...]
            | a=named_expression b=[',' c=args { c }] {
                [...] }


If you try building with clang-10 with `--with-pydebug --with-address-sanitizer`, you should see a crash like the following during the `generate-posix-vars` step:

    =================================================================
    ==39814==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x625000264148 at pc 0x000001ff3be8 bp 0x7ffec90e5d00 sp 0x7ffec90e5cf8
    READ of size 4 at 0x625000264148 thread T0
        #0 0x1ff3be7 in _PyPegen_collect_call_seqs /build/python/cpython/Parser/pegen.c:2253:61
        #1 0x218ab08 in args_rule /build/python/cpython/Parser/parser.c:12240:20
        #2 0x20f8906 in arguments_rule /build/python/cpython/Parser/parser.c:12159:18
        #3 0x2158c61 in t_primary_raw /build/python/cpython/Parser/parser.c:14063:18
        #4 0x21416fb in t_primary_rule /build/python/cpython/Parser/parser.c:13896:22
        #5 0x246d944 in single_subscript_attribute_target_rule /build/python/cpython/Parser/parser.c:13265:18
        #6 0x2433a19 in _tmp_20_rule /build/python/cpython/Parser/parser.c:16717:54
        #7 0x24016e3 in assignment_rule /build/python/cpython/Parser/parser.c:2093:18
        #8 0x23e6617 in small_stmt_rule /build/python/cpython/Parser/parser.c:1526:31
        #9 0x2018581 in simple_stmt_rule /build/python/cpython/Parser/parser.c:1424:18
        #10 0x200c22c in statement_rule /build/python/cpython/Parser/parser.c:1258:32
        #11 0x2007026 in _loop1_11_rule /build/python/cpython/Parser/parser.c:16174:30
        #12 0x200455a in statements_rule /build/python/cpython/Parser/parser.c:1193:18
        #13 0x230193f in block_rule /build/python/cpython/Parser/parser.c:6257:18
        #14 0x205886b in function_def_raw_rule /build/python/cpython/Parser/parser.c:4927:18
        #15 0x20229a4 in function_def_rule /build/python/cpython/Parser/parser.c:4856:37
        #16 0x200e2da in compound_stmt_rule /build/python/cpython/Parser/parser.c:1872:33
        #17 0x200a873 in statement_rule /build/python/cpython/Parser/parser.c:1234:18
        #18 0x2007026 in _loop1_11_rule /build/python/cpython/Parser/parser.c:16174:30
        #19 0x200455a in statements_rule /build/python/cpython/Parser/parser.c:1193:18
        #20 0x230193f in block_rule /build/python/cpython/Parser/parser.c:6257:18
        #21 0x2392ac3 in class_def_raw_rule /build/python/cpython/Parser/parser.c:6196:18
        #22 0x202fb74 in class_def_rule /build/python/cpython/Parser/parser.c:6139:34
        #23 0x2010e47 in compound_stmt_rule /build/python/cpython/Parser/parser.c:1914:30
        #24 0x200a873 in statement_rule /build/python/cpython/Parser/parser.c:1234:18
        #25 0x2007026 in _loop1_11_rule /build/python/cpython/Parser/parser.c:16174:30
        #26 0x200455a in statements_rule /build/python/cpython/Parser/parser.c:1193:18
        #27 0x230193f in block_rule /build/python/cpython/Parser/parser.c:6257:18
        #28 0x238f31b in else_block_rule /build/python/cpython/Parser/parser.c:3787:18
        #29 0x204e3c4 in try_stmt_rule /build/python/cpython/Parser/parser.c:4460:19
        #30 0x2014f68 in compound_stmt_rule /build/python/cpython/Parser/parser.c:1977:29
        #31 0x200a873 in statement_rule /build/python/cpython/Parser/parser.c:1234:18
        #32 0x2007026 in _loop1_11_rule /build/python/cpython/Parser/parser.c:16174:30
        #33 0x200455a in statements_rule /build/python/cpython/Parser/parser.c:1193:18
        #34 0x1ff8c93 in file_rule /build/python/cpython/Parser/parser.c:726:18
        #35 0x1ff742d in _PyPegen_parse /build/python/cpython/Parser/parser.c:24794:18
        #36 0x1fc1128 in _PyPegen_run_parser /build/python/cpython/Parser/pegen.c:1111:17
        #37 0x1fc5e38 in _PyPegen_run_parser_from_string /build/python/cpython/Parser/pegen.c:1238:14
        #38 0x1a8952b in PyParser_ASTFromStringObject /build/python/cpython/Parser/peg_api.c:27:21
        #39 0x1339bef in Py_CompileStringObject /build/python/cpython/Python/pythonrun.c:1203:11
        #40 0x1f2ac43 in builtin_compile_impl /build/python/cpython/Python/bltinmodule.c:819:14
        #41 0x1f1902a in builtin_compile /build/python/cpython/Python/clinic/bltinmodule.c.h:249:20
        #42 0x1cab483 in cfunction_vectorcall_FASTCALL_KEYWORDS /build/python/cpython/Objects/methodobject.c:440:24
        #43 0x58bfa8 in PyVectorcall_Call /build/python/cpython/Objects/call.c:242:24
        #44 0x58c3c6 in _PyObject_Call /build/python/cpython/Objects/call.c:265:16
        #45 0x58c8b1 in PyObject_Call /build/python/cpython/Objects/call.c:292:12
        #46 0x1017138 in do_call_core /build/python/cpython/Python/ceval.c:5141:9
        #47 0xfeada4 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3609:22
        #48 0xf4068b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #49 0x1025062 in _PyEval_EvalCode /build/python/cpython/Python/ceval.c:4376:14
        #50 0x58ec4e in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:395:12
        #51 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #52 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #53 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #54 0xfe8b6a in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3564:19
        #55 0xf4068b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #56 0x1025062 in _PyEval_EvalCode /build/python/cpython/Python/ceval.c:4376:14
        #57 0x58ec4e in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:395:12
        #58 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #59 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #60 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #61 0xfe5f00 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3533:23
        #62 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #63 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #64 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #65 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #66 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #67 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #68 0xfe5f00 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3533:23
        #69 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #70 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #71 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #72 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #73 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #74 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #75 0xfe5f00 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3533:23
        #76 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #77 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #78 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #79 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #80 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #81 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #82 0xfe71eb in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3547:19
        #83 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #84 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #85 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #86 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #87 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #88 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #89 0xfe71eb in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3547:19
        #90 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #91 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #92 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #93 0x5946bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #94 0x5959b3 in object_vacall /build/python/cpython/Objects/call.c:791:14
        #95 0x595f4c in _PyObject_CallMethodIdObjArgs /build/python/cpython/Objects/call.c:882:24
        #96 0x11ffca4 in import_find_and_load /build/python/cpython/Python/import.c:1765:11
        #97 0x11fbbc7 in PyImport_ImportModuleLevelObject /build/python/cpython/Python/import.c:1866:15
        #98 0x100dc88 in import_name /build/python/cpython/Python/ceval.c:5242:15
        #99 0xfc8157 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3126:19
        #100 0xf4068b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #101 0x1025062 in _PyEval_EvalCode /build/python/cpython/Python/ceval.c:4376:14
        #102 0x102a75f in _PyEval_EvalCodeWithName /build/python/cpython/Python/ceval.c:4408:12
        #103 0xf402c5 in PyEval_EvalCodeEx /build/python/cpython/Python/ceval.c:4424:12
        #104 0xf3ff2d in PyEval_EvalCode /build/python/cpython/Python/ceval.c:857:12
        #105 0x1f2cc4e in builtin_exec_impl /build/python/cpython/Python/bltinmodule.c:1035:13
        #106 0x1f1aa5d in builtin_exec /build/python/cpython/Python/clinic/bltinmodule.c.h:371:20
        #107 0x1caae6e in cfunction_vectorcall_FASTCALL /build/python/cpython/Objects/methodobject.c:424:24
        #108 0x58bcb5 in PyVectorcall_Call /build/python/cpython/Objects/call.c:230:16
        #109 0x58c3c6 in _PyObject_Call /build/python/cpython/Objects/call.c:265:16
        #110 0x58c8b1 in PyObject_Call /build/python/cpython/Objects/call.c:292:12
        #111 0x1017138 in do_call_core /build/python/cpython/Python/ceval.c:5141:9
        #112 0xfeada4 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3609:22
        #113 0xf4068b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #114 0x1025062 in _PyEval_EvalCode /build/python/cpython/Python/ceval.c:4376:14
        #115 0x58ec4e in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:395:12
        #116 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #117 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #118 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #119 0xfe5a21 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3516:23
        #120 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #121 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #122 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #123 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #124 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #125 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #126 0xfe5f00 in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3533:23
        #127 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #128 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #129 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #130 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #131 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #132 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #133 0xfe71eb in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3547:19
        #134 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #135 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #136 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #137 0x10365bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #138 0x1014541 in PyObject_Vectorcall /build/python/cpython/./Include/cpython/abstract.h:123:12
        #139 0x10151c6 in call_function /build/python/cpython/Python/ceval.c:5121:13
        #140 0xfe71eb in _PyEval_EvalFrameDefault /build/python/cpython/Python/ceval.c:3547:19
        #141 0x59721b in _PyEval_EvalFrame /build/python/cpython/./Include/internal/pycore_ceval.h:40:12
        #142 0x58f614 in function_code_fastcall /build/python/cpython/Objects/call.c:329:24
        #143 0x58d86d in _PyFunction_Vectorcall /build/python/cpython/Objects/call.c:366:20
        #144 0x5946bd in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:114:11
        #145 0x5959b3 in object_vacall /build/python/cpython/Objects/call.c:791:14
        #146 0x595f4c in _PyObject_CallMethodIdObjArgs /build/python/cpython/Objects/call.c:882:24
        #147 0x11ffca4 in import_find_and_load /build/python/cpython/Python/import.c:1765:11
        #148 0x11fbbc7 in PyImport_ImportModuleLevelObject /build/python/cpython/Python/import.c:1866:15
        #149 0x1f1601a in builtin___import__ /build/python/cpython/Python/bltinmodule.c:280:12
        #150 0x1cb2676 in cfunction_call /build/python/cpython/Objects/methodobject.c:537:18
        #151 0x5873a9 in _PyObject_MakeTpCall /build/python/cpython/Objects/call.c:191:18
        #152 0x594638 in _PyObject_VectorcallTstate /build/python/cpython/./Include/cpython/abstract.h:112:16
        #153 0x591aaa in _PyObject_CallFunctionVa /build/python/cpython/Objects/call.c:542:18
        #154 0x590ea2 in PyObject_CallFunction /build/python/cpython/Objects/call.c:564:14
        #155 0x11f8d6e in PyImport_Import /build/python/cpython/Python/import.c:2067:9
        #156 0x11eb740 in PyImport_ImportModule /build/python/cpython/Python/import.c:1482:14
        #157 0x12f10ac in init_sys_streams /build/python/cpython/Python/pylifecycle.c:1962:19
        #158 0x12eed4d in init_interp_main /build/python/cpython/Python/pylifecycle.c:1034:14
        #159 0x12e3727 in pyinit_main /build/python/cpython/Python/pylifecycle.c:1107:23
        #160 0x12e41cf in Py_InitializeFromConfig /build/python/cpython/Python/pylifecycle.c:1151:18
        #161 0x4e6b2d in pymain_init /build/python/cpython/Modules/main.c:66:14
        #162 0x4df188 in pymain_main /build/python/cpython/Modules/main.c:694:23
        #163 0x4df653 in Py_BytesMain /build/python/cpython/Modules/main.c:727:12
        #164 0x4dcec7 in main /build/python/cpython/./Programs/python.c:15:12
        #165 0x7fda2537d0b2 in __libc_start_main /build/glibc-YYA7BZ/glibc-2.31/csu/../csu/libc-start.c:308:16
        #166 0x43501d in _start (/build/python/cpython/python+0x43501d)

    0x625000264148 is located 16 bytes to the right of 8248-byte region [0x625000262100,0x625000264138)
    allocated by thread T0 here:
        #0 0x4ad75d in malloc (/build/python/cpython/python+0x4ad75d)
        #1 0x8afdec in _PyMem_RawMalloc /build/python/cpython/Objects/obmalloc.c:99:12
        #2 0x8b909e in _PyMem_DebugRawAlloc /build/python/cpython/Objects/obmalloc.c:2145:24
        #3 0x8b0a36 in _PyMem_DebugRawMalloc /build/python/cpython/Objects/obmalloc.c:2178:12
        #4 0x8b39e3 in _PyMem_DebugMalloc /build/python/cpython/Objects/obmalloc.c:2330:12
        #5 0x8b5b32 in PyMem_Malloc /build/python/cpython/Objects/obmalloc.c:605:12
        #6 0x1f906d2 in block_new /build/python/cpython/Python/pyarena.c:80:25
        #7 0x1f93300 in block_alloc /build/python/cpython/Python/pyarena.c:111:24
        #8 0x1f91bd1 in PyArena_Malloc /build/python/cpython/Python/pyarena.c:182:15
        #9 0x1cf79e7 in _Py_arg /build/python/cpython/Python/Python-ast.c:3516:17
        #10 0x233c6d3 in param_rule /build/python/cpython/Parser/parser.c:5946:20
        #11 0x2338044 in param_with_default_rule /build/python/cpython/Parser/parser.c:5790:18
        #12 0x2323c16 in _loop1_153_rule /build/python/cpython/Parser/parser.c:24561:39
        #13 0x2319647 in _tmp_134_rule /build/python/cpython/Parser/parser.c:23634:31
        #14 0x23093c5 in invalid_parameters_rule /build/python/cpython/Parser/parser.c:15128:29
        #15 0x22f6a43 in params_rule /build/python/cpython/Parser/parser.c:5124:39
        #16 0x2058782 in function_def_raw_rule /build/python/cpython/Parser/parser.c:4917:23
        #17 0x20229a4 in function_def_rule /build/python/cpython/Parser/parser.c:4856:37
        #18 0x200e2da in compound_stmt_rule /build/python/cpython/Parser/parser.c:1872:33
        #19 0x200a873 in statement_rule /build/python/cpython/Parser/parser.c:1234:18
        #20 0x2007026 in _loop1_11_rule /build/python/cpython/Parser/parser.c:16174:30
        #21 0x200455a in statements_rule /build/python/cpython/Parser/parser.c:1193:18
        #22 0x230193f in block_rule /build/python/cpython/Parser/parser.c:6257:18
        #23 0x2392ac3 in class_def_raw_rule /build/python/cpython/Parser/parser.c:6196:18
        #24 0x202fb74 in class_def_rule /build/python/cpython/Parser/parser.c:6139:34
        #25 0x2010e47 in compound_stmt_rule /build/python/cpython/Parser/parser.c:1914:30
        #26 0x200a873 in statement_rule /build/python/cpython/Parser/parser.c:1234:18
        #27 0x2007026 in _loop1_11_rule /build/python/cpython/Parser/parser.c:16174:30
        #28 0x200455a in statements_rule /build/python/cpython/Parser/parser.c:1193:18
        #29 0x230193f in block_rule /build/python/cpython/Parser/parser.c:6257:18

    SUMMARY: AddressSanitizer: heap-buffer-overflow /build/python/cpython/Parser/pegen.c:2253:61 in _PyPegen_collect_call_seqs
    Shadow bytes around the buggy address:
      0x0c4a800447d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x0c4a800447e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x0c4a800447f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x0c4a80044800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x0c4a80044810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    =>0x0c4a80044820: 00 00 00 00 00 00 00 fa fa[fa]fa fa fa fa fa fa
      0x0c4a80044830: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c4a80044840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c4a80044850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c4a80044860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c4a80044870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07 
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
      Shadow gap:              cc
    ==39814==ABORTING

In particular, the stack trace here in the ASAN output indicates that it's the `EXTRA_EXPR(first, last)` macro use in this line that's problematic:

    return _Py_Call(_PyPegen_dummy_name(p), args, keywords, EXTRA_EXPR(first, last));
msg376278 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 04:24
So that's the last line of the new helper function (in fact the end of the file). Maybe args can still be empty at this point?
msg376279 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 04:37
I do have a repro of the crash (with clang 11 no less). But it could be anywhere in sysconfig.py or anything it imports...
msg376280 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 04:41
Hm, it's not in sysconfig.py, it's in the next step:

./python.exe -E ./setup.py  build
msg376281 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 04:54
Looking at it in lldb, 'last' seems to be bogus.

  kind = 6961392
  ...
  lineno = -33686019
  col_offset = -33686019
  end_lineno = 0
  end_col_offset = 0

Here I'm stuck.
msg376283 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 04:54
Adding 3.9 since the offending commit was backported there.
msg376285 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 05:16
Smaller repro:

./python.exe -m configparser

Bisection shows that this is the call that causes the crash (line 1301):

        return _impl(self._name, option, raw=raw, vars=vars,
                     fallback=fallback, **kwargs)

But just that line doesn't trigger this -- apparently *everything* that came before it in that file is implicated. Or at least *something* that I haven't found yet.

Sorry for the play-by-play...
msg376294 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 10:46
I think I have the problem. The problem is that the asdl_seq that is b is filled with KeywordOrStarred* not with expr_ty.

We need to do:

KeywordOrStarred* last = asdl_seq_GET(b, asdl_seq_LEN(b)-1);

return _Py_Call(_PyPegen_dummy_name(p), args, keywords, EXTRA_EXPR(first, last->element));
msg376297 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 11:48
The reason this does not manifest itself without the address sanitizer is because that information is thrown away later, and the line and col numbers for the Call node end being correct.
msg376300 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 13:45
I have added a new builbot builder to detect these problems in the future more immediately:

https://buildbot.python.org/all/#/builders/582

For example, building the current master:

https://buildbot.python.org/all/#/builders/582/builds/1

And with PR 22077:

https://buildbot.python.org/all/#/builders/581/builds/1
msg376301 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 14:08
> return _Py_Call(_PyPegen_dummy_name(p), args, keywords, EXTRA_EXPR(first, last->element));

Actually, this is not enough because last->element may be a keyword_ty.

I have updated PR 22077 to receive the EXTRA macro in the call because that simplifies the code and allows us to not need to distinguish between keywords and expr_ty elements.
msg376303 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 14:29
New changeset 315a61f7a9418d904e0eea14b1f054fac3a90e9f by Pablo Galindo in branch 'master':
bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in the parser (GH-22077)
https://github.com/python/cpython/commit/315a61f7a9418d904e0eea14b1f054fac3a90e9f
msg376309 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 15:35
New changeset be17295280c89771c80f317da072f6c0d016cc60 by Pablo Galindo in branch '3.9':
[3.9] bpo-41697: Correctly handle KeywordOrStarred when parsing arguments in the parser (GH-22077) (GH-22079)
https://github.com/python/cpython/commit/be17295280c89771c80f317da072f6c0d016cc60
msg376310 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 16:52
Thanks for the report Brad, and thanks for the quick fix Pablo!
msg376323 - (view) Author: Brad Larsen (blarsen) * Date: 2020-09-03 21:44
Nice work with the quick fix!  I'm also happy to see the addition of the Linux ASAN builder -- that should help detect memory errors earlier on in the future.
msg376324 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 21:53
FWIW the bug was a classic type error -- in a decent language the asdl_seq
type would have been generic and this would have been caught without an
ASAN builder...
msg376327 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 22:28
Given that using asdl_seq currently means casting from void*, we could maybe have a set of macros like asdl_seq_GET that are type specialized (there aren't many of them) and in debug mode they can check that the ->kind attribute is consistent with the specific type that the macro represents. For instance asdl_seq_GET_EXPR will check that the ->kind attribute is consistent with a valid EXPR. It will not catch all problems but it will catch almost all the typical ones in simple sequences.
msg376328 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-03 22:38
Hm, I don't believe that will work -- each node type (e.g. `expr_ty`,
`mod_ty`) has its own enum for `kind` (e.g. `_expr_kind`, `_mod_kind`) and
some don't have a `kind` field at all (e.g. `keyword_ty`). So you'd have to
add an extra field to each node type and initialize that with yet another
enum that can be used to distinguish between node *types*. (There are other
options, but they all sound like a lot of work.)
msg376329 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-03 22:55
That's true. Is a bit sad because a considerable amount of problems we experienced with the new parser were due to invalid casts from these structures :(
History
Date User Action Args
2022-04-11 14:59:35adminsetnosy: + lukasz.langa
github: 85863
2020-09-03 23:55:02gvanrossumsetstage: resolved ->
2020-09-03 22:55:24pablogsalsetmessages: + msg376329
2020-09-03 22:38:14gvanrossumsetmessages: + msg376328
2020-09-03 22:28:38pablogsalsetmessages: + msg376327
2020-09-03 21:53:05gvanrossumsetmessages: + msg376324
2020-09-03 21:44:11blarsensetmessages: + msg376323
2020-09-03 16:52:57gvanrossumsetmessages: + msg376310
2020-09-03 15:47:26pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-09-03 15:35:24pablogsalsetmessages: + msg376309
2020-09-03 14:32:38pablogsalsetpull_requests: + pull_request21166
2020-09-03 14:29:43pablogsalsetmessages: + msg376303
2020-09-03 14:08:44pablogsalsetmessages: + msg376301
2020-09-03 13:45:08pablogsalsetmessages: + msg376300
2020-09-03 11:48:10pablogsalsetmessages: + msg376297
2020-09-03 10:48:19pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request21164
2020-09-03 10:46:42pablogsalsetmessages: + msg376294
2020-09-03 05:16:19gvanrossumsetmessages: + msg376285
2020-09-03 04:54:57gvanrossumsetmessages: + msg376283
versions: + Python 3.9
2020-09-03 04:54:32gvanrossumsetmessages: + msg376281
2020-09-03 04:41:41gvanrossumsetmessages: + msg376280
2020-09-03 04:37:33gvanrossumsetmessages: + msg376279
2020-09-03 04:24:31gvanrossumsetpriority: normal -> release blocker
assignee: pablogsal
messages: + msg376278
2020-09-03 01:18:07blarsencreate