Use After Free in python3.11 (commit 2ab27c4)
Steps to reproduce:

1. ./configure --with-address-sanitizer
2. make
3. ./python <input>

I attach some of the input that lead to the undefined behavior

For the complete description you can find the asan report here:

==1082579==ERROR: AddressSanitizer: heap-use-after-free on address 0x626000045a40 at pc 0x000000735155 bp 0x7fffffffbed0 sp 0x7fffffffbec8
READ of size 8 at 0x626000045a40 thread T0
#0 0x735154 in ascii_decode /home/elmanto/ddg/other_targets/cpython/Objects/unicodeobject.c:5091:28
#1 0x735154 in unicode_decode_utf8 /home/elmanto/ddg/other_targets/cpython/Objects/unicodeobject.c:5158:10
#2 0xc98381 in syntaxerror /home/elmanto/ddg/other_targets/cpython/Parser/tokenizer.c:1087:15
#3 0xc8d616 in tok_get /home/elmanto/ddg/other_targets/cpython/Parser/tokenizer.c
#4 0xc8696b in PyTokenizer_Get /home/elmanto/ddg/other_targets/cpython/Parser/tokenizer.c:1884:18
#5 0xead74c in _PyPegen_check_tokenizer_errors /home/elmanto/ddg/other_targets/cpython/Parser/pegen.c:1260:17
#6 0xead74c in _PyPegen_run_parser /home/elmanto/ddg/other_targets/cpython/Parser/pegen.c:1292:17
#7 0xeaebca in _PyPegen_run_parser_from_file_pointer /home/elmanto/ddg/other_targets/cpython/Parser/pegen.c:1377:14
#8 0xc83a91 in _PyParser_ASTFromFile /home/elmanto/ddg/other_targets/cpython/Parser/peg_api.c:26:12
#9 0xa0abf1 in pyrun_file /home/elmanto/ddg/other_targets/cpython/Python/pythonrun.c:1197:11
#10 0xa0abf1 in _PyRun_SimpleFileObject /home/elmanto/ddg/other_targets/cpython/Python/pythonrun.c:455:13
#11 0xa09b19 in _PyRun_AnyFileObject /home/elmanto/ddg/other_targets/cpython/Python/pythonrun.c:89:15
#12 0x4dfe94 in pymain_run_file_obj /home/elmanto/ddg/other_targets/cpython/Modules/main.c:353:15
#13 0x4dfe94 in pymain_run_file /home/elmanto/ddg/other_targets/cpython/Modules/main.c:372:15
#14 0x4dfe94 in pymain_run_python /home/elmanto/ddg/other_targets/cpython/Modules/main.c:587:21
#15 0x4dfe94 in Py_RunMain /home/elmanto/ddg/other_targets/cpython/Modules/main.c:666:5
#16 0x4e154c in pymain_main /home/elmanto/ddg/other_targets/cpython/Modules/main.c:696:12
#17 0x4e1874 in Py_BytesMain /home/elmanto/ddg/other_targets/cpython/Modules/main.c:720:12
#18 0x7ffff7a2e0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16
#19 0x43501d in _start (/home/elmanto/ddg/other_targets/cpython/python+0x43501d)

0x626000045a40 is located 2368 bytes inside of 10560-byte region [0x626000045100,0x626000047a40)
freed by thread T0 here:
#0 0x4ada79 in realloc (/home/elmanto/ddg/other_targets/cpython/python+0x4ada79)
#1 0x638e61 in PyMem_RawRealloc /home/elmanto/ddg/other_targets/cpython/Objects/obmalloc.c:602:12
#2 0x638e61 in _PyObject_Realloc /home/elmanto/ddg/other_targets/cpython/Objects/obmalloc.c:2339:12

previously allocated by thread T0 here:
#0 0x4ada79 in realloc (/home/elmanto/ddg/other_targets/cpython/python+0x4ada79)
#1 0x638e61 in PyMem_RawRealloc /home/elmanto/ddg/other_targets/cpython/Objects/obmalloc.c:602:12
#2 0x638e61 in _PyObject_Realloc /home/elmanto/ddg/other_targets/cpython/Objects/obmalloc.c:2339:12

SUMMARY: AddressSanitizer: heap-use-after-free /home/elmanto/ddg/other_targets/cpython/Objects/unicodeobject.c:5091:28 in ascii_decode
Shadow bytes around the buggy address:
0x0c4c80000af0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c4c80000b40: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
0x0c4c80000b50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c4c80000b90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==1082579==ABORTING

Lysandros and Pablo, this *only* occurs when the lexer is reading directly from a file, not when it's reading the same source code from a (bytes) string. All examples are syntax errors (some raise ValueError in the parser).

Here is a smaller reproducer:

x = "ijosdfsd\
def blech():
    pass

This seems to be an error with:

commit a698d52
Author: Batuhan Taskaya <isidentical@gmail.com>
Date: Thu Jan 21 00:38:47 2021 +0300

bpo-40176: Improve error messages for unclosed string literals (GH-19346)



Automerge-Triggered-By: GH:isidentical

Batuhan, can you take a look?

I think this should fix the issue, but someone should validate this:

diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
index 6002f3e05a..1c28737183 100644
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -1084,17 +1084,16 @@ syntaxerror(struct tok_state *tok, const char *format, ...)
goto error;
}

-    errtext = PyUnicode_DecodeUTF8(tok->line_start, tok->cur - tok->line_start,
+    errtext = PyUnicode_DecodeUTF8(tok->buf, tok->inp - tok->buf,
                                    "replace");
     if (!errtext) {
         goto error;
     }
     int offset = (int)PyUnicode_GET_LENGTH(errtext);
-    Py_ssize_t line_len = strcspn(tok->line_start, "\n");
-    if (line_len != tok->cur - tok->line_start) {
+    Py_ssize_t line_len = strcspn(tok->buf, "\n");
+    if (line_len != tok->buf - tok->inp) {
         Py_DECREF(errtext);
-        errtext = PyUnicode_DecodeUTF8(tok->line_start, line_len,
-                                       "replace");
+        errtext = PyUnicode_DecodeUTF8(tok->buf, line_len, "replace");
     }
     if (!errtext) {
         goto error;

This affects 3.10 as well

Ok, found the problem, we are not resetting the multi-line-start pointer when we are reallocating the tokenizer buffers.

New changeset a342cc5 by Pablo Galindo in branch 'main':
bpo-44396: Update multi-line-start location when reallocating tokenizer buffers (GH-26676)
a342cc5

alessandro mantovani, one question, how did you generate the crash scripts?

New changeset d03f342 by Miss Islington (bot) in branch '3.10':
bpo-44396: Update multi-line-start location when reallocating tokenizer buffers (GH-26676) (GH-26695)
d03f342

Fuzzing experimental techniques, but then I observed the same behavior was happening with vanilla afl++. As a starting queue I used the *.py files that I found in the repo under ‘test’ or so

Best

Alessandro Mantovani

Inviato da iPhone

Il giorno 12.06.2021, alle ore 19:57, Pablo Galindo Salgado <report@bugs.python.org> ha scritto:


Pablo Galindo Salgado <pablogsal@gmail.com> added the comment:

alessandro mantovani, one question, how did you generate the crash scripts?

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue44396\>

---