This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author pablogsal
Recipients BTaskaya, elmanto, gvanrossum, lys.nikolaou, pablogsal
Date 2021-06-11.16:54:50
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1623430490.39.0.149382539908.issue44396@roundup.psfhosted.org>
In-reply-to
Content
I think this should fix the issue, but someone should validate this:

diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
index 6002f3e05a..1c28737183 100644
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -1084,17 +1084,16 @@ syntaxerror(struct tok_state *tok, const char *format, ...)
         goto error;
     }

-    errtext = PyUnicode_DecodeUTF8(tok->line_start, tok->cur - tok->line_start,
+    errtext = PyUnicode_DecodeUTF8(tok->buf, tok->inp - tok->buf,
                                    "replace");
     if (!errtext) {
         goto error;
     }
     int offset = (int)PyUnicode_GET_LENGTH(errtext);
-    Py_ssize_t line_len = strcspn(tok->line_start, "\n");
-    if (line_len != tok->cur - tok->line_start) {
+    Py_ssize_t line_len = strcspn(tok->buf, "\n");
+    if (line_len != tok->buf - tok->inp) {
         Py_DECREF(errtext);
-        errtext = PyUnicode_DecodeUTF8(tok->line_start, line_len,
-                                       "replace");
+        errtext = PyUnicode_DecodeUTF8(tok->buf, line_len, "replace");
     }
     if (!errtext) {
         goto error;
History
Date User Action Args
2021-06-11 16:54:50pablogsalsetrecipients: + pablogsal, gvanrossum, lys.nikolaou, BTaskaya, elmanto
2021-06-11 16:54:50pablogsalsetmessageid: <1623430490.39.0.149382539908.issue44396@roundup.psfhosted.org>
2021-06-11 16:54:50pablogsallinkissue44396 messages
2021-06-11 16:54:50pablogsalcreate