Title: Fix attributes of syntax errors raized in the tokenizer
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2020-01-05 11:33 by serhiy.storchaka, last changed 2020-02-12 10:57 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17828 merged serhiy.storchaka, 2020-01-11 10:51
PR 18479 merged miss-islington, 2020-02-12 10:17
Messages (5)
msg359331 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-01-05 11:33
SyntaxError can be raised at different stages of compiling. In some places the source text is not available and should be read from the file using the file name. Which does not work in case of compiling a string or reading from stdin.

>>> 0z
  File "<stdin>", line 1
SyntaxError: invalid syntax
>>> 0xz
  File "<stdin>", line 1
SyntaxError: invalid hexadecimal literal

In the second example above the source line and the caret are absent in REPL.

The proposed PR fixes two errors in raising a SyntaxError in the tokenizer.

1. The text of the source line was not set if an exception was raised in the tokenizer. Since most of these exceptions (with more detailed description) were added in 3.8 I consider this a regressions.

2. The offset attribute was an offset in bytes. Now it is an offset in characters.

It only fixes errors in the tokenizer. There are similar bugs in other parts of the compiler.

This issue is based on the article .
msg359769 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-01-10 23:29
On entry of '0xz', IDLE from 3.6 to date highlights the '0x' part of the original entry.  I presume it can do this because it ignores the text attribute, and because bytes == chars for at least  '0x'.

The 'proposed PR' is not listed here.  Is it not yet submitted, or is the issue number wrong?
msg359791 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-01-11 10:59
Sorry, wrong issue number.
msg361873 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-02-12 10:17
New changeset 0cc6b5e559b8303b18fdd56c2befd900fe7b5e35 by Serhiy Storchaka in branch 'master':
bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828)
msg361875 - (view) Author: miss-islington (miss-islington) Date: 2020-02-12 10:35
New changeset efd878cdb46d9c7038d93fb36eb1ff7dc5baf9ec by Miss Islington (bot) in branch '3.8':
bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828)
Date User Action Args
2020-02-12 10:57:48serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-02-12 10:35:14miss-islingtonsetnosy: + miss-islington
messages: + msg361875
2020-02-12 10:17:14miss-islingtonsetpull_requests: + pull_request17849
2020-02-12 10:17:07serhiy.storchakasetmessages: + msg361873
2020-01-11 10:59:53serhiy.storchakasetmessages: + msg359791
2020-01-11 10:51:31serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request17360
2020-01-10 23:29:00terry.reedysetnosy: + terry.reedy
messages: + msg359769
2020-01-05 11:33:45serhiy.storchakacreate