classification
Title: Enrich SyntaxError with additional information
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, alonho, pablogsal, terry.reedy
Priority: normal Keywords:

Created on 2013-09-23 20:42 by alonho, last changed 2021-04-02 22:51 by terry.reedy. This issue is now closed.

Messages (4)
msg198341 - (view) Author: alon horev (alonho) * Date: 2013-09-23 20:42
Some context for this feature request:
I'm using the wonderful ast module for a library that translates python code to MongoDB queries (https://github.com/alonho/pql). I also did the same for SQL queries using sqlalchemy as a part of another project (https://github.com/alonho/pytrace).

One of the things I find lacking in python's parser is additional information about SyntaxErrors. This could help users of the 'ast' module, IDE and developers.

Here are some examples of what I'd like to see
1. ast.parse('* 2') -> SyntaxError('Unexpected operator at start of an expression')
2. ast.parse('2 *') -> SyntaxError('Missing right hand side operand')
3. ast.parse('if a = 1: pass') -> SyntaxError('Cannot assign inside an expression')

There are several challenges here:
1. Does the parser have this information and doesn't surface it?
2. Can such messages be automatically generated without filling the code with error handling code? 3. Which part of the code could be responsible for this kind of a task? I've looked at the BNF and it contains more than just syntax legality but operator precedence and such. Perhaps there's another (simpler) grammar definition somewhere?

I was curious to see what Ruby does, and it uses a simple solution of providing raw information along with the exception:
>> a == * 1
SyntaxError: compile error
(irb):17: syntax error, unexpected tSTAR
a == * 1
      ^
	from (irb):17
msg198555 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-28 21:11
Currently,
>>> * 2 # or * a
SyntaxError: can use starred expression only as assignment target
This conflicts with your desired message. It cannot change because *a, at least, is a valid statement prefix in a way that '/ a' is not.

Most other things give an uninformative generic 'invalid syntax' message (plus a more informative location indicator). If you can find a way to improve this, lots of people would be happy ;-).
msg356928 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2019-11-18 22:59
I am not sure about how to implement this to the core, but you can combine tokenize with ast to reproduce this kind of messages in your projects. You can match tokens with SyntaxErrors by lineno and offset.
msg390110 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-04-02 22:51
I am closing because 2 of your 3 examples (and many others) have had the messages changed in an effort to be more informative.  But Pablo can reverse this if he wants.

In 3.10:

>>> * 2
SyntaxError: can't use starred expression here

+2 and -2 are also valid expressions.  /2 is not and still gets the old message.  


>>> 2 *
SyntaxError: invalid syntax  # Same.


>>> if a=1: pass
SyntaxError: expected ':'  # At '=', which *is* an error.

Also not what you you suggested, which points to the impossibility of guessing what the coder meant.
History
Date User Action Args
2021-04-02 22:51:09terry.reedysetstatus: open -> closed

nosy: + pablogsal
messages: + msg390110

resolution: out of date
stage: resolved
2019-11-18 22:59:34BTaskayasetnosy: + BTaskaya
messages: + msg356928
2013-09-28 21:11:39terry.reedysetnosy: + terry.reedy

messages: + msg198555
versions: + Python 3.4
2013-09-23 20:42:05alonhocreate