classification
Title: Indentation.offset and SyntaxError.offset mismatch
Type: behavior Stage: commit review
Components: Interpreter Core, Library (Lib) Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, flox, georg.brandl, python-dev, r.david.murray, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2013-04-24 07:46 by flox, last changed 2014-01-22 01:01 by flox. This issue is now closed.

Files
File name Uploaded Description Edit
issue17825_cases.py flox, 2013-04-27 18:51
test_indenterror.py flox, 2013-04-27 18:52
patch_indenterror_offset.diff flox, 2013-04-27 20:31 review
test_traceback_caret.py flox, 2013-04-27 21:07
patch_indenterror_offset_v2.diff flox, 2013-04-30 21:06 review
Messages (17)
msg187690 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2013-04-24 07:46
I noticed a difference between computation of column offset for SyntaxError and IndentationError (a subclass of SyntaxError).
It is slightly confusing.


def report(exc):
    print('lineno %s, offset %s' % (exc.lineno, exc.offset))
    raise

try:
    exec('except IOError:')
except Exception as e:
    report(e)

try:
    exec('    open(filepath)')
except Exception as e:
    report(e)


** OUTPUT **
lineno 1, offset 6
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "<stdin>", line 2, in <module>
  File "<string>", line 1
    except IOError:
         ^
SyntaxError: invalid syntax

lineno 1, offset 4
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "<stdin>", line 2, in <module>
  File "<string>", line 1
    open(filepath)
    ^
IndentationError: unexpected indent


** Analysis **

Case (1): offset is 6, and caret is below column 5 for SyntaxError
Case (2): offset is 4, and caret is below column 4 for IndentationError
msg187703 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-24 13:56
Well, that would be because the invalid indent is detected when the 'o' is encountered, whereas the invalid syntax is detected when the 't' completes the keyword 'except'.

I don't think there is a bug here.
msg187879 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-26 21:58
Perhaps Florent is pointing to the fact that for 'except X:', the syntax error is detected at the ' ' at offset 6 but the caret is backed up under the 't' at offset 5*, whereas it is not backed up to the space before '    open'. Which behavior do you propose to change?

Since these details of presentation are not documented, I do not consider the current difference a bug. I would not change them in current versions since change could confuse people or software. I suspect there is also danger of making some cases worse. For the same reasons, I would not change in future versions either without good offsetting reason. So I also think this should be closed.

On python-list, we tell people that the caret is a starting point for detecting the error, not an absolute pointer.

For syntax errors, Idle omits the traceback and highlights in red the presumed error. For the first, it marks the entire word 'except'. For the second, it marks the last whitespace before open. This is equivalent to backing up the caret.

* I suspect this backup is special to keywords. For 'except+', both systems also mark the keyword, not the '+'. Either the keyword is misused or is a regular name misspelled. In either case, someone thought it better to point to the keyword than the space or other character after. I agree. So did the Idle author who decided to highlight the keyword rather than the character after it.
msg187915 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-04-27 16:49
I tried a few different lines that result in a syntax error and the position of the '^' always seems ok to me, so I'm not sure there's any bug here:

    foo:bar
       ^
    from import *
              ^
    try:+
        ^
    try+
       ^
    1?
     ^
    1+*1
      ^
    1+-*1
       ^
    [}]
     ^
    foo)
       ^
msg187925 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2013-04-27 18:51
Hello again,

actually, the attached script demonstrates some issues with the computation of the offset.

In rare cases, the computed offset does not match the position of the caret '^' in the traceback. It makes it difficult to reuse exc.offset to display the exact position.
msg187926 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2013-04-27 18:52
And this additional script (test_indenterror.py) demonstrates a different issue in the traceback module.
msg187928 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-27 19:56
Indeed, there is an inconsistence between the traceback module and builtin exception print function. Behavior of traceback is intentional as noted in format_exception_only() (if this comment is not misleading).
msg187932 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2013-04-27 20:27
The proposed patch deals with the case of the IndentationError in the traceback module.
msg187933 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-04-27 20:32
Your diff file looks broken.

I'm not sure it's the traceback module should be fixed and not Python internals. Georg, what you say?
msg187935 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2013-04-27 21:07
The previous patch fixes some cases where the caret was wrongly positioned.

However, there's probably stuff to fix in Python internals too.
See attached test cases (test_traceback_caret.py) which demonstrates other issues with non-ASCII chars or <tab> in the source code.
msg188185 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2013-04-30 21:06
Patch updated with latest changes of trunk.
msg206420 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-17 11:42
patch_indenterror_offset_v2.diff LGTM.
msg207871 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-01-10 17:16
Florent, can you commit your patch?
msg208722 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-22 00:12
New changeset 6d1372237607 by Florent Xicluna in branch '3.3':
Issue #17825: Cursor ^ is correctly positioned for SyntaxError and IndentationError.
http://hg.python.org/cpython/rev/6d1372237607
msg208723 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-22 00:17
New changeset aeb204b8f6c4 by Florent Xicluna in branch 'default':
Issue #17825: Cursor ^ is correctly positioned for SyntaxError and IndentationError.
http://hg.python.org/cpython/rev/aeb204b8f6c4
msg208725 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-22 00:34
New changeset 0041be34edbf by Florent Xicluna in branch '2.7':
Issue #17825: Cursor ^ is correctly positioned for SyntaxError and IndentationError.
http://hg.python.org/cpython/rev/0041be34edbf
msg208727 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2014-01-22 01:01
Done.

The alignment error with non-ASCII chars should be fixed too, thanks to the work of Serhiy on issue #2382.
History
Date User Action Args
2014-01-22 01:01:41floxsetstatus: open -> closed
resolution: fixed
messages: + msg208727
2014-01-22 00:34:13python-devsetmessages: + msg208725
2014-01-22 00:17:09python-devsetmessages: + msg208723
2014-01-22 00:12:07python-devsetnosy: + python-dev
messages: + msg208722
2014-01-10 17:16:05serhiy.storchakasetmessages: + msg207871
2013-12-17 11:42:12serhiy.storchakasetstage: commit review
messages: + msg206420
versions: + Python 2.7, Python 3.3
2013-04-30 21:06:45floxsetfiles: + patch_indenterror_offset_v2.diff

messages: + msg188185
2013-04-28 08:46:44floxsetcomponents: + Interpreter Core, Library (Lib)
2013-04-27 21:07:55floxsetfiles: + test_traceback_caret.py

messages: + msg187935
2013-04-27 20:32:37serhiy.storchakasetmessages: + msg187933
2013-04-27 20:31:38floxsetfiles: + patch_indenterror_offset.diff
2013-04-27 20:31:00floxsetfiles: - patch_indenterror_offset.diff
2013-04-27 20:27:58floxsetfiles: + patch_indenterror_offset.diff
resolution: works for me -> (no value)
messages: + msg187932

keywords: + patch
2013-04-27 19:56:06serhiy.storchakasetnosy: + serhiy.storchaka, georg.brandl
type: enhancement -> behavior
messages: + msg187928
2013-04-27 18:55:28floxsetstatus: closed -> open
2013-04-27 18:52:47floxsetfiles: + test_indenterror.py

messages: + msg187926
2013-04-27 18:51:27floxsetfiles: + issue17825_cases.py

messages: + msg187925
2013-04-27 18:40:19terry.reedysetstatus: open -> closed
type: behavior -> enhancement
resolution: works for me
2013-04-27 16:49:24ezio.melottisetnosy: + ezio.melotti

messages: + msg187915
versions: + Python 3.4, - Python 2.7, Python 3.3
2013-04-26 21:58:17terry.reedysetnosy: + terry.reedy
messages: + msg187879
2013-04-24 13:56:23r.david.murraysetnosy: + r.david.murray
messages: + msg187703
2013-04-24 07:46:22floxcreate