classification
Title: Update doctect SyntaxErrors for location range
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: andrei.avk, blueyed, kj, miss-islington, pablogsal, terry.reedy
Priority: low Keywords: patch

Created on 2021-09-20 16:30 by andrei.avk, last changed 2021-10-16 17:51 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28567 closed andrei.avk, 2021-09-26 14:10
PR 28575 closed pablogsal, 2021-09-26 22:59
PR 28587 merged miss-islington, 2021-09-27 20:59
PR 28854 closed pablogsal, 2021-10-10 15:13
PR 28855 merged pablogsal, 2021-10-10 15:15
PR 28994 merged miss-islington, 2021-10-16 17:27
Messages (18)
msg402257 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-20 16:30
It seems like fine grained error locations do not work in failed doctest traceback output:

version 3.11.0a0

file contents:
------------------
def a(x):
    """
    >>> 1 1
    1
    """
import doctest
doctest.testmod()

OUTPUT
-------

Failed example:
    1 1
Exception raised:
    Traceback (most recent call last):
      File "/Users/ak/opensource/cpython/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<doctest __main__.a[0]>", line 1
        1 1
        ^
    SyntaxError: invalid syntax. Perhaps you forgot a comma?


The location in doctests that causes this:

https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/doctest.py#L1348
msg402258 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-20 16:34
I've ran into this when looking at doctest docs, the issue is that they use the old example where a single column is highlighted, I want to update it to explain why doctest output differs from the one you get from REPL, but I probably need to understand why it happens to provide a good explanation.

Alternatively this may be fixed to be consistent if it's easy enough to do.
msg402260 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-20 16:39
Hummmm, could you explain a bit more in detail what is the expected output? I can see highlighting in the exec call that you pasted:

        exec(compile(example.source, filename, "single",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The fact that you see those "^^^^^^" indicate that is working no? What is missing?
msg402263 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-20 18:01
Sorry, I should have noted I’m referring to the line 1 1
Which is underlined by a single caret, but on the command line it has 3
carets as expected.
msg402268 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-20 18:31
Ah, but that is a different issue. This is not PEP 657, this is a SyntaxError, so is related how those are printed, which I think is separared.
msg402269 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-20 18:31
Can you try a doctest that fails on something that is not a SyntaxError.

Something like:

>>> def foo(x):
...    return x + 42

>>> foo(None)
msg402271 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-20 18:45
Pablo: that works fine, thanks!

I will look into updating the doctest docs, and will close this issue later
today ( or you can close it if you like).
msg402284 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-21 02:17
Thinking a bit more on this, I'm not sure this can be closed, as SyntaxError indicators seems not to be working. I'm not sure if this is limited to doctests or not. I've updated the title to narrow it down to SyntaxErrors.

I can look more into this if needed. If it's just doctests, it doesn't matter because the doctests ignore the indicators anyway, but if it can happen in some other cases, perhaps it's worth fixing.

However I don't have experience with the C parts of python, so I'm not sure where / how to look.

Pablo: let me know if you have any pointers on how to debug this, or if you think this should be closed.
msg402589 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-09-24 22:13
In 3.10+, end_lineno and end_offset fields were added to SyntaxError objects and the args tuple.

>>> try: compile('1 1', '', 'single')
... except SyntaxError as e: print(e.args)
... 
('invalid syntax. Perhaps you forgot a comma?', ('', 1, 1, '1 1', 1, 4))

Here, line 1, offset 4 is the first 1-based column not part of the error.

The builtin default sys.excepthook was modified to read and use this new information and mark (end_offset - offset) columns with '^'s.  This default prints what it does to sys.stderr.

The syntax error formatting in the traceback module was not altered.  However, a new method, TracebackException._format_syntax_error, was extracted from TracebackException.format_exception_only so that the former could be overridden by software that simulates interaction.

The printed traceback does not come from line 1348.  That *executes* the user code, but all Exceptions, including SyntaxError, are caught.  If the exception is not expected and the run is not quiet, the exception is output by report_unexpected_exception(), as seen above as 'OUTPUT' and the lines that follows.
https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/doctest.py#L1264

This calls _exception_traceback(exc_info).
https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/doctest.py#L244
This calls traceback.print_exception, which I believe, for syntax errors, ultimately calls TracebackException._format_syntax_error.

I believe that the options for a fix are either
1. Call default sys.excepthook while capturing its output into a StringIO instance.
2. Assuming I am correct above about _format_syntax_error being called, monkeypatch it.  In line 779,
https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/traceback.py#L779
replace '^' with a field with a calculated number of ^s.

I need to do one of these two for IDLE, and may try both.
msg402652 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-25 22:09
Terry: I got it mostly working using your 2nd suggestion, I will do some testing and should be able to put up a PR in the next couple of days. Thanks for looking at this and explaining!
msg402662 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-09-26 14:37
Terry: please take a look - https://github.com/python/cpython/pull/28567/files .
msg402670 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-26 16:58
One important thing is that "traceback.print_exception" should correctly print syntax errors .
msg402739 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-27 20:59
New changeset 20f439b6b9e1032930a31b88694ab9f37a09e6b4 by Pablo Galindo Salgado in branch 'main':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
https://github.com/python/cpython/commit/20f439b6b9e1032930a31b88694ab9f37a09e6b4
msg402742 - (view) Author: miss-islington (miss-islington) Date: 2021-09-27 21:26
New changeset c7fdd6879b5c8447ee65b1bec95a1db43837a7a5 by Miss Islington (bot) in branch '3.10':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
https://github.com/python/cpython/commit/c7fdd6879b5c8447ee65b1bec95a1db43837a7a5
msg403157 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-10-04 19:18
New changeset 1e2058214fffcb3919e0e127145106ade40a0420 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
https://github.com/python/cpython/commit/1e2058214fffcb3919e0e127145106ade40a0420
msg403588 - (view) Author: daniel hahler (blueyed) * Date: 2021-10-10 14:22
I've noticed a regression/change with the code change for this issue.

When not catching the exception from `compile("invalid(", "<stdin>", "single")` it has a caret below the opening parenthesis:

```
Traceback (most recent call last):
  File "…/t-syntaxerror-chained.py", line 2, in <module>
    compile("invalid(", "<stdin>", "single")
  File "<stdin>", line 1
    invalid(
           ^
SyntaxError: '(' was never closed
```

When using `traceback.print_exc` however this is missing:
```
Traceback (most recent call last):
  File "…/t-syntaxerror-chained.py", line 2, in <module>
    compile("invalid(", "<stdin>", "single")
  File "<stdin>", line 1
    invalid(

SyntaxError: '(' was never closed
```

The file used for testing:
```
try:
    compile("invalid(", "<stdin>", "single")
except Exception:
    # raise
    __import__("traceback").print_exc()
```

(this change was noticed between 3.10.0rc2 and the final release with pdbpp's test suite)

I've not investigated further (yet), and also feel free to ask for creating a new issue, but I've figured it would be good to notify you here first (where the code was changed).
msg404103 - (view) Author: miss-islington (miss-islington) Date: 2021-10-16 17:27
New changeset fe0d9e22a52a10c4cbe52254b51f2d4e74d83568 by Pablo Galindo Salgado in branch 'main':
bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855)
https://github.com/python/cpython/commit/fe0d9e22a52a10c4cbe52254b51f2d4e74d83568
msg404106 - (view) Author: miss-islington (miss-islington) Date: 2021-10-16 17:51
New changeset 5df35faf3611b459f7f32bfe9fd2ffa7fb2dc59e by Miss Islington (bot) in branch '3.10':
bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855)
https://github.com/python/cpython/commit/5df35faf3611b459f7f32bfe9fd2ffa7fb2dc59e
History
Date User Action Args
2021-10-16 17:51:11miss-islingtonsetmessages: + msg404106
2021-10-16 17:37:28pablogsalsetstatus: open -> closed
stage: patch review -> resolved
2021-10-16 17:27:55miss-islingtonsetpull_requests: + pull_request27277
2021-10-16 17:27:51miss-islingtonsetmessages: + msg404103
2021-10-10 15:15:39pablogsalsetpull_requests: + pull_request27162
2021-10-10 15:13:21pablogsalsetstage: resolved -> patch review
pull_requests: + pull_request27161
2021-10-10 15:08:06pablogsalsetstatus: closed -> open
2021-10-10 14:23:32blueyedsetversions: + Python 3.10
2021-10-10 14:22:42blueyedsetnosy: + blueyed
messages: + msg403588
2021-10-04 19:18:42pablogsalsetmessages: + msg403157
2021-09-29 15:48:28pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-09-27 21:26:47miss-islingtonsetmessages: + msg402742
2021-09-27 20:59:26pablogsalsetmessages: + msg402739
2021-09-27 20:59:14miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26969
2021-09-26 22:59:15pablogsalsetpull_requests: + pull_request26958
2021-09-26 16:58:33pablogsalsetmessages: + msg402670
2021-09-26 14:37:48andrei.avksetmessages: + msg402662
2021-09-26 14:10:12andrei.avksetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request26951
2021-09-25 22:09:13andrei.avksetmessages: + msg402652
2021-09-24 22:13:12terry.reedysettitle: SyntaxError location range indicator does not work in doctests -> Update doctect SyntaxErrors for location range
nosy: + terry.reedy

messages: + msg402589

components: + Library (Lib), - Interpreter Core, Tests
stage: needs patch
2021-09-21 02:48:39andrei.avksettitle: Syntax error location range indicator does not work in doctests -> SyntaxError location range indicator does not work in doctests
2021-09-21 02:17:16andrei.avksetmessages: + msg402284
title: Fine grained error locations do not work in doctests -> Syntax error location range indicator does not work in doctests
2021-09-20 18:45:19andrei.avksetmessages: + msg402271
2021-09-20 18:31:53pablogsalsetmessages: + msg402269
2021-09-20 18:31:13pablogsalsetmessages: + msg402268
2021-09-20 18:01:35andrei.avksetmessages: + msg402263
2021-09-20 16:39:48pablogsalsetmessages: + msg402260
2021-09-20 16:34:13andrei.avksetmessages: + msg402258
2021-09-20 16:30:58andrei.avkcreate