It seems like fine grained error locations do not work in failed doctest traceback output:

version 3.11.0a0

file contents:
------------------
def a(x):
    """
    >>> 1 1
    1
    """
import doctest
doctest.testmod()

OUTPUT
-------

Failed example:
    1 1
Exception raised:
    Traceback (most recent call last):
      File "/Users/ak/opensource/cpython/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<doctest __main__.a[0]>", line 1
        1 1
        ^
    SyntaxError: invalid syntax. Perhaps you forgot a comma?


The location in doctests that causes this:

https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/doctest.py#L1348

I've ran into this when looking at doctest docs, the issue is that they use the old example where a single column is highlighted, I want to update it to explain why doctest output differs from the one you get from REPL, but I probably need to understand why it happens to provide a good explanation.

Alternatively this may be fixed to be consistent if it's easy enough to do.

Hummmm, could you explain a bit more in detail what is the expected output? I can see highlighting in the exec call that you pasted:

        exec(compile(example.source, filename, "single",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The fact that you see those "^^^^^^" indicate that is working no? What is missing?

Sorry, I should have noted I’m referring to the line 1 1
Which is underlined by a single caret, but on the command line it has 3
carets as expected.

Ah, but that is a different issue. This is not PEP 657, this is a SyntaxError, so is related how those are printed, which I think is separared.

Can you try a doctest that fails on something that is not a SyntaxError.

Something like:

>>> def foo(x):
...    return x + 42

>>> foo(None)

Pablo: that works fine, thanks!

I will look into updating the doctest docs, and will close this issue later
today ( or you can close it if you like).

Thinking a bit more on this, I'm not sure this can be closed, as SyntaxError indicators seems not to be working. I'm not sure if this is limited to doctests or not. I've updated the title to narrow it down to SyntaxErrors.

I can look more into this if needed. If it's just doctests, it doesn't matter because the doctests ignore the indicators anyway, but if it can happen in some other cases, perhaps it's worth fixing.

However I don't have experience with the C parts of python, so I'm not sure where / how to look.

Pablo: let me know if you have any pointers on how to debug this, or if you think this should be closed.

In 3.10+, end_lineno and end_offset fields were added to SyntaxError objects and the args tuple.

>>> try: compile('1 1', '', 'single')
... except SyntaxError as e: print(e.args)
... 
('invalid syntax. Perhaps you forgot a comma?', ('', 1, 1, '1 1', 1, 4))

Here, line 1, offset 4 is the first 1-based column not part of the error.

The builtin default sys.excepthook was modified to read and use this new information and mark (end_offset - offset) columns with '^'s.  This default prints what it does to sys.stderr.

The syntax error formatting in the traceback module was not altered.  However, a new method, TracebackException._format_syntax_error, was extracted from TracebackException.format_exception_only so that the former could be overridden by software that simulates interaction.

The printed traceback does not come from line 1348.  That *executes* the user code, but all Exceptions, including SyntaxError, are caught.  If the exception is not expected and the run is not quiet, the exception is output by report_unexpected_exception(), as seen above as 'OUTPUT' and the lines that follows.
https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/doctest.py#L1264

This calls _exception_traceback(exc_info).
https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/doctest.py#L244
This calls traceback.print_exception, which I believe, for syntax errors, ultimately calls TracebackException._format_syntax_error.

I believe that the options for a fix are either
1. Call default sys.excepthook while capturing its output into a StringIO instance.
2. Assuming I am correct above about _format_syntax_error being called, monkeypatch it.  In line 779,
https://github.com/python/cpython/blob/5846c9b71ee9277fe866b1bdee4cc6702323fe7e/Lib/traceback.py#L779
replace '^' with a field with a calculated number of ^s.

I need to do one of these two for IDLE, and may try both.

Terry: I got it mostly working using your 2nd suggestion, I will do some testing and should be able to put up a PR in the next couple of days. Thanks for looking at this and explaining!

Terry: please take a look - https://github.com/python/cpython/pull/28567/files .

One important thing is that "traceback.print_exception" should correctly print syntax errors .

New changeset 20f439b6b9e1032930a31b88694ab9f37a09e6b4 by Pablo Galindo Salgado in branch 'main':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
https://github.com/python/cpython/commit/20f439b6b9e1032930a31b88694ab9f37a09e6b4

New changeset c7fdd6879b5c8447ee65b1bec95a1db43837a7a5 by Miss Islington (bot) in branch '3.10':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
https://github.com/python/cpython/commit/c7fdd6879b5c8447ee65b1bec95a1db43837a7a5

New changeset 1e2058214fffcb3919e0e127145106ade40a0420 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-45249: Ensure the traceback module prints correctly syntax errors with ranges (GH-28575)
https://github.com/python/cpython/commit/1e2058214fffcb3919e0e127145106ade40a0420

I've noticed a regression/change with the code change for this issue.

When not catching the exception from `compile("invalid(", "<stdin>", "single")` it has a caret below the opening parenthesis:

```
Traceback (most recent call last):
  File "…/t-syntaxerror-chained.py", line 2, in <module>
    compile("invalid(", "<stdin>", "single")
  File "<stdin>", line 1
    invalid(
           ^
SyntaxError: '(' was never closed
```

When using `traceback.print_exc` however this is missing:
```
Traceback (most recent call last):
  File "…/t-syntaxerror-chained.py", line 2, in <module>
    compile("invalid(", "<stdin>", "single")
  File "<stdin>", line 1
    invalid(

SyntaxError: '(' was never closed
```

The file used for testing:
```
try:
    compile("invalid(", "<stdin>", "single")
except Exception:
    # raise
    __import__("traceback").print_exc()
```

(this change was noticed between 3.10.0rc2 and the final release with pdbpp's test suite)

I've not investigated further (yet), and also feel free to ask for creating a new issue, but I've figured it would be good to notify you here first (where the code was changed).

New changeset fe0d9e22a52a10c4cbe52254b51f2d4e74d83568 by Pablo Galindo Salgado in branch 'main':
bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855)
https://github.com/python/cpython/commit/fe0d9e22a52a10c4cbe52254b51f2d4e74d83568

New changeset 5df35faf3611b459f7f32bfe9fd2ffa7fb2dc59e by Miss Islington (bot) in branch '3.10':
bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855)
https://github.com/python/cpython/commit/5df35faf3611b459f7f32bfe9fd2ffa7fb2dc59e