Guido:
> I don't understand why the traceback module is implicated.

The traceback module is implicated, because it currently does not allow the caret to point to a position after the error line.

In format_exception_only there is the following snippet of code:

if offset is not None:
    caretspace = badline.rstrip('\n')
    offset = min(len(caretspace), offset) - 1
    caretspace = caretspace[:offset].lstrip()
    # non-space whitespace (likes tabs) must be kept for alignment
    caretspace = ((c.isspace() and c or ' ') for c in caretspace)
    yield '    {}^\n'.format(''.join(caretspace))

There are two things here, that put together contradict the way that pegen reports errors. `badline` gets rstripped and then the offset is changed to point to the end of the string, in case it is currently pointing after its end. That basically does not allow SyntaxErrors to be formatted the way they currently are, when using the new parser.

For example:
╰─ ./python.exe
Python 3.9.0a6+ (heads/master:4638c64295, May  7 2020, 16:47:53)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> x = 1 | 2 |
  File "<stdin>", line 1
    x = 1 | 2 |
               ^
SyntaxError: invalid syntax

But:
╰─ ./python.exe
Python 3.9.0a6+ (heads/master:4638c64295, May  7 2020, 16:47:53)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import traceback
>>> try:
...     exec('x = 1 | 2 |')
... except SyntaxError as se:
...     exc = se
...
>>> traceback.print_exception(type(exc), exc, exc.__traceback__)
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<string>", line 1
    x = 1 | 2 |
              ^
SyntaxError: invalid syntax

I might be missing something here, but I think that that's the traceback module's "fault".