If we could include column offsets from the AST nodes for every bytecode instructions we could leverage these to offer much better tracebacks and a lot more information to debuggers and similar tools. For instance, in an expression such as:

z['aaaaaa']['bbbbbb']['cccccc']['dddddd].sddfsdf.sdfsdf

where one of these elements is None, we could tell exactly what of these pieces is the actual None and we could do some cool highlighting on tracebacks.

Similarly, coverage tools and debuggers could also make this distinction and therefore could offer reports with more granularity.

The cost is not 0: it would be two integers per bytecode instruction, but I think it may be worth the effort.

I'm going to prepare a PEP since the discussion regarding if the two integers per bytecode are worth enough is going to be eternal.

The additional cost will not only be the line number table, but we need to store the line for exceptions that are reraised after cleanup.
Adding a column will mean more stack consumption.

Yup, but I still think is worth the cost, giving that debugging improvements are usually extremely popular among users.

Specific examples of current messages and proposed improvements would help focus discussion.

If you are willing to only handle code lines up to 256 chars, only 2 bytes should be needed.  (0,0) or (255,255) could mean 'somewhere beyond the 256th char'.

> Specific examples of current messages and proposed improvements would help focus discussion.


Yeah, I am proposing going from:

>>> x['aaaaaa']['bbbbbb']['cccccc']['dddddd'].sddfsdf.sdfsdf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not subscriptable

to

>>> x['aaaaaa']['bbbbbb']['cccccc']['dddddd'].sddfsdf.sdfsdf
                         ^^^^^^^^^^
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not subscriptable

Basically, to highlight in all exceptions the range in the displayed line where the error ocurred. For instance:


>>> foo(a, b/z+2, c, 132432 /x, d /y)
                     ^^^^^^^^^
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

Marking what expression evaluated to None would be extremely helpful.  So would marking the 0 denominator when there is more than one candidate: "e = a/b + c/d".  It should be easy to revise IDLE Shell's print_exception to tag the span.  In some cases, the code that goes from a traceback line to a line in a file and marks it could do the same.

What would you do when the expression is not the last line?

try:
    x/y
    ...
except Exception as e:
   ...
   raise e

The except and raise might even be in a separate module.
I look forward to the PEP and discussion.

> So would marking the 0 denominator when there is more than one candidate: "e = a/b + c/d".

No, it will mark the offset of the bytecode that was getting executed when the exception was raised. Is just a way to mark what is raising the particular exception

> What would you do when the expression is not the last line?

There is some logic needed to re-raise exceptions. But basically it boils down to comparate the line numbers to decide if you want to propagate or not the offsets.

New changeset 98eee94421dcb42c15f2d7fc4cd21357722fbe2a by Pablo Galindo in branch 'main':
bpo-43950: Add code.co_positions (PEP 657) (GH-26955)
https://github.com/python/cpython/commit/98eee94421dcb42c15f2d7fc4cd21357722fbe2a

New changeset ec8759b060eff83ff466f42c5a96d83a685016ce by Batuhan Taskaya in branch 'main':
bpo-43950: optimize column table assembling with pre-sizing object (GH-26997)
https://github.com/python/cpython/commit/ec8759b060eff83ff466f42c5a96d83a685016ce

New changeset 44f91fc802a2b71312e9abd9e8e9dcbf02e8216d by Batuhan Taskaya in branch 'main':
bpo-43950: use 0-indexed column offsets for bytecode positions (GH-27011)
https://github.com/python/cpython/commit/44f91fc802a2b71312e9abd9e8e9dcbf02e8216d

New changeset 693cec0e2dcafa393ed5cdaa606f8dc8e3876adf by Batuhan Taskaya in branch 'main':
bpo-43950: include position in dis.Instruction (GH-27015)
https://github.com/python/cpython/commit/693cec0e2dcafa393ed5cdaa606f8dc8e3876adf

New changeset 5644c7b3ffd49bed58dc095be6e6148e0bb4431e by Ammar Askar in branch 'main':
bpo-43950: Print columns in tracebacks (PEP 657) (GH-26958)
https://github.com/python/cpython/commit/5644c7b3ffd49bed58dc095be6e6148e0bb4431e

New changeset 4823d9a51281ebbc8e8d82a0dd3edc7d13ea8ac7 by Ammar Askar in branch 'main':
bpo-43950: Add option to opt-out of PEP-657 (GH-27023)
https://github.com/python/cpython/commit/4823d9a51281ebbc8e8d82a0dd3edc7d13ea8ac7

New changeset 1890dd235f618d60c938f6904d2e1a8a56f99c1c by Batuhan Taskaya in branch 'main':
bpo-43950: Specialize tracebacks for subscripts/binary ops (GH-27037)
https://github.com/python/cpython/commit/1890dd235f618d60c938f6904d2e1a8a56f99c1c

New changeset 9c3eaf88dc5d5bed80cc45936de06b7b3162bc6d by Ammar Askar in branch 'main':
bpo-43950: Add documentation for PEP-657 (GH-27047)
https://github.com/python/cpython/commit/9c3eaf88dc5d5bed80cc45936de06b7b3162bc6d

New changeset 919ad537510fdc2c750109e0bc4eceea234324b2 by Batuhan Taskaya in branch 'main':
bpo-43950: make BinOp specializations more reliable (GH-27126)
https://github.com/python/cpython/commit/919ad537510fdc2c750109e0bc4eceea234324b2

As reported by Will McGugan on twitter:

Printing may be wrong for some unicode characters like:


Traceback (most recent call last):
  File "test.py", line 1, in <module>
    x = ("该" / 0) + (1 / 2)
         ~~~~~~^
TypeError: unsupported operand type(s) for /: 'str' and 'int'


and


Traceback (most recent call last):
  File "test.py", line 3, in <module>
    x = ("💩" / 0) + (1 / 2)
         ~~~~~~~
TypeError: unsupported operand type(s) for /: 'str' and 'int'

Ah, this is our friend _PyPegen_byte_offset_to_character_offset. I am glad we refactor it to be used by the Parser and the traceback mechanism. Ammar, would you like to take a look?

Aah, I won't have time to investigate until Monday. I'll take a look then unless someone gets to it first :)

We can probably leverage the unicodedata_UCD_east_asian_width_impl function in the unicodedata extension module to get the width of every character as emojis normally take 2 characters in the terminal. We could expose the necessary functions from there and use them in the _PyPegen_byte_offset_to_character_offset function

Had some time to look into this. Just to summarize this problem, it deals with unicode points that are single characters but take up more than the width of a single character, even with a monospace font [1].

In the examples from above, the Chinese character itself counts as one character in a Python string. However, notice that it needs two carets:

>>> x = "该"
>>> print(x)
该
>>> len(x)
1
>>> print(x + '\n' + '^^')
该
^^

This issue is somewhat font dependent, in the case of the emoji I know that windows sometimes renders emojis as single-character wide black-and-white glyphs or colorful ones depending on the program.

As Pablo alluded to, unicodedata.east_asian_width is probably the best solution we can implement. For these wide characters it provides:

>>> unicodedata.east_asian_width('💩')
'W'
>>> unicodedata.east_asian_width('该')
'W'

W corresponding to Wide. Whereas for regular width characters:

>>> unicodedata.east_asian_width('b')
'Na'
>>> unicodedata.east_asian_width('=')
'Na'

we get Neutral (Not East Asian). This can be used to count the "displayed width" of the characters and hence the carets. However, organization is going to be a bit tricky since we're currently using _PyPegen_byte_offset_to_character_offset to get offsets to use for string slicing in the ast segment parsing code. We might have to make a separate function that gets the font display-width.

-------------

[1] Way more details on this issue here: https://denisbider.blogspot.com/2015/09/when-monospace-fonts-arent-unicode.html and an example of a Python library that tries to deal with this issue here: https://github.com/jquast/wcwidth

The effort to match caret lines to general unicode is similar to a previous issue that was closed as futile.  (But I could not find it.)  It has a downside that should be considered.

The fundamental problem is that there is no fixed pitch font for unicode. (Let alone any font covering all of unicode.) Nor is there a single-double width definition, or font, for all of unicode.  Some character sets are not amenable to such treatment.

To see the problem easier, open, for instance, IDLE's option/settings dialog, showing the fonts tab and a multi-script sample.  On Windows, select what I believe is the most 'fixed' font -- Courier New.  ASCII, Latin1, IPA, Greek, Cyrillic, Hebrew, and Arabic are all rendered in the same fixed pitch.  But the last 4 Cyrillic characters of "...ЪъЭэѠѤѬӜ" are extremely cramped and may be rendered differently from the rest.  The East Asian characters are in a different fixed pitch, about 1.6 times the Ascii, etc. fixed pitch.  (So the double-wide 2 is 1.6 rounded up.  And with some fonts, the East Asian scripts are not all the same pitch.)  The South Asian script are variable pitch and for the sample chars, average wider than 1 (they have 20 chars, like the Ascii, etc, lines).  Tamil, especially, has a wide range of widths, with the widest as wide as the East Asian chars.

On Windows, on my machine, pasting the sample text between quotes results in the Greek chars, the last 4 Cyrillic chars, and all Asian chars (including Hebrew and Arabic) being replaced by replacement chars.  (I thought that this was better once, but maybe I mis-remember.)  While one can get script-specific fonts, the fixed-pitch South Asian fonts I tried on Mac were hardly readable.  My conclusion is that people using certain scripts and anyone wanting a wide variety of scripts needs to use a GUI-based editor and shell rather than a fixed-pitch terminal/console.

As long as the caret line has 1 char per code char, a GUI program can use it to mark code characters, and do so differently for '~' and '^'.  If some of these chars are doubled, exact character information is lost.  If you go ahead with this, please use a third character, such as '-', for additions.  GUI programs could then ignore these, given that they can otherwise can get the start character information.

I think this is the previous issue you are talking about Terry. https://bugs.python.org/issue24665

The correct algorithm is a little more complex than just using east_asian_widths. Ideally we would like to use the wcwidth function (https://man7.org/linux/man-pages/man3/wcwidth.3.html) which implements all the logic to somewhat properly figure out widths.

Given Terry's comments from above and the previous issues I'm starting to think the fix for this issue might not be worth the added complexity.

New changeset fbc349ff790c21f1a59af939d42033470790c530 by Batuhan Taskaya in branch 'main':
bpo-43950: Distinguish errors happening on character offset decoding (GH-27217)
https://github.com/python/cpython/commit/fbc349ff790c21f1a59af939d42033470790c530

I don't know, seems that rust can deal with this just fine:

❯ cargo run
   Compiling rtest v0.1.0 (/home/pablogsal/rtest)
error[E0308]: mismatched types
 --> src/main.rs:7:10
  |
7 |     test("hellooooooo: 🥳 😏 😒 😞 😔 some emojis");
  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |          |
  |          expected struct `String`, found `&str`
  |          help: try using a conversion method: `"hellooooooo: 🥳 😏 😒 😞 😔 some emojis".to_string()`

In bpo this looks bad, but test this small program yourself and check the error in your terminal:


fn test(number: String) -> () {
    println!("Hello, world: {}", number);
}

fn main() {
    test("hellooooooo: 🥳 😏 😒 😞 😔 some emojis");
}

They seem to be checking the unicode width as we are proposing here. Check the calls to unicode_width::UnicodeWidthChar::width:

https://github.com/rust-lang/rust/blob/master/compiler/rustc_errors/src/emitter.rs#L729-L733

Indeed, and the unicode_width package seems to implement the wcwidth algorithm:

* https://github.com/unicode-rs/unicode-width/blob/master/src/tables.rs#L39-L48
* https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c

https://bugs.python.org/issue12568 is an issue from the past that proposed adding wcwidth to the standard library.

Even if this doesn't cover all cases it seems that everyone else is going with this solution for what I can see, and currently this looks suboptimal on python because we get this wrong on the vast majority of cases so my suggestion is to try to fix this the same way rust on others are doing it even if there is some subset of cases where it won't fully work

bpo-44719 managed to make the `assert(source_line);` fail

New changeset c8362314cce53a5b59da7523fbdfa00f122aa319 by Batuhan Taskaya in branch 'main':
bpo-43950: ensure source_line is present when specialising the traceback (GH-27313)
https://github.com/python/cpython/commit/c8362314cce53a5b59da7523fbdfa00f122aa319

New changeset ef8b8535cb3cd705392af9b927d6ff336d98427d by Batuhan Taskaya in branch 'main':
bpo-43950: check against the raw string, not the pyobject (GH-27337)
https://github.com/python/cpython/commit/ef8b8535cb3cd705392af9b927d6ff336d98427d

New changeset 4f5980a4f57dab68b9137304f58bd08891d43d5a by Batuhan Taskaya in branch 'main':
bpo-43950: support long lines in traceback.py (GH-27336)
https://github.com/python/cpython/commit/4f5980a4f57dab68b9137304f58bd08891d43d5a

There is a subtle difference between traceback.py and Python/traceback.c which makes the latter (C version) output the line without trimming the trailing whitespace. The Python version simply uses `.strip()` which strips both the left (starting) and right (ending) side. I'd personally go with adapting the C version to use full strip, since it makes more sense but I am open to comments on this. 

Here is an example test (needs GH 27339 first) (func( call has 2 trailing whitespace characters);
https://gist.github.com/isidentical/37ab1693d551b678a52626298d99e582

New changeset 3e235e0447e373d81f195f4292959c7be9c013dc by Batuhan Taskaya in branch 'main':
bpo-43950: support some multi-line expressions for PEP 657 (GH-27339)
https://github.com/python/cpython/commit/3e235e0447e373d81f195f4292959c7be9c013dc

New changeset 85ea2d6165dec0cffa6302eb6dc40406eae1edf5 by Batuhan Taskaya in branch 'main':
bpo-43950: support positions for dis.Instructions created through dis.Bytecode (GH-28142)
https://github.com/python/cpython/commit/85ea2d6165dec0cffa6302eb6dc40406eae1edf5

I've opened a new issue about unused `dis.Positions` here: https://bugs.python.org/issue46422