Title: Implement PEP 626 -- Precise line numbers for debugging
Type: enhancement Stage: resolved
Components: Versions: Python 3.10
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Mark.Shannon Nosy List: BTaskaya, Mark.Shannon, alex.henrie, methane, nascheme, nedbat, pablogsal, zhtw1234
Priority: normal Keywords: patch

Created on 2020-11-02 15:32 by Mark.Shannon, last changed 2021-03-17 11:46 by Mark.Shannon. This issue is now closed.

File name Uploaded Description Edit
IMAG0629_1(1).jpg zhtw1234, 2020-12-11 20:04
IMAG0629_1.jpg zhtw1234, 2020-12-11 20:06
IMAG0629_1.jpg zhtw1234, 2020-12-11 20:08
Pull Requests
URL Status Linked Edit
PR 23113 merged Mark.Shannon, 2020-11-02 15:43
PR 23245 merged Mark.Shannon, 2020-11-12 10:10
PR 23251 merged Mark.Shannon, 2020-11-12 15:57
PR 23256 merged Mark.Shannon, 2020-11-13 11:53
PR 23495 merged Mark.Shannon, 2020-11-24 13:57
PR 23636 merged Mark.Shannon, 2020-12-04 10:33
PR 23743 merged Mark.Shannon, 2020-12-11 14:41
PR 23803 merged Mark.Shannon, 2020-12-16 14:50
PR 23896 merged Mark.Shannon, 2020-12-22 20:32
PR 24094 merged Mark.Shannon, 2021-01-04 13:01
PR 24697 merged alex.henrie, 2021-03-02 01:39
PR 24708 closed nascheme, 2021-03-02 18:59
PR 24714 closed nascheme, 2021-03-03 02:29
PR 24759 merged nascheme, 2021-03-05 02:34
PR 24860 merged Mark.Shannon, 2021-03-14 18:06
PR 24892 merged Mark.Shannon, 2021-03-16 11:43
PR 24902 merged Mark.Shannon, 2021-03-17 11:46
Messages (28)
msg380231 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-11-02 15:32
Implementation of PEP 626 requires:

1. Implementation of the new line number table and associated APIs.
2. Removal of BEGIN_DO_NOT_EMIT_BYTECODE and END_DO_NOT_EMIT_BYTECODE from the compiler as they do not understand line numbers and may remove lines from the bytecode that they shouldn't.
3. Enhance compiler front-end and CFG optimizer to avoid the negative performance impact of PEP.
 a) Duplicate the tests in while blocks to avoid the extra jump instruction at the end of the loop.
 b) Duplicate and renumber terminator blocks that have no line numbers.

Guaranteeing that f_lineno is correct without hurting performance

PEP 626 mandates that the f_lineno attribute of a frame is always correct, even after a return or raise, but we don't want to hurt performance.
Since the interpreter ensures that the f_lasti attribute of a frame is always correct, we can ensure correctness of f_lineno at zero cost, by ensuring that all RETURN_VALUE, RAISE_VARARGS and RERAISE instruction have a non-negative line number. Then f_lineno can always be lazily computed from f_lasti.

The front-end generates artificial RERAISEs and RETURN_VALUE that have no line number. To give these instructions a valid line number we can take advantage of the fact that such terminator blocks (blocks with no successors) can be freely duplicated. Once duplicated, each terminator block will have only one predecessor and can acquire the line number of the preceding block, without causing false line events.
msg380245 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-02 19:08
I'm happy that we are removing BEGIN_DO_NOT_EMIT_BYTECODE and END_DO_NOT_EMIT_BYTECODE but could you elaborate how this is related? These macros protect the compiler from emitting bytecode that corresponds to deaf code and by definition, unreachable. Could you give an example of a situation in which they create something that messes up the line numbers? Is this something to be with cleanup blocks in dead code or something similar?
msg380268 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-11-03 10:42
The following code is completely eliminated by the macros.

1. if 0:
2.     secret_debugging_code()

PEP 626 says that all executed lines of code must generate trace events,
so we need to emit an instruction for line 1.

Dead code elimination will remove the `secret_debugging_code()`, but leave the test. The peephole optimiser can then reduce it to a NOP, but won't eliminate it as it is the only instruction for line 1.
msg380276 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-11-03 13:49
> Dead code elimination will remove the `secret_debugging_code()`, but leave the test. The peephole optimiser can then reduce it to a NOP, but won't eliminate it as it is the only instruction for line 1.

Gotcha. I am pretty sure that this will have a similar problem as the coverage people were claiming when we were not properly removing all dead code (slightly less coverage percentage). This is not a problem of course, but we should ping the coverage folks so they are aware of this.
msg380809 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-11-12 09:43
New changeset 877df851c3ecdb55306840e247596e7b7805a60a by Mark Shannon in branch 'master':
bpo-42246: Partial implementation of PEP 626. (GH-23113)
msg380846 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-11-12 19:49
New changeset cc75ab791dd5ae2cb9f6e0c3c5f734a6ae1eb2a9 by Mark Shannon in branch 'master':
bpo-42246: Eliminate jumps to exit blocks by copying those blocks. (#23251)
msg380865 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-13 00:52
> New changeset 877df851c3ecdb55306840e247596e7b7805a60a by Mark Shannon in branch 'master':
> bpo-42246: Partial implementation of PEP 626. (GH-23113)

This change introduced reference leaks:

5 tests failed:
    test_asyncgen test_builtin test_coroutines test_exceptions

For example:

$ ./python -m test test_syntax -R 3:3 -m test.test_syntax.SyntaxTestCase.test_no_indent
0:00:00 load avg: 1.59 Run tests sequentially
0:00:00 load avg: 1.59 [1/1] test_syntax
beginning 6 repetitions
test_syntax leaked [27, 27, 27] references, sum=81
test_syntax leaked [20, 20, 20] memory blocks, sum=60
test_syntax failed

== Tests result: FAILURE ==

1 test failed:

Total duration: 955 ms
Tests result: FAILURE
msg380880 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-11-13 12:54
New changeset fd009e606a48e803e7187983bf9a5682e938fddb by Mark Shannon in branch 'master':
bpo-42246: Fix memory leak in compiler (GH-23256)
msg382312 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-02 13:31
New changeset 5977a7989d49c3e095c7659a58267d87a17b12b1 by Mark Shannon in branch 'master':
bpo-42246: Make sure that line number is correct after a return, as required by PEP 626 (GH-23495)
msg382492 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-04 15:22
New changeset eaccc12aa986f92ea05f3f0a63cedbff78dd67f1 by Mark Shannon in branch 'master':
bpo-42246: Don't forget the entry block when ensuring that all exits have a line number (GH-23636)
msg382871 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2020-12-11 15:26
Mark, BTW: I have run the test suite on 3.10.0a3, and as expected there are failures.  I haven't dug into it yet to see what looks expected and what does not.  I also see there are still changes happening on master, so I'm not sure when to commit the time.
msg382872 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-11 15:50
What are the failures?

I'd like to take a look, to see if things are as expected, and if there are any tests we can add to CPython.
msg382884 - (view) Author: 雅雯 (zhtw1234) Date: 2020-12-11 20:03
msg382886 - (view) Author: 雅雯 (zhtw1234) Date: 2020-12-11 20:04
msg382888 - (view) Author: 雅雯 (zhtw1234) Date: 2020-12-11 20:06
msg382889 - (view) Author: 雅雯 (zhtw1234) Date: 2020-12-11 20:08
msg382890 - (view) Author: 雅雯 (zhtw1234) Date: 2020-12-11 20:09
msg382950 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2020-12-14 00:27
Mark, I'm categorizing and characterizing the test failures.  Here's the start of it:  Let me know what other information would be useful.
msg382985 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-14 13:02
Thanks Ned, that's really helpful. I'll go through those points:

Code after break/continue is no longer compiled.

First line number of modules

Except clause when no exception

Double loops (this also covers End-of-loop jumps, I think)

I want to merge before I fix any of the others, but here is a summary of what I think are the root causes.

    Exit block duplication does not preserve line number of jump to final block

Finally handling
    Combination of two things. Not preserving line numbers when performing jump-to-jump elimination and not marking try cleanup code as artificial.
msg383042 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-15 11:07
New changeset 8473cf89bdbf2cb292b39c972db540504669b9cd by Mark Shannon in branch 'master':
bpo-42246: Remove DO_NOT_EMIT_BYTECODE macros, so that while loops and if statements conform to PEP 626. (GH-23743)
msg383159 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-16 12:29 fixes the finally handling.
The if-break case was fixed by earlier changes.
msg383245 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-17 13:55
New changeset bf353f3c2d937772a8cf30b15fd8eb7b82665ccb by Mark Shannon in branch 'master':
bpo-42246: Make sure that `f_lasti`, and thus `f_lineno`, is set correctly after raising or reraising an exception (GH-23803)
msg383643 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-12-23 11:43
New changeset 28b75c80dcc1e17ed3ac1c69362bf8dc164b760a by Mark Shannon in branch 'master':
bpo-42246: Don't eliminate jumps to jumps, if it will break PEP 626. (GH-23896)
msg387910 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-03-02 11:08
All done :)
msg388217 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2021-03-06 21:34
New changeset 87ec26b812e9c4095c017dc60f246eda37b83ab2 by Neil Schemenauer in branch 'master':
bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759)
msg388836 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-03-16 11:16
Is there a reason PEP 626 isn't yet mentioned in ?
msg388837 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-03-16 11:31
No. We should add it.
msg388900 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-17 00:03
PEP 626 deprecates co_lnotab, co_lnotab:

This doesn't seem to be mentioned in the What's new document and is quite important. Mark, do you mind creating a PR for this? I could do it and add you as a reviewer if you wish
Date User Action Args
2021-03-17 11:46:38Mark.Shannonsetpull_requests: + pull_request23665
2021-03-17 00:03:57pablogsalsetmessages: + msg388900
2021-03-16 11:43:46Mark.Shannonsetpull_requests: + pull_request23654
2021-03-16 11:31:32Mark.Shannonsetmessages: + msg388837
2021-03-16 11:16:51nedbatsetmessages: + msg388836
2021-03-14 18:06:35Mark.Shannonsetpull_requests: + pull_request23620
2021-03-06 21:34:06naschemesetmessages: + msg388217
2021-03-05 02:34:56naschemesetpull_requests: + pull_request23532
2021-03-03 02:29:54naschemesetpull_requests: + pull_request23492
2021-03-02 18:59:32naschemesetnosy: + nascheme

pull_requests: + pull_request23485
2021-03-02 11:08:24Mark.Shannonsetstatus: open -> closed
resolution: fixed
messages: + msg387910

stage: patch review -> resolved
2021-03-02 01:39:37alex.henriesetnosy: + alex.henrie
pull_requests: + pull_request23475
2021-01-04 13:01:07Mark.Shannonsetpull_requests: + pull_request22923
2020-12-23 11:43:19Mark.Shannonsetmessages: + msg383643
2020-12-22 22:37:30BTaskayasetnosy: + BTaskaya
2020-12-22 20:32:21Mark.Shannonsetpull_requests: + pull_request22751
2020-12-22 00:54:50methanesetnosy: + methane
2020-12-17 13:55:38Mark.Shannonsetmessages: + msg383245
2020-12-16 14:50:24Mark.Shannonsetpull_requests: + pull_request22663
2020-12-16 12:29:22Mark.Shannonsetmessages: + msg383159
2020-12-15 11:07:59Mark.Shannonsetmessages: + msg383042
2020-12-14 13:02:14Mark.Shannonsetmessages: + msg382985
2020-12-14 00:27:45nedbatsetmessages: + msg382950
2020-12-11 21:14:14rhettingersettitle: Implement PEP 626 -> Implement PEP 626 -- Precise line numbers for debugging
2020-12-11 20:09:40zhtw1234setmessages: + msg382890
2020-12-11 20:08:02zhtw1234setfiles: + IMAG0629_1.jpg

messages: + msg382889
2020-12-11 20:06:23zhtw1234setfiles: + IMAG0629_1.jpg

messages: + msg382888
2020-12-11 20:04:38zhtw1234setfiles: - IMAG0629_1.jpg
2020-12-11 20:04:23zhtw1234setfiles: + IMAG0629_1(1).jpg

messages: + msg382886
2020-12-11 20:03:10zhtw1234setfiles: + IMAG0629_1.jpg
nosy: + zhtw1234
messages: + msg382884

2020-12-11 15:50:18Mark.Shannonsetmessages: + msg382872
2020-12-11 15:26:07nedbatsetmessages: + msg382871
2020-12-11 14:41:40Mark.Shannonsetpull_requests: + pull_request22601
2020-12-04 15:22:22Mark.Shannonsetmessages: + msg382492
2020-12-04 10:33:02Mark.Shannonsetpull_requests: + pull_request22504
2020-12-02 13:31:43Mark.Shannonsetmessages: + msg382312
2020-11-24 14:02:29vstinnersetnosy: - vstinner
2020-11-24 13:57:57Mark.Shannonsetpull_requests: + pull_request22384
2020-11-13 12:54:22Mark.Shannonsetmessages: + msg380880
2020-11-13 11:53:50Mark.Shannonsetpull_requests: + pull_request22153
2020-11-13 00:52:47vstinnersetnosy: + vstinner
messages: + msg380865
2020-11-12 19:49:41Mark.Shannonsetmessages: + msg380846
2020-11-12 15:57:50Mark.Shannonsetpull_requests: + pull_request22148
2020-11-12 10:10:09Mark.Shannonsetstage: patch review
pull_requests: + pull_request22142
2020-11-12 09:43:56Mark.Shannonsetmessages: + msg380809
2020-11-03 15:41:19Mark.Shannonsetnosy: + nedbat
2020-11-03 13:49:18pablogsalsetmessages: + msg380276
2020-11-03 10:42:59Mark.Shannonsetmessages: + msg380268
stage: patch review -> (no value)
2020-11-02 19:08:13pablogsalsetmessages: + msg380245
2020-11-02 15:43:17Mark.Shannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request22030
2020-11-02 15:32:42Mark.Shannoncreate