This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: the f_lineno getter is broken
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, blueyed, serhiy.storchaka, xdegaye
Priority: normal Keywords: patch

Created on 2015-07-04 20:36 by xdegaye, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
f_lineno.patch xdegaye, 2015-07-04 20:36
f_lineno_tests.patch xdegaye, 2015-07-04 20:37 review
Pull Requests
URL Status Linked Edit
PR 6233 closed serhiy.storchaka, 2018-03-25 12:31
PR 12419 closed xdegaye, 2019-03-18 19:06
Messages (8)
msg246275 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-07-04 20:36
The last paragraph of Objects/lnotab_notes.txt explains that the f_lineno member of the PyFrameObject structure is needed to store the line number of the last "line" tracing event so that this value may be used as the line number of the "return" event instead of the (sometimes confusing) value computed from f_lasti.  The f_lineno getter must then return the value of f->f_lineno (instead of the value computed from f->f_lasti) when tracing is set. The current implementation translates "tracing is set" as "the local f_trace trace function is not NULL", this is wrong for the following reasons:

* AFAIK it is not documented anywhere that Python users implementing a trace function must delete the local f_trace functions of all the frames when setting the
  global trace function to None via sys.settrace(None) (issue 7238).

* Bdb.set_continue() in the bdb module of the std lib seems to know about this and attempts to do it, but fails to delete f_trace from the topmost frame, named botframe (sic) (issue 16482, issue 17697).

* It is not obvious how to delete the f_trace of all suspended generators when setting the global trace function to None (issue 17277).

* _PyTraceback_Add() in Python/traceback.c sets frame->f_lineno, obviously forgetting that it is useless since its f_trace is NULL.

This patch changes the semantics of f_lineno by stating that f_lineno is invalid when its value is -1. When tracing, the f_lineno of all the frames on the stack is valid. The f_lineno of a suspended generator is always invalid.
msg246276 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-07-04 20:37
Uploading the corresponding test cases.
msg314237 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-22 08:59
Could you please convert your patches to a PR Xavier?
msg314243 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2018-03-22 10:05
I will work on it shortly.
msg314409 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2018-03-25 12:08
A trace function may also be set in extension modules by PyEval_SetTrace() and it may not use f->f_trace. This is another reason why f->f_trace cannot be used in PyFrame_GetLineNumber() to know when f->f_lineno is valid.
msg314410 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2018-03-25 12:27
Added PR 6233.
One of the GitHub bots failed to link this issue with PR 6233. Maybe this is related to the fact that connections to bpo are currently failing intermitently with the message:

An error occurred during a connection to bugs.python.org. Peer’s certificate has an invalid signature. Error code: SEC_ERROR_BAD_SIGNATURE
msg339667 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-04-08 19:39
Fixed a bug in the implementation of PR 12419 while running the coverage.py test suite: f_lineno must be valid upon 'call' trace events. The confusion stems from the reason why until now we have prevented line jumps from 'call' trace events, see below.

Summary:
  * Full coverage.py tests OK (including the C tracer).
  * Fixes issues #7238, #16482, #17277 and #17697.
  * We could now remove the restriction that prevents jumps from a call trace event. The restriction was based on the fact that f->f_trace is NULL until after the first return from call_trampoline() and setting f_lineno from within this first call to call_trampoline() would have had the f_lineno getter still return the value returned by PyCode_Addr2Line(), very confusing for the user !
  * A trace function may be set in an extension modules by PyEval_SetTrace() and the extension module may not use f->f_trace and still make jumps using f_lineno.
  * Fixes _PyTraceback_Add() at https://github.com/python/cpython/blob/58721a903074d28151d008d8990c98fc31d1e798/Python/traceback.c#L272
msg357845 - (view) Author: daniel hahler (blueyed) * Date: 2019-12-05 11:21
This is likely covered by existing/linked issues already, but wanted to leave it here nonetheless:

Given t-pdb.py:
```
import sys


def main():
    sys.stdout.write("main...\n")
    assert 0


if __name__ == "__main__":
    main()
```

Without the fix from the PR:
```
% python3.8 -m pdb -c cont t-pdb.py
main...
Traceback (most recent call last):
  File "…/pyenv/3.8.0/lib/python3.8/pdb.py", line 1702, in main
    pdb._runscript(mainpyfile)
  File "…/pyenv/3.8.0/lib/python3.8/pdb.py", line 1571, in _runscript
    self.run(statement)
  File "…/pyenv/3.8.0/lib/python3.8/bdb.py", line 587, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "…/Vcs/cpython/t-pdb.py", line 1, in <module>
    import sys
  File "…/Vcs/cpython/t-pdb.py", line 6, in main
    assert 0
AssertionError
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> …/Vcs/cpython/t-pdb.py(6)main()
-> assert 0
(Pdb)
```

With the fix:
```
% /tmp/cpython-bisect/bin/python3.8 -m pdb -c cont t-pdb.py
main...
Traceback (most recent call last):
  File "/tmp/cpython-bisect/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/tmp/cpython-bisect/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/tmp/cpython-bisect/lib/python3.8/bdb.py", line 583, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "…/Vcs/cpython/t-pdb.py", line 10, in <module>
    main()
  File "…/Vcs/cpython/t-pdb.py", line 6, in main
    assert 0
AssertionError
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> …/Vcs/cpython/t-pdb.py(6)main()
-> assert 0
(Pdb)
```

As you can see the traceback in the fixed case contains `main()` correctly,
while it has `import sys` (the scripts first line) otherwise.

I can only repeat myself to ask for reviewing/merging https://github.com/python/cpython/pull/12419.
History
Date User Action Args
2022-04-11 14:58:18adminsetgithub: 68753
2019-12-05 11:21:41blueyedsetnosy: + blueyed
messages: + msg357845
2019-04-08 19:39:39xdegayesetmessages: + msg339667
2019-03-22 10:01:39blueyedsetversions: + Python 3.7, Python 3.8, Python 3.9
2019-03-18 19:06:53xdegayesetpull_requests: + pull_request12373
2018-03-25 12:31:37serhiy.storchakasetstage: patch review
pull_requests: + pull_request5972
2018-03-25 12:27:30xdegayesetmessages: + msg314410
2018-03-25 12:08:23xdegayesetmessages: + msg314409
2018-03-22 10:05:59xdegayesetmessages: + msg314243
2018-03-22 08:59:47serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg314237
2015-07-04 20:42:06xdegayesettype: behavior
2015-07-04 20:37:06xdegayesetfiles: + f_lineno_tests.patch

messages: + msg246276
2015-07-04 20:36:38xdegayecreate