classification
Title: incorrect line numbers in backtrace after removing a trace function
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, ishimoto, jcea, nikratio, pconnell, xdegaye
Priority: normal Keywords: patch

Created on 2013-02-22 16:42 by xdegaye, last changed 2017-12-20 08:56 by ishimoto.

Files
File name Uploaded Description Edit
tracer.py xdegaye, 2013-02-22 16:42
generator.py xdegaye, 2013-02-22 16:43
backtrace_lno.patch xdegaye, 2013-02-23 14:34 review
traced_frame.patch xdegaye, 2013-02-24 16:57 review
lineno_getter.patch xdegaye, 2014-07-09 14:45 review
Messages (8)
msg182672 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2013-02-22 16:42
It seems that using f_trace in the f_lineno getter PyFrame_GetLineNumber(), as
the condition to decide when tracing is active, is incorrect.
See the following two examples.

In the backtrace printed by tracer.py running with python 3.3,
the last entry should be line 12 instead of line 10:

$ python3 /tmp/tracer.py
Traceback (most recent call last):
  File "/tmp/tracer.py", line 15, in <module>
    foo()
  File "/tmp/tracer.py", line 10, in foo
    bar()
ZeroDivisionError: division by zero


This simple case does not occur with pdb, because pdb takes care of deleting
the f_trace attribute of all the frames in the call stack when removing the
trace function (see set_continue() in bdb.py). But this is not good enough when
a generator is involved as can be seen when running generator.py. In the
backtrace the last entry should be line 6 instead of line 8:

$ python3 /tmp/generator.py
> /tmp/generator.py(16)<module>()
-> foo()
(Pdb) step
--Call--
> /tmp/generator.py(10)foo()
-> def foo():
(Pdb) step
> /tmp/generator.py(11)foo()
-> it = gen()
(Pdb) step
> /tmp/generator.py(12)foo()
-> next(it)
(Pdb) step
--Call--
> /tmp/generator.py(3)gen()
-> def gen():
(Pdb) return
--Return--
> /tmp/generator.py(8)gen()->0
-> yield i
(Pdb) step
> /tmp/generator.py(13)foo()
-> next(it)
(Pdb) continue
Traceback (most recent call last):
  File "/tmp/generator.py", line 16, in <module>
    foo()
  File "/tmp/generator.py", line 13, in foo
    next(it)
  File "/tmp/generator.py", line 8, in gen
    yield i
ZeroDivisionError: division by zero


It seems that it could be possible to fix this issue by replacing the test for
f->f_trace in PyFrame_GetLineNumber, by a test for f->f_tstate->use_tracing,
and updating accordingly the f_lineno and f_trace setters.
msg182710 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2013-02-23 03:36
Xavier, could you possibly provide a patch and a test?
msg182746 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2013-02-23 14:34
The patch (on the default branch) reverts one of the changes made in r72488 to
introduce the new PyFrame_GetLineNumber() function (issue 5954): tb_lineno is
now back again the result of the call to PyCode_Addr2Line() instead of the
call to PyFrame_GetLineNumber().

The other changes made by r72488 in _warnings.c and ceval.c should also
probably be reverted as well.

The patch updates bdb set_continue() for consistency.

The patch adds a test to test_sys_settrace.
msg182867 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2013-02-24 11:34
The proposed patch fixes the backtrace line numbers issue, but it does not fix
PyFrame_GetLineNumber() which is the recommended way to get the frame line
number.

As mentionned in the original message, testing for f->f_trace to implement the
f_lineno getter is not correct. The f_lineno setter is also wrong in allowing to
modify f_lineno when the frame is not the one that is being traced (pdb prevents
that to happen though, in do_jump()).

I am working on another patch that should fix the issue by changing
PyFrame_GetLineNumber() and the f_lineno accessors.
msg182879 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2013-02-24 16:57
> fix the issue by changing PyFrame_GetLineNumber() and the f_lineno accessors

The new patch named traced_frame.patch has been uploaded.

Also, now it is not allowed anymore to set the f_lineno attribute of a frame
that is not the frame being traced, as f_lasti is invalidated anyway on
returning to the evaluation of that frame.
msg182955 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2013-02-25 16:05
The traced_frame.patch fixes also issue 7238 and issue 16482.
msg222620 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-07-09 14:45
The previous patch changed a field in the PyThreadState structure. This new patch is simpler and does not prevent to change f_lineno when it is not the attribute of the frame being traced. The new patch fixes also issue 7238, issue 16482 and issue 17697.
msg246274 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-07-04 20:31
The patch is wrong, the frame may not be run by the current PyThreadState.
History
Date User Action Args
2017-12-20 08:56:12ishimotosetnosy: + ishimoto
2015-07-04 20:31:42xdegayesetmessages: + msg246274
2014-07-09 14:45:55xdegayesetfiles: + lineno_getter.patch

messages: + msg222620
2014-06-03 03:57:46nikratiosetnosy: + nikratio
2014-04-24 05:53:41pconnellsetnosy: + pconnell
2013-02-25 16:05:59xdegayesetmessages: + msg182955
2013-02-24 16:57:43xdegayesetfiles: + traced_frame.patch

messages: + msg182879
2013-02-24 11:34:51xdegayesetmessages: + msg182867
2013-02-23 14:34:38xdegayesetfiles: + backtrace_lno.patch
keywords: + patch
messages: + msg182746
2013-02-23 03:36:02jceasetnosy: + jcea
messages: + msg182710
2013-02-22 20:31:19terry.reedysetnosy: + belopolsky
2013-02-22 16:43:21xdegayesetfiles: + generator.py
2013-02-22 16:42:49xdegayecreate