classification
Title: pdb skips frames after hitting a breakpoint and running step
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: georg.brandl, loewis, meador.inge, neologix, orsenthil, pitrou, python-dev, tshepang, xdegaye
Priority: normal Keywords: needs review

Created on 2011-10-14 18:56 by xdegaye, last changed 2012-05-09 15:44 by orsenthil. This issue is now closed.

Files
File name Uploaded Description Edit
restore_trace.patch xdegaye, 2011-10-14 18:56 review
restore_trace_2.patch xdegaye, 2011-10-16 19:23 py3 patch and improved test case review
restore_trace.py27.patch xdegaye, 2011-10-16 19:24 py2 patch and test case
Messages (24)
msg145557 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2011-10-14 18:56
Pdb skips frames after hitting a breakpoint and running step commands
that walk back the frame stack.

Run the following two tests with the two files named foo.py and bar.py:

===   foo.py   ============================================
from bar import bar

def foo():
    bar()

def nope():
    pass

def foobar():
    foo()
    nope()

foobar()

===   bar.py   ============================================
def bar():
    print('1')

===   test_1   ============================================
$ python3 --version
Python 3.2

$ python3 -m pdb foo.py
> /path/to/foo.py(1)<module>()
-> from bar import bar
(Pdb) from bar import bar
(Pdb) break bar
Breakpoint 1 at /path/to/bar.py:1
(Pdb) continue
> /path/to/bar.py(2)bar()
-> print('1')
(Pdb) step
1
--Return--
> /path/to/bar.py(2)bar()->None
-> print('1')
(Pdb) step
--Call--
> /path/to/foo.py(6)nope()
-> def nope():
(Pdb)

===   test_2   ============================================
$ python3 -m pdb foo.py
> /path/to/foo.py(1)<module>()
-> from bar import bar
(Pdb) break nope
Breakpoint 1 at /path/to/foo.py:6
(Pdb) from bar import bar
(Pdb) break bar
Breakpoint 2 at /path/to/bar.py:1
(Pdb) continue
> /path/to/bar.py(2)bar()
-> print('1')
(Pdb) step
1
--Return--
> /path/to/bar.py(2)bar()->None
-> print('1')
(Pdb) step
--Return--
> /path/to/foo.py(4)foo()->None
-> bar()
(Pdb)

===========================================================

Note: stop_here, break_anywhere and dispatch_call are methods of the
Bdb class.

test_1 fails to stop in foo() after the second 'step' command because
the trace function is not set for all the frames being created in the
foo module, since stop_here() and break_anywhere() are both False
whenever dispatch_call() is invoked in this module. So after the
second 'step' command, trace_dispatch is not invoked by the
interpreter until a new frame is created, which happens when nope() is
called.

test_2 succeeds and stops in foo() after the second 'step' command.
After setting the dummy breakpoint 1 in the foo module in test_2,
break_anywhere() becomes True in the foo module and the trace function
is set for all the frames created in this module (with an associated
performance penalty).

The problem exists in all python versions.

The attached patch fixes this problem by restoring the trace function
on returning from a frame when the command is 'step'.

The patch includes a test case.
msg145633 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2011-10-16 19:23
Uploaded restore_trace_2.patch that improves the test case.
msg145634 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2011-10-16 19:24
Uploaded restore_trace.py27.patch with a fix and test case for python 2.7.
msg154889 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2012-03-04 12:13
This message is just a reminder that this 4 months old issue raises
the point that the step command in pdb is broken. A patch and test
case have been proposed.
No comment so far.

As the author of pyclewn, a Vim front end to pdb and gdb, I would be
grateful for any progress on this problem.
msg154933 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-03-05 09:19
It's an interesting issue. Thanks for the report and patch, Xavier. I am setting patch to needing review.
msg159706 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-04-30 16:53
Hello Xavier,

This issue required some tracing through the calls and I see the problem that you have mentioned and patch fixes the problem.

One comment on the patch, for the tests in the module, this line - 

self.frame_returning = None 

does not seem to have a coverage. Is there a specific significance in setting this to None and would a test help?


     def dispatch_return(self, frame, arg):
         if self.stop_here(frame) or frame == self.returnframe:
+            self.frame_returning = frame
             self.user_return(frame, arg)
+            self.frame_returning = None
             if self.quitting: raise BdbQuit
         return self.trace_dispatch

Sorry for the delay, I shall commit the fix in 3.3.
msg159734 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2012-04-30 20:00
Hi Senthil,

Thanks for your help with this issue.

self.frame_returning is both a flag to indicate that we are returning
from the current frame and a value (the current frame). We need both
as set_step() (the method invoked when the user runs the step command)
does not know the current frame and wether we are returning from the
current frame.

Here is a raw sketch of the call chain in the case where the user
types the step command on returning from the current frame (Pdb
subclasses both bdb.Bdb and cmd.Cmd):

Bdb::dispatch_return
    Pdb::user_return (Bdb overriden method)
        Pdb::interaction
            Cmd::cmdloop
                Cmd::onecmd
                    Pdb::do_step
                        Bdb::set_step

So self.frame_returning must be set to None after the call to
self.user_return() so that its value is not used in another later step
command, where we are not returning from this frame. Actually it is
more explicit and more robust to use a try-finally clause, such as:

    def dispatch_return(self, frame, arg):
        if self.stop_here(frame) or frame == self.returnframe:
            try:
                self.frame_returning = frame
                self.user_return(frame, arg)
            finally:
                self.frame_returning = None
            if self.quitting: raise BdbQuit
        return self.trace_dispatch


Xavier
msg159741 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-01 02:37
New changeset 96cb47f8142e by Senthil Kumaran in branch '3.2':
issue13183 - Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye
http://hg.python.org/cpython/rev/96cb47f8142e

New changeset 5ea23739e9ba by Senthil Kumaran in branch '2.7':
issue13183 - Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye
http://hg.python.org/cpython/rev/5ea23739e9ba

New changeset ab63e874265e by Senthil Kumaran in branch 'default':
issue13183 - Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye
http://hg.python.org/cpython/rev/ab63e874265e
msg159742 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-05-01 02:49
Hello Xavier,

Thanks for the explanation. Understood the reason for setting the frame_returning to None. The patch is committed in all python versions and the issue is fixed now. Thanks for the patch and prodding through this.

-- 
Senthil
msg159761 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-01 18:02
The test fails on Windows. Whereas on Unix, the two step commands produce this output:

-> print('1')
(Pdb) step
1
--Return--
> /net/pao/export/home/staff/loewis/work/33/bar.py(2)bar()->None
-> print('1')
(Pdb) step
--Return--
> /net/pao/export/home/staff/loewis/work/33/main.py(5)foo()->None
-> bar()
(Pdb) quit

on Windows, they produce this output:

-> print('1')
(Pdb) step
--Call--
> c:\users\martin\33\python\lib\encodings\cp850.py(18)encode()
-> def encode(self, input, final=False):
(Pdb) step
> c:\users\martin\33\python\lib\encodings\cp850.py(19)encode()
-> return codecs.charmap_encode(input,self.errors,encoding_map)[0]
(Pdb) quit

I.e. the stepping enters the print, and breaks in the codec.

Reopening the issue.
msg159763 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2012-05-01 18:47
My fault :(
The call to print is useless for the test, so I suggest to replace it
with a plain 'pass' statement.
msg159765 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-01 19:06
That indeed makes the test pass.
msg159769 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-02 00:01
New changeset 3b2aa777b725 by Senthil Kumaran in branch '2.7':
fix windows test failure - issue13183
http://hg.python.org/cpython/rev/3b2aa777b725

New changeset d17ecee3f752 by Senthil Kumaran in branch '3.2':
fix windows test failure - issue13183
http://hg.python.org/cpython/rev/d17ecee3f752

New changeset 0269c592c8b1 by Senthil Kumaran in branch 'default':
fix closes issue13183 - windows test failure
http://hg.python.org/cpython/rev/0269c592c8b1
msg159771 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-05-02 02:03
Fixed again with replacing print with pass.

But it is strange behavior that "stepping through" enters print in Windows and does not so in Unix. What's the difference in windows that could cause this? Not sure if this was expected behavior.
msg159774 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-05-02 05:38
> But it is strange behavior that "stepping through" enters print in
> Windows and does not so in Unix. What's the difference in windows
> that could cause this? Not sure if this was expected behavior.

On Unix, the codec most likely is UTF-8, which is directly written
in C, i.e. doesn't support Python single-stepping.
msg159775 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-02 05:41
New changeset 6c9ce7e34511 by Martin v. Löwis in branch 'default':
Issue #13183: Revert 0b53b70a40a0 (reenable test on windows)
http://hg.python.org/cpython/rev/6c9ce7e34511
msg159813 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-05-02 18:42
All 3.2 and 2.7 buildbots are still broken:
http://python.org/dev/buildbot/all/builders/x86 OpenIndiana 3.2/builds/1080

"""
======================================================================
FAIL: test_issue13183 (test.test_pdb.PdbTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.2.cea-indiana-x86/build/Lib/test/test_pdb.py", line 662, in test_issue13183
    'Fail to step into the caller after a return')
AssertionError: 'main.py(5)foo()->None' not found in '-> bar()' : Fail to step into the caller after a return

----------------------------------------------------------------------
Ran 2 tests in 0.558s
"""
msg159818 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2012-05-02 20:05
The test has been changed in the default branch by changeset
1b174a117e19.  This change replaces the assertIn by a less restrictive
assertTrue. These changes should also probably be made in 3.2 and 2.7
and hopefully this will fix the problem in 3.2 and 2.7.

The changeset 1b174a117e19 in the default branch is:

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -604,6 +604,7 @@
         filename = 'main.py'
         with open(filename, 'w') as f:
             f.write(textwrap.dedent(script))
+        self.addCleanup(support.unlink, filename)
         cmd = [sys.executable, '-m', 'pdb', filename]
         stdout = stderr = None
         with subprocess.Popen(cmd, stdout=subprocess.PIPE,
@@ -660,9 +661,11 @@
         """
         with open('bar.py', 'w') as f:
             f.write(textwrap.dedent(bar))
+        self.addCleanup(support.unlink, 'bar.py')
         stdout, stderr = self.run_pdb(script, commands)
-        self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3],
-                         'Fail to step into the caller after a return')
+        self.assertTrue(
+            any('main.py(5)foo()->None' in l for l in stdout.splitlines()),
+            'Fail to step into the caller after a return')
msg160065 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-06 09:32
If the failures don't get fixed, the offending commit should be reverted.
msg160067 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-06 09:49
New changeset e275a9f7daa9 by Georg Brandl in branch '3.2':
#13183: backport fixes to test_pdb to 3.2 branch
http://hg.python.org/cpython/rev/e275a9f7daa9
msg160068 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-06 09:54
New changeset 2644e4ea02d3 by Georg Brandl in branch '2.7':
#13183: backport fixes to test_pdb to 2.7 branch
http://hg.python.org/cpython/rev/2644e4ea02d3
msg160069 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-05-06 09:54
Should be fixed now.
msg160168 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2012-05-07 19:53
Note that now that this issue is fixed, issue 14743 has become more
visible.
msg160301 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-05-09 15:44
> Georg Brandl <georg@python.org> added the comment:
> 
> Should be fixed now.

Thanks for the commit fix, Georg. The comment on buildbot failures had
escaped my attention. Sorry for that.
History
Date User Action Args
2012-05-09 15:44:04orsenthilsetmessages: + msg160301
2012-05-07 19:53:05xdegayesetmessages: + msg160168
2012-05-06 09:54:32georg.brandlsetstatus: open -> closed

messages: + msg160069
2012-05-06 09:54:04python-devsetmessages: + msg160068
2012-05-06 09:49:56python-devsetmessages: + msg160067
2012-05-06 09:32:05pitrousetnosy: + pitrou
messages: + msg160065
2012-05-02 20:05:18xdegayesetmessages: + msg159818
2012-05-02 18:42:04neologixsetstatus: closed -> open
nosy: + neologix
messages: + msg159813

2012-05-02 05:41:30python-devsetmessages: + msg159775
2012-05-02 05:38:59loewissetmessages: + msg159774
2012-05-02 02:03:37orsenthilsetmessages: + msg159771
2012-05-02 00:01:16python-devsetstatus: open -> closed

messages: + msg159769
2012-05-01 19:06:29loewissetmessages: + msg159765
2012-05-01 18:47:10xdegayesetmessages: + msg159763
2012-05-01 18:02:06loewissetstatus: closed -> open
nosy: + loewis
messages: + msg159761

2012-05-01 02:49:47orsenthilsetstatus: open -> closed
resolution: fixed
messages: + msg159742

stage: resolved
2012-05-01 02:37:58python-devsetnosy: + python-dev
messages: + msg159741
2012-04-30 20:00:54xdegayesetmessages: + msg159734
2012-04-30 16:53:16orsenthilsetassignee: orsenthil
messages: + msg159706
2012-03-06 10:10:17tshepangsetnosy: + tshepang
2012-03-05 09:19:39orsenthilsetversions: + Python 2.7, Python 3.3
nosy: + orsenthil

messages: + msg154933

keywords: + needs review, - patch
2012-03-04 12:13:34xdegayesetmessages: + msg154889
2011-10-17 14:47:48meador.ingesetnosy: + meador.inge
2011-10-17 05:48:18georg.brandlsetnosy: + georg.brandl
2011-10-16 19:24:29xdegayesetfiles: + restore_trace.py27.patch

messages: + msg145634
2011-10-16 19:23:11xdegayesetfiles: + restore_trace_2.patch

messages: + msg145633
2011-10-14 18:57:13xdegayesettype: behavior
2011-10-14 18:56:31xdegayecreate