classification
Title: Skip stack unwinding when "next", "until" and "return" pdb commands executed in generator context
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: asvetlov, georg.brandl, gvanrossum, larry, pconnell, pitrou, python-dev, xdegaye
Priority: release blocker Keywords: patch

Created on 2012-12-02 23:23 by asvetlov, last changed 2014-03-10 10:12 by xdegaye. This issue is now closed.

Files
File name Uploaded Description Edit
issue16596.diff asvetlov, 2012-12-02 23:30 review
issue16596_v2.diff asvetlov, 2012-12-04 11:44 review
issue16596_nostate.diff xdegaye, 2012-12-06 19:45 review
issue16596_nostate_2.diff xdegaye, 2012-12-11 17:04 review
issue16596_nostate_3.diff xdegaye, 2013-11-19 10:27 review
issue16596_nostate_4.diff xdegaye, 2013-11-20 22:02 review
issue16596_leak.diff pconnell, 2013-11-23 12:28 Fix for ref leak in committed patch review
pdb_doc.diff xdegaye, 2014-03-10 10:12 review
Messages (27)
msg176815 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-12-02 23:23
GvR in http://mail.python.org/pipermail/python-ideas/2012-November/017991.html has requested for improving pdb to jump over yield instead of following it.
msg176817 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-12-02 23:30
Patch attached.
It modifies "next", "return" and "until" commands behavior when they are executed if current debugged function is generator.
1. "next" skips stack unwindind if yield tries to follow it. Now it goes just to next executed line.
2. "until" does the same: skips all yields until target is reached.
3. "return" waits to return *from* generator (by explicit or implicit StopIteration or GeneratorExit exception) ignoring following yields inside generator.
msg176862 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-12-03 18:48
Thanks! I will try it out shortly.
msg176898 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-12-04 11:44
Rename skip_yield to skipyield
msg177050 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2012-12-06 19:44
In the test named 'test_pdb_return_command_for_generator' in the patch, the
return command does not cause pdb to stop at the StopIteration debug event as
expected. Instead the following step command steps into the generator.

With the patch applied, in the following debugging session, pdb does not stop
after the 'next' command.  Pdb should stop to stay consistent with the way
'next' behaves in the same case on a plain function.

===   bar.py   ===
def g():
    yield 0

it = g()
while True:
    try:
        x = next(it)
        print(x)
    except StopIteration:
        break
==================
$ ./python -m pdb /tmp/bar.py
> /tmp/bar.py(1)<module>()
-> def g():
(Pdb) break g
Breakpoint 1 at /tmp/bar.py:1
(Pdb) continue
> /tmp/bar.py(2)g()
-> yield 0
(Pdb) next
0
The program finished and will be restarted
> /tmp/bar.py(1)<module>()
-> def g():
(Pdb)
==================
msg177051 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2012-12-06 19:45
This new patch fixes the two problems described in my previous message.
The patch is different from Andrew's patch in that it does not use a new state
variable, and the test cases in the patch are a copy of Andrew's patch except
for test_pdb_return_command_for_generator.
msg177059 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2012-12-06 21:04
When the generator is used in a for loop, the interpreter handles the
StopIteration in its eval loop, and the exception is not raised. So it may be
considered as confusing to have pdb behave differently with a generator
depending on its context. A way to fix this would be to not ignore the return
debug events, with the drawback of a more verbose debug process with
generators.
msg177319 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-12-11 00:34
It looks like xdegaye's patch breaks 'n' when not debugging a generator.
msg177344 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2012-12-11 17:04
The 'until' command is also broken (by xdegaye's patch) when issued at a return
debug event and not debugging a generator.

This new patch fixes both problems.

The patch also adds another test case to check that pdb stops after a 'next',
'until' or 'return' command has been issued at a return debug event.
msg202991 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-11-16 00:26
I'd love it if someone could review this.  This would be a great improvement to debugging coroutines in asyncio.
msg202999 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-11-16 01:00
I think this is not ready for inclusion.  It works wonderfully when stepping over a yield[from], but I can't seem to get it to step nicely *out* of a generator.  (Details on request -- basically I put a "pdb.set_trace()" call in Tulip's fetch3.py example and step around.)
msg203309 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2013-11-18 17:25
A description of what goes wrong when stepping out of the generator
would be helpful.
msg203312 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-11-18 18:09
Basically the debugger lost control and the program ran to completion after I hit 'n' that returned from the coroutine.
msg203350 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2013-11-19 10:27
This is a consequence of the problem mentioned in msg 177059 above.

New patch 'issue16596_nostate_3.diff' fixes both problems by having the interpreter
issue an exception debug event when processing a StopIteration in target FOR_ITER:
* The same debug events are issued now, wether the generator is run within a for loop or not.
* 'n' stops now at both the return and exception debug events on returning from the generator.
msg203352 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2013-11-19 10:30
Forgot to say that the only difference between this patch and the previous one is in Python/ceval.c.
msg203423 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-11-19 21:21
It's not fixed.  Let me paste in a session.  This uses the latest Tulip repo (simple_tcp_server.py was just added).  I've added "import pdb; pdb.set_trace()" to the top of the client() coroutine, to set a breakpoint (I'm a very unsophisticated pdb user :-).  I step over a yield-from, great.  Then I step into recv().  Note the final 'n' command.  This is at the return statement in recv().  At this point I expect to go back into the client() coroutine, but somehow the debugger loses control and the program finishes execution without giving control back.

bash-3.2$ ~/cpython/python.exe -m examples.simple_tcp_server
~/cpython/python.exe -m examples.simple_tcp_server
> /Users/guido/tulip/examples/simple_tcp_server.py(119)client()
-> reader, writer = yield from asyncio.streams.open_connection(
(Pdb) n
n
> /Users/guido/tulip/examples/simple_tcp_server.py(120)client()
-> '127.0.0.1', 12345, loop=loop)
(Pdb) 

> /Users/guido/tulip/examples/simple_tcp_server.py(122)client()
-> def send(msg):
(Pdb) 

> /Users/guido/tulip/examples/simple_tcp_server.py(126)client()
-> def recv():
(Pdb) 

> /Users/guido/tulip/examples/simple_tcp_server.py(132)client()
-> send("add 1 2")
(Pdb) 

> add 1 2
> /Users/guido/tulip/examples/simple_tcp_server.py(133)client()
-> msg = yield from recv()
(Pdb) s
s
--Call--
> /Users/guido/tulip/examples/simple_tcp_server.py(126)recv()
-> def recv():
(Pdb) n
n
> /Users/guido/tulip/examples/simple_tcp_server.py(127)recv()
-> msgback = (yield from reader.readline()).decode("utf-8").rstrip()
(Pdb) n
n
> /Users/guido/tulip/examples/simple_tcp_server.py(128)recv()
-> print("< " + msgback)
(Pdb) n
n
< 3.0
> /Users/guido/tulip/examples/simple_tcp_server.py(129)recv()
-> return msgback
(Pdb) n
n
> repeat 5 hello
< begin
< 1. hello
< 2. hello
< 3. hello
< 4. hello
< 5. hello
< end
client task done: Task(<_handle_client>)<result=None>
bash-3.2$
msg203537 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2013-11-20 22:02
Hopefully issue16596_nostate_4.diff should fix this.
The patch issues a StopIteration debug event in ceval.c (similar to the change made in the previous
patch for the for loop), when the subgenerator is exhausted. This debug event is printed as
'Internal StopIteration' by pdb to indicate that it is not a real user exception. Two tests have
been added: test 'next' when returning from a generator in a for loop and 'test' next when returning
from a subgenerator.
msg203543 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-11-20 22:46
This version works beautifully in that scenario!

Does anyone else reading this bug report object to this being committed?
msg203668 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-21 19:30
New changeset 95eea8624d05 by Guido van Rossum in branch 'default':
Better behavior when stepping over yield[from]. Fixes issue 16596. By Xavier de Gaye.
http://hg.python.org/cpython/rev/95eea8624d05
msg203986 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-23 11:06
This commit created a reference leak:

./python -m test -R 3:2 test_trace
[1/1] test_trace
beginning 5 repetitions
12345
.....
test_trace leaked [128, 128] references, sum=256
msg203992 - (view) Author: Phil Connell (pconnell) * Date: 2013-11-23 12:28
It looks like call_exc_trace is leaking refs to Py_None.

I believe the attached patch fixes the issue (it certainly fixes Antoine's failing invokation :)
msg203995 - (view) Author: Phil Connell (pconnell) * Date: 2013-11-23 13:00
Full run of the test suite was clean, so the fix is ready to go.
msg203996 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-23 13:03
Yes, actually, 4f730c045f5f is the culprit.
msg204001 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-23 13:13
New changeset 8f556ee0f6ba by Antoine Pitrou in branch 'default':
Fix refleak introduced by 4f730c045f5f (issue #18408) and unveiled by 95eea8624d05 (issue #16596).
http://hg.python.org/cpython/rev/8f556ee0f6ba
msg204002 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-23 13:13
I committed a simpler fix.
msg213008 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-10 01:35
New changeset 5c6c96c82afb by R David Murray in branch 'default':
whatsnew: pdb works for debugging asyncio programs (#16596).
http://hg.python.org/cpython/rev/5c6c96c82afb
msg213022 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2014-03-10 10:12
Documentation update attached.
History
Date User Action Args
2014-03-10 10:12:38xdegayesetfiles: + pdb_doc.diff

messages: + msg213022
2014-03-10 01:35:04python-devsetmessages: + msg213008
2013-11-23 13:13:35pitrousetstatus: open -> closed

messages: + msg204002
2013-11-23 13:13:09python-devsetmessages: + msg204001
2013-11-23 13:03:36pitrousetmessages: + msg203996
2013-11-23 13:00:01pconnellsetmessages: + msg203995
2013-11-23 12:28:13pconnellsetfiles: + issue16596_leak.diff

messages: + msg203992
2013-11-23 11:06:56pitrousetstatus: closed -> open
nosy: + pitrou
messages: + msg203986

2013-11-21 19:31:35gvanrossumsetstatus: open -> closed
assignee: gvanrossum
resolution: fixed
stage: patch review -> resolved
2013-11-21 19:30:42python-devsetnosy: + python-dev
messages: + msg203668
2013-11-20 22:46:53gvanrossumsetmessages: + msg203543
2013-11-20 22:02:54xdegayesetfiles: + issue16596_nostate_4.diff

messages: + msg203537
2013-11-19 21:21:13gvanrossumsetmessages: + msg203423
2013-11-19 10:30:47xdegayesetmessages: + msg203352
2013-11-19 10:27:25xdegayesetfiles: + issue16596_nostate_3.diff

messages: + msg203350
2013-11-18 18:09:24gvanrossumsetmessages: + msg203312
2013-11-18 17:25:20xdegayesetmessages: + msg203309
2013-11-16 12:58:57pconnellsetnosy: + pconnell
2013-11-16 01:00:52gvanrossumsetmessages: + msg202999
2013-11-16 00:27:07gvanrossumsetpriority: normal -> release blocker
nosy: + larry
2013-11-16 00:26:50gvanrossumsetmessages: + msg202991
stage: patch review
2012-12-11 17:04:39xdegayesetfiles: + issue16596_nostate_2.diff

messages: + msg177344
2012-12-11 00:34:50gvanrossumsetmessages: + msg177319
2012-12-06 21:04:43xdegayesetmessages: + msg177059
2012-12-06 19:45:38xdegayesetfiles: + issue16596_nostate.diff

messages: + msg177051
2012-12-06 19:44:21xdegayesetmessages: + msg177050
2012-12-05 13:54:53xdegayesetnosy: + xdegaye
2012-12-04 11:44:43asvetlovsetfiles: + issue16596_v2.diff

messages: + msg176898
2012-12-03 22:29:14pitrousetnosy: + georg.brandl
2012-12-03 18:48:44gvanrossumsetmessages: + msg176862
2012-12-02 23:47:22asvetlovsettitle: Skip stack unwinding when "next", "until" and "return" pdb commands executed -> Skip stack unwinding when "next", "until" and "return" pdb commands executed in generator context
2012-12-02 23:30:19asvetlovsetfiles: + issue16596.diff
keywords: + patch
messages: + msg176817
2012-12-02 23:23:04asvetlovcreate