classification
Title: list comprehensions don't see local variables in pdb in python3
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Jurko.Gospodnetić, blueyed, flox, georg.brandl, kay, r.david.murray, xdegaye
Priority: normal Keywords: patch

Created on 2014-04-05 11:55 by kay, last changed 2019-08-09 15:13 by blueyed. This issue is now closed.

Files
File name Uploaded Description Edit
genexp.patch xdegaye, 2014-04-12 09:28 review
default.patch xdegaye, 2014-05-20 15:30 review
Pull Requests
URL Status Linked Edit
PR 15194 open blueyed, 2019-08-09 15:13
Messages (10)
msg215595 - (view) Author: Kay (kay) Date: 2014-04-05 11:55
Using generators in pdb are very handy but since Python3 they don't work properly. For example:

import pdb
def foo():
  items = [1, 2, 3]
  limit = 5
  pdb.set_trace()

foo()

in pdb prompt the following fails:

(pdf) all(x < limit for x in items)
*** NameError: global name 'items' is not defined

I can express that in a lambda expression (i.e. pass items as an argument) but this seems unnecessary and very inelegant.
msg215625 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-04-05 19:33
Your failure appears to be not pasted from an interpreter session; the actual failure is:

(Pdb) all(x < limit for x in items)
*** NameError: global name 'limit' is not defined

(i.e. "limit" is not found, not "items").  This actually does not work in Python 2 either.  What did work in Python 2 and doesn't work in 3, is using a list comprehension like

all([x < limit for x in items])

This is because list comprehensions are now implemented with their own function object like generator expressions have always been.  To make the code in pdb compile "as if" it was put in the debugged function will be as good as impossible, so I'm closing this as won't fix.
msg215876 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2014-04-10 13:01
Just ran into this problem and it's sooooo uncomfortable
researching dynamic structures at run-time using PDB
without this. :-(

As a workaround, you can use a trick similar to this one
to 'import' your locals into the list comprehension body:

[l['r'](x) for l in (locals(),) for x in l['some_local']]

assuming 'r' & 'some_local' are two local variables in
your surrounding scope.

Ugly, but at least it can be made/forced to work if needed...

Best regards,
  Jurko Gospodnetić
msg215963 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-04-12 07:09
FWIW this generator expression can be evaluated with the 'interact' command:

--Return--
> /test/comprehension.py(5)foo()->None
-> pdb.set_trace()
(Pdb) interact
*interactive*
>>> all(x < limit for x in items)
True
>>> 
(Pdb)
msg215966 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-04-12 09:28
The runcode() method of InteractiveInterpreter in code.py uses the 'self.locals' dictionary as the 'globals' parameter of the invoked exec() function. And the do_interact() method of Pdb instantiates InteractiveInterpreter with 'locals' as a merge of the current frame's locals and globals dictionary. This explains why the interact command of pdb evaluates sucessfully the generator expression: the generator function object is evaluated by the interpreter in a frame where 'locals' is NULL (see fast_function() in ceval.c) and 'globals' includes now the debugged frame locals dictionary.

So a fix for this problem is to have the default() method of pdb be implemented in the same manner as do_interact() and the runcode() method of InteractiveInterpreter. The attached patch does this.
msg215967 - (view) Author: Jurko Gospodnetić (Jurko.Gospodnetić) * Date: 2014-04-12 10:03
Thanks for looking into this Xavier.

Could someone reopen this issue so it can gets looked at again?
msg216033 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-04-13 20:12
The NameError exception occuring on a generator expression referencing a local variable when the generator is called within exec() is the object of multiple entries in the bug tracker, see issue 13557.

msg 149096 in this issue suggests using exec(code, locals()) to fix the problem. It seems that what does currently the do_interact() method, and what is proposed in the patch is the recommended practice to handle this problem.
msg216171 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-04-14 18:59
Interestingly, the interact command was added at changeset:

$ hg log -v --pager cat -r $(hg blame Lib/pdb.py | grep do_interact | awk -F : '{print $1}')
changeset:   66691:c2578a68879d
user:        Georg Brandl <georg@python.org>
date:        Sat Dec 04 11:20:26 2010 +0000
files:       Doc/library/pdb.rst Lib/pdb.py Misc/NEWS
description:
Add the "interact" pdb command from pdb++.


And the included documentation, added at that time, states (emphasis added by me):
Start an interative interpreter (using the code module) whose global namespace contains all the ***(global and local)*** names found in the current scope.


I can provide a test case for the patch when this issue is re-opened, unless someone else is willing to do it.
msg218847 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-05-20 15:30
The patch fails to invoke exec() with the locals argument set to the current frame locals.
The attached patch fixes this, and test_pdb runs now fine with it.
msg348829 - (view) Author: daniel hahler (blueyed) * Date: 2019-08-01 00:25
Georg, please consider re-opening.
This patch here appears to fix this after all.
History
Date User Action Args
2019-08-09 15:13:04blueyedsetpull_requests: + pull_request14926
2019-08-01 00:25:28blueyedsetnosy: + blueyed

messages: + msg348829
versions: + Python 3.6, Python 3.7, Python 3.8, Python 3.9
2014-12-02 15:20:45floxsetnosy: + flox
2014-05-20 15:30:03xdegayesetfiles: + default.patch

messages: + msg218847
2014-04-14 18:59:44xdegayesetmessages: + msg216171
2014-04-13 20:12:38xdegayesetmessages: + msg216033
2014-04-12 10:03:58Jurko.Gospodnetićsetmessages: + msg215967
2014-04-12 09:28:16xdegayesetfiles: + genexp.patch
keywords: + patch
messages: + msg215966
2014-04-12 07:09:51xdegayesetnosy: + xdegaye
messages: + msg215963
2014-04-10 13:01:46Jurko.Gospodnetićsetnosy: + Jurko.Gospodnetić

messages: + msg215876
versions: + Python 3.4, Python 3.5
2014-04-05 19:33:03georg.brandlsetstatus: open -> closed
resolution: wont fix
messages: + msg215625
2014-04-05 19:16:13kaysetnosy: + georg.brandl
2014-04-05 13:22:51r.david.murraysetnosy: + r.david.murray
2014-04-05 12:08:36kaysettype: behavior
2014-04-05 11:55:43kaycreate