Title: pdb fails to access variables closed over
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.3, Python 3.4, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Chun-Yu Tseng, Jesús Gómez, georg.brandl, xdegaye
Priority: normal Keywords: patch

Created on 2016-01-10 05:16 by Antony.Lee, last changed 2016-11-22 11:54 by Chun-Yu Tseng.

File name Uploaded Description Edit
default.patch Chun-Yu Tseng, 2016-10-16 09:55 show proper information when Pdb fails to create a closure review
free_variables.patch xdegaye, 2016-11-16 20:37 review
Messages (10)
msg257888 - (view) Author: Antony Lee (Antony.Lee) * Date: 2016-01-10 05:16
Consider the following example:

    def f(x=1):
        def g():
            y = 2
            raise Exception


$ python -mpdb -ccontinue
<... traceback ...>
> /tmp/
-> raise Exception
(Pdb) p x
##### this can be worked around using "up"
*** NameError: name 'x' is not defined
(Pdb) p y
(Pdb) p (lambda: y)()
##### this is more awkward to work around, e.g. (lambda *, y=y: y)()
*** NameError: name 'y' is not defined

Use case: I wan to pass a lambda to a numerical optimizer, but the optimizer fails using the default starting point.  Within pdb, I'd like to pass a different starting point but the same lambda, to see whether this helps.
msg277181 - (view) Author: Jesús Gómez (Jesús Gómez) Date: 2016-09-21 20:01

Another use case is the use any lambda, or function definition inside the scope of a function, for checking conditions in the REPL.

Suppose two inner functions named condition1 and condition2, and a parameter X as a Collection:

(Pdb) any(condition1(x) and condition2(x) for x in X)
*** NameError: name 'condition1' is not defined
msg277578 - (view) Author: Chun-Yu Tseng (Chun-Yu Tseng) * Date: 2016-09-28 03:57
What Antony Lee mentioned are two different cases. 

The former is what PDB should behave because it is not reasonable to inspect a variable does not exist in the current frame. If you want to do so, you have to reference the variable `x` as a closure inside inner function `g` in your source code before running PDB.

The latter is same as what Jesús Gómez confirmed. It's a problem about creating correct closures in an interaction prompt of PDB. It can be found in almost every versions of Python. The following are several forms of the problem:

# (1) raise exception 

  1      def f():
  2          x = 2
  3  ->        import pdb; pdb.set_trace();
(Pdb) (lambda: x)()
*** NameError: name 'x' is not defined

# (2) no exception, but get the wrong value

  1      x = 100
  2      def f():
  3          x = 2
  4  ->        import pdb; pdb.set_trace();
  5      f()
(Pdb) x
(Pdb) (lambda: x)()

# (3) similar to (1), but this one usually makes me upset if I forget the cause of the problem

(Pdb) ll
  1      def f():
  2  ->        import pdb; pdb.set_trace();
(Pdb) x = 5
(Pdb) [i for i in range(10) if (i % x) == 0]
*** NameError: name 'x' is not defined
(Pdb) x

It's easy to alleviate the problem by using `interact` command to start an interactive interpreter.

  1      def f():
  2  ->        import pdb; pdb.set_trace();
(Pdb) x = 5
(Pdb) interact
>>> [i for i in range(10) if (i % x) == 0]
[0, 5]

Now the behavior looks right.

However, the problem still exists for those PDB users don't know how to pass it, and it's quite inconvenient to start an interactive interpreter every time. Maybe it's worth to keep the behavior of PDB consistent and expectable just like the behavior inside interactive interpreter.

I will try to send a patch to solve the problem in these days. Please stop me if it's inappropriate.
msg278761 - (view) Author: Chun-Yu Tseng (Chun-Yu Tseng) * Date: 2016-10-16 09:55

After digging into the implementation of how CPython handles closures, I am sure that it is impossible to solve this problem now correctly. What I can do is create a patch to let Pdb print enough information when the problem occurs. It would look like:

(Pdb) ll
  3      def f():
  4          x = 1
  5  ->        import pdb; pdb.set_trace()
(Pdb) (lambda: x)()

                Pdb can not use the local variable 'x' to create a closure in
                your evaluation. Instead, it tries to use 'x' in globals().
                This behavior is due to the limitation of dynamic
                interpretation within a debugger session.

                Hint: type 'help interact' to check 'interact' command.

*** NameError: name 'x' is not defined

I believe it would be helpful and less annoyed for those who encounters this problem. Call for maintainers review, please.

1. Ruby does not have this problem. Whether it exists depends on how a programming language to implement closures. However, I think that what CPython do (by `PyCellObject`) is smart and efficient.

2. I tried to solve the problem by copying local variables to globals() (just like what `interact` command do), but it results in **more** problems. The most typical one is that if I eval `globals()` in such an affected environment, I will always get the wrong result.

3. I also tried to patch and evaluate again what user inputs when catching a closure-related `NameError`. Obviously, it is not a good idea due to side effects of evaluation.

4. The last possible solution I think of is import `ast` and do some hacking ... it's overkill, and it also brings up other side effects.
msg280912 - (view) Author: Chun-Yu Tseng (Chun-Yu Tseng) * Date: 2016-11-16 05:12
Call for review again.

Maybe xdegaye would like to take a look?
I found this related issue: #21161
msg280930 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-11-16 09:32
It seems that the last patch in issue 21161 fixes all the problems described here, msg 149096 explains why. Can you confirm that this is a duplicate of issue 21161.
msg280933 - (view) Author: Chun-Yu Tseng (Chun-Yu Tseng) * Date: 2016-11-16 10:53
The last patch in #21161 fixes some problems but also brings up critical issues:

(Pdb) list .
  1      y = 2
  3      def f():
  4          y = 9
  5          z = 10
  6  ->     import pdb; pdb.set_trace();
  7      f()
(Pdb) globals()['y']
(Pdb) global y; print(y)
(Pdb) globals()['z']

I think that we should not copy local variables to globals()  while doing execution. It will always bring out the wrong result of `globals()`.

So, the patch I proposed here is focused on "Showing Friendly Error Message" to let the users be less confused. 

# The patch works when a user tries to bound a free variable to a list comprehension. It will show the proper error message, too.
msg280988 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-11-16 20:37
This patch fixes the problems raised in this issue and allows accessing the globals at the Pdb prompt with the globals() dictionary:

(Pdb) list
  1     y = 2
  3     def f():
  4         y = 9
  5         z = 10
  6  ->     import pdb; pdb.set_trace();
  7     f()
(Pdb) globals()['y']
msg281016 - (view) Author: Chun-Yu Tseng (Chun-Yu Tseng) * Date: 2016-11-17 01:50
Your solution is quite neat. 
But it still misses use cases of the `global` statement:

  1      y = 2
  3      def f():
  4          y = 9
  5  ->     import pdb; pdb.set_trace();
  7      f()
(Pdb) global y; y
(Pdb) global y; y += 1; y
(Pdb) globals()['y']
msg281469 - (view) Author: Chun-Yu Tseng (Chun-Yu Tseng) * Date: 2016-11-22 11:54
Hey xdegaye, have you confirmed it?
Date User Action Args
2016-11-22 11:54:13Chun-Yu Tsengsetmessages: + msg281469
2016-11-17 01:50:55Chun-Yu Tsengsetmessages: + msg281016
2016-11-16 20:37:32xdegayesetfiles: + free_variables.patch

messages: + msg280988
2016-11-16 10:53:37Chun-Yu Tsengsetmessages: + msg280933
2016-11-16 09:32:30xdegayesetmessages: + msg280930
2016-11-16 05:23:36Antony.Leesetnosy: - Antony.Lee
2016-11-16 05:12:46Chun-Yu Tsengsetmessages: + msg280912
2016-10-16 09:55:22Chun-Yu Tsengsetfiles: + default.patch
keywords: + patch
messages: + msg278761
2016-09-28 03:57:15Chun-Yu Tsengsetnosy: + Chun-Yu Tseng

messages: + msg277578
versions: + Python 2.7, Python 3.3, Python 3.4, Python 3.7
2016-09-21 20:01:54Jesús Gómezsetnosy: + Jesús Gómez
messages: + msg277181
2016-01-16 10:28:31xdegayesetnosy: + xdegaye
2016-01-15 12:44:07SilentGhostsetnosy: + georg.brandl
type: behavior
2016-01-10 05:16:24Antony.Leecreate