This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: code.InteractiveInterpreter fails to change locals when invoked in a function
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, ericp, r.david.murray
Priority: normal Keywords:

Created on 2010-07-16 17:21 by ericp, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
bug87405a.py ericp, 2010-07-16 17:21 Standalone Python code that shows how local vars aren't modified in a function
Messages (8)
msg110464 - (view) Author: Eric Promislow (ericp) Date: 2010-07-16 17:21
Similar to bug http://bugs.python.org/issue5215 which found
a workaround in pdb.  Here I want to use code.InteractiveInterpreter
to modify code interactively (see Komodo bug
http://bugs.activestate.com/show_bug.cgi?id=87405 )

I can do this at the top-level, but not inside a function.  The
attached file shows the problem:
msg110468 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-16 18:34
Apparently I can't even see the activestate bug without creating an account there, so posting a link to it is not helpful.

Please post more details about how you are calling InteractiveInterpreter and what the problem is that you are observing.
msg110471 - (view) Author: Eric Promislow (ericp) Date: 2010-07-16 18:56
I've modified the bug status so anyone can read it.  You
don't need an account to read ActiveState bugs, only to
add or comment on one.

Please note that I closed bug http://bugs.activestate.com/show_bug.cgi?id=87405, as we're now writing
to frame->f_localsplus[] to make sure changes to locals
stick.  I logged a different bug on Komodo's dependence
on this bug at

http://bugs.activestate.com/show_bug.cgi?id=87417

-------------------------------------------------
Here's how we run into this bug:

In Komodo, while you're debugging, you can push an
interactive shell, that uses the current state of the
program.  We build each block of code the user
types in a variable called source, and try executing
it like so: 

code.InteractiveInterpreter(locals()).runsource(source, "<console>")

In other words, we're letting the Python core do all the
heavy lifting identifying multi-line stmts, indented blocks,
etc.

We've had problems with modifying local variables in the
Python debugger for years (almost a decade now).  I got
a C extension working using frame->f_localsplus to
make sure modifications are persisted, but noticed that
changes in the interactive shell weren't being persisted.

I distilled the code we use into the attached file, which
shows that changes aren't being persisted here.  I'm
not an expert on core internals, but I suspect that 
python code "locals()" maps to C code "frame->f_locals",
and we still have the problem that inside functions,
frame->f_locals is a temporary object that is rebuilt on
each access.  From what I've observed, frame->f_localsplus
points to the actual items, and these are the
pointers that need to be changed.
msg110523 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-16 23:08
locals() does not give you a copy of the locals dictionary that you can modify and expect the values to affect the actual locals they were copied from.  This is documented:

http://docs.python.org/library/functions.html#locals

You would need to pass InteractiveInterpreter the "real" locals dict from the frame to make your code work, and it sounds like you've already done this.  However, I don't believe that this is going to work in the general case (eg: in the face of nested scopes), which is why locals() is not updatable. The use case for InteractiveInterpreter is implementing a python-command-line-like utility, and thus has no need to update the locals in the current function.

A doc note about this in InteractiveInterpreter along the lines of the one in the 'exec' docs is almost certainly appropriate, so I'll leave this open as a doc bug.
msg110524 - (view) Author: Eric Promislow (ericp) Date: 2010-07-16 23:14
Thanks for the response.

Note that our use case *is* to implement Python-console
functionality, but sometimes we do this in the context
of a currently running Python program, inside a function.
That's why I wrote the repro that way.  Using
code.InteractiveInterpreter(locals()).runsource(...)
in a function corresponds to interacting with the
debugger in a function, while the second call to
runsource() corresponds to interacting with the
program when it's stepping through top-level code.

Keep in mind that all of this takes place while
control flow of the main program is stuck in the
debugger's read-eval-print loop.

It would be useful if there was a way of accessing the
localsplus container in Python code
msg110525 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-16 23:23
Well, you could discuss that possibility on python-dev, I suppose.

As long as you don't need the changes to persist outside of the debugging loop, you can use InteractiveInterpreter as designed and keep passing it the same (modified) dict every time, as was done in the pdb fix.
msg110527 - (view) Author: Eric Promislow (ericp) Date: 2010-07-16 23:31
David, that won't work for a reason I'll get into in a bit.

I would drop the priority of this ... Komodo has a had a
Python debugger since late 2000, we only fixed updating
variables while debugging (using the variable viewer) yesterday,
and received fewer than 10 complaints about that over the years.

While looking at the code, something seemed wrong with our
handling of the variables in the interactive shell.  I tested
it, and it failed, and then wrote the standalone Python code
to isolate the problem.

Your solution will work sometimes, but not completely.  When
you're in interactive mode while debugging, you can type arbitrary
Python code, but can also use the variable viewer in the UI to change
variables.  These two functions are handled by different code
paths.

My recommendation is to resolve this as a limitation, and
document the module.
msg228118 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-10-01 20:58
@Eric can you provide a doc patch for this?
History
Date User Action Args
2022-04-11 14:57:03adminsetgithub: 53520
2019-04-26 20:43:43BreamoreBoysetnosy: - BreamoreBoy
2014-10-01 20:58:47BreamoreBoysetnosy: + BreamoreBoy

messages: + msg228118
versions: + Python 3.4, Python 3.5, - Python 2.6, Python 3.1, Python 3.2
2010-07-16 23:31:30ericpsetmessages: + msg110527
2010-07-16 23:23:12r.david.murraysetmessages: + msg110525
2010-07-16 23:14:46ericpsetmessages: + msg110524
2010-07-16 23:08:44r.david.murraysetassignee: docs@python
components: + Documentation, - Interpreter Core
versions: + Python 3.1, Python 2.7, Python 3.2
nosy: + docs@python

messages: + msg110523
stage: needs patch
2010-07-16 18:56:58ericpsetmessages: + msg110471
2010-07-16 18:34:22r.david.murraysetnosy: + r.david.murray
messages: + msg110468
2010-07-16 17:21:04ericpcreate