classification
Title: Request for locals().update() to work, it is
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: steven.daprano, terry.reedy, xuancong84
Priority: normal Keywords:

Created on 2021-05-04 05:05 by xuancong84, last changed 2021-05-10 03:21 by xuancong84.

Messages (5)
msg392852 - (view) Author: wang xuancong (xuancong84) Date: 2021-05-04 05:05
In general, the ability to update local variables is very important and it simplifies life tremendously. For example, in machine learning, it allows saving/loading arbitrary-format datasets and arbitrary-structure neural networks (NN) using a single line of code. In computer games, no matter how many complex data structures are there, saving and loading can be done in a single line.

Imagine you are game developer or a deep neural network (DNN) researcher, if all local variables are serializable, then no matter how complicated your game or your DNN structure is, saving the entire game or DNN (to STDOUT) can be simply put into one line as `print(locals())`, and loading the entire game or DNN (from STDIN) can be simply put into one line as `locals().update(eval(sys.stdin.read()))`.

Currently, `globals().update(...)` takes immediate effect but `locals().update(...)` does not work because Python documentation says:

> The default locals act as described for function locals() below:
> modifications to the default locals dictionary should not be
> attempted. Pass an explicit locals dictionary if you need to see
> effects of the code on locals after function exec() returns.

Why they design Python in such way is because of optimization and conforming the `exec` statement into a function:

> To modify the locals of a function on the fly is not possible without
> several consequences: normally, function locals are not stored in a
> dictionary, but an array, whose indices are determined at compile time
> from the known locales. This collides at least with new locals added
> by exec. The old exec statement circumvented this, because the
> compiler knew that if an exec without globals/locals args occurred in
> a function, that namespace would be "unoptimized", i.e. not using the
> locals array. Since exec() is now a normal function, the compiler does
> not know what "exec" may be bound to, and therefore can not treat is
> specially.

Since `global().update(...)` works, the following piece of code will work in root namespace (i.e., outside any function) because locals() is the same as globals() in root namespace:
```
locals().update({'a':3, 'b':4})
print(a, b)
```
But this will not work inside a function.

I have explored a few ways of hacking updating locals() on Python 3, it seems there is no way so far. The following piece of code seems to works:
```
def f1():
  sys._getframe(1).f_locals.update({'a':3, 'b':4})
  print(a, b)

f1()
```
However, that is because `sys._getframe(1)` is the root namespace, so `sys._getframe(1).f_locals.update()` is essentially `globals().update()`.

From the above Python developer documentation, I understand that in Python 2, local namespace lookup has 2 modes: optimized mode if there is no `exec` statement, un-optimized mode if there exists an `exec` statement. But in Python 3, `exec` becomes a function, so the compiler cannot determine which namespace optimization mode at compile time (because `exec` can be overridden or aliased into a different name). Therefore, Python 3 has only optimized namespace lookup. My suggestion is that whenever this optimized local namespace lookup fails, perform an un-optimized lookup (which will include locals()). This should solve the problem.

Do you have any other ideas or suggestions for doing this? Thanks!
msg392856 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2021-05-04 07:35
> loading the entire game or DNN (from STDIN) can be simply put into one line as `locals().update(eval(sys.stdin.read()))`

This is how you get command injection attacks.

https://owasp.org/www-community/attacks/Command_Injection

https://cwe.mitre.org/data/definitions/77.html
msg392860 - (view) Author: wang xuancong (xuancong84) Date: 2021-05-04 08:16
Of course, I am aware of that. As elite-level Python programmers, we should all be aware of security issues whenever we deal with exec() and eval().
msg393238 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-05-08 05:45
When quoting, please include the source, preferably a direct link.

In Python 1 and 2, there were two function local namespace *implementations* -- array index by number or dictionary indexed by name.  The implementation, and hence the lookup mode, was fixed at compile time.  In Python 3, the one *implementation*, and its lookup mode, are fixed.  The slower implementation was dropped because it was not thought worth the bother.  

When you invoke the save function while playing a game, I am imagine that the save function does not have access to and does not same the locals of whatever function was executing at the time you hit the save key.  Rather a game and player states are serialized, and likely not in one line of code.
msg393347 - (view) Author: wang xuancong (xuancong84) Date: 2021-05-10 03:21
Thanks @terry.reedy for your expert-level good comments!

1.
"In Python 3, the one *implementation*, and its lookup mode, are fixed.  The slower implementation was dropped because it was not thought worth the bother."
If I remember correctly, the performance penalty due to the slower lookup mode is not quite significant, in most Python benchmarks, Python2 still performs much faster than Python3 because most codes that need speed does not contain exec/eval, so the slow mode won't affect in practice.

2.
"When you invoke the save function while playing a game, I am imagine that the save function does not have access to and does not same the locals of whatever function was executing at the time you hit the save key.  Rather a game and player states are serialized, and likely not in one line of code."
I have personally tried this on one implementation of deep neural network using Tensorflow, it works pretty well, especially on saving the network parameters at every Nth epoch, or resuming training from a particular epoch. The biggest advantage is that it does not scale with network size or complexity, so the Python code size has a O(1) complexity with network size/complexity and that is a small constant O(1) as it does not involve any Python loop. In practice, you can select what to save/load, such as those not starting with '_'.
History
Date User Action Args
2021-05-10 03:21:06xuancong84setmessages: + msg393347
2021-05-08 05:45:54terry.reedysetnosy: + terry.reedy
messages: + msg393238
2021-05-04 08:16:44xuancong84setmessages: + msg392860
2021-05-04 07:35:32steven.dapranosetnosy: + steven.daprano
messages: + msg392856
2021-05-04 05:05:16xuancong84create