classification
Title: Clarify the required behaviour of locals()
Type: behavior Stage: patch review
Components: Documentation Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: benjamin.peterson, ezio.melotti, flox, gvanrossum, ncoghlan, njs, xgdomingo
Priority: normal Keywords: patch

Created on 2013-05-12 07:46 by ncoghlan, last changed 2017-10-11 02:16 by ncoghlan.

Pull Requests
URL Status Linked Edit
PR 3640 open ncoghlan, 2017-09-18 08:07
Messages (9)
msg188986 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-12 07:46
As proposed at [1], I would like to tighten up the definition of locals so that defining enum members programmatically is officially supported behaviour.

I'll come up with a patch soonish.

[1] http://mail.python.org/pipermail/python-dev/2013-May/125917.html
msg188987 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-12 07:51
See also #17546 and #7083.
msg296880 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-06-26 11:52
Since "soonish" turned out to be "4 years and counting", copying in the specifics of the proposal in from the old python-dev thread:

1. While nominally undefined, in practice lots of Python programs depend on the locals() builtin behaving exactly how it behaves in CPython.

2. PyPy at least has replicated that behaviour faithfully (to the extent of replicating our weird trace function related misbehaviour, as recently pointed out in issue #30744)

3. For module scopes and class scopes (and the corresponding forms of exec and eval), the expected behaviour is relatively straightforward to both define and implement:

* at module scope, as well as when using exec() or eval() with a
single namespace, locals() must return the same thing as globals(),
which must be the actual execution namespace. Subsequent execution may
change the contents of the returned mapping, and changes to the
returned mapping must change the execution environment.

* at class scope, as well as when using exec() or eval() with separate
global and local namespaces, locals() must return the specified local
namespace (which may be supplied by the metaclass __prepare__ method
in the case of classes). Subsequent execution may change the contents
of the returned mapping, and changes to the returned mapping must
change the execution environment. For classes, this mapping will not
be used as the actual class namespace underlying the defined class
(the class creation process will copy the contents to a fresh
dictionary that is only accessible by going through the class
machinery).

4. For function scopes, the appropriate semantics are less clear, as what CPython currently does is fairly weird and quirky.

* actual execution uses the fast locals array and cell references (for nonlocal variables)
* there's a PyFrame_FastToLocals operation that populates the frame's "f_locals" attribute based on the current state of the fast locals array and any referenced cells
* a direct reference to f_locals is returned from locals(), so if you hand out multiple concurrent references, then all those references will be to the exact same dictionary
* the two common calls to the reverse operation, PyFrame_LocalsToFast, were removed in the migration to Python 3: exec is no longer a statement and hence can longer affect function local namespaces, and the compiler now disallows the use of "from module import *" operations at function scope
* however, two obscure calling paths remain: PyFrame_LocalsToFast is called as part of returning from a trace function, and you can also still inject the IMPORT_STAR opcode when creating a function directly from a code object rather than via the compiler

It would be a lot simpler to document the expected behaviour at function scope if locals() were to be updated to return a true snapshot (i.e. a copy of f_locals, rather than a direct reference), with direct access to the shared locals reference requiring going through the frame attribute. That way, trace functions could still modify local variables (since they use `frame.f_locals`), but setting a trace function wouldn't suddenly have the side effect of making modifications to locals() take effect at function scope.
msg296909 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-06-26 15:52
I've tried thinking though a few scenarios, and I think I'm +1 (or at least
+0 or +0.5) on the proposed change to locals(), and of course I'm happy
that we're going to specify its behavior better.
msg297019 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-06-27 08:59
Guido: perhaps I should run this update through the PEP process?

Even though the actual proposed change is only to a pretty obscure edge case (having multiple concurrent live references to the result of locals() for a function namespace), the extra visibility should help developers of alternative implementations be clear on what is happening.
msg297328 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-06-29 23:47
Yeah, I think a short PEP would be helpful here.
msg301742 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-09-08 22:00
Posted as PEP 558:

* https://github.com/python/peps/blob/master/pep-0558.rst
* https://www.python.org/dev/peps/pep-0558/
msg302423 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-09-18 08:31
Status update: I've posted an initial PR to issue 30744 that relies on the trace hook semantic change proposed in the PEP to resolve the trace hook/cell reference incompatibility reported there.

That provides confidence that it really is only the semantics of *trace hooks* that we need to change, rather than anything about locals() or frame.f_locals in general.

So the next steps will be to do a final editing pass on the current PEP to account for the reference implementation, and then send it to python-dev for official review and pronouncement.
msg304100 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-10-11 02:16
Nathaniel raised a valid concern about the draft PEP over in https://bugs.python.org/issue30744#msg302475, so I've been considering whether or not it would be possible to make the write-through proxy idea work without introducing other problems.

I think I have a workable design concept for that approach now: https://bugs.python.org/issue30744#msg304099

So the next step will be to see if that actually does work as well as I think it will, and if so, update the PEP accordingly.
History
Date User Action Args
2017-10-11 02:16:40ncoghlansetnosy: + njs
messages: + msg304100
2017-09-18 08:31:02ncoghlansetmessages: + msg302423
2017-09-18 08:19:15ncoghlanlinkissue30744 dependencies
2017-09-18 08:07:19ncoghlansetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request3635
2017-09-08 22:00:37ncoghlansetmessages: + msg301742
2017-06-30 02:21:11ncoghlansetassignee: ncoghlan
2017-06-29 23:47:12gvanrossumsetmessages: + msg297328
2017-06-27 08:59:18ncoghlansetmessages: + msg297019
2017-06-26 16:30:22xgdomingosetnosy: + xgdomingo
2017-06-26 15:52:46gvanrossumsetmessages: + msg296909
2017-06-26 11:52:52ncoghlansetmessages: + msg296880
2017-06-26 03:11:29ncoghlansetversions: + Python 3.7, - Python 3.4
2015-07-21 07:08:33ethan.furmansetnosy: - ethan.furman
2015-06-28 03:21:42ncoghlansetassignee: ncoghlan -> (no value)
2013-05-13 01:29:18ethan.furmansetnosy: + ethan.furman
2013-05-12 08:12:05floxsetnosy: + flox
2013-05-12 07:51:25ezio.melottisetnosy: + ezio.melotti
messages: + msg188987
2013-05-12 07:46:47ncoghlancreate