Author dmalcolm
Recipients alex, belopolsky, benjamin.peterson, brett.cannon, dmalcolm, jhylton, nnorwitz, orsenthil, pitrou, rhettinger, sdahlbac, thomaslee, titanstar
Date 2010-11-20.21:11:18
SpamBayes Score 0.0
Marked as misclassified No
Message-id <>
Thanks for reading through this.

> There are a couple places you mention not doing the optimization when
> specific functions are used (e.g. dir, globals, locals), how exactly do you
> verify that, given those functions could be bound to any name?

I don't: I'm only looking at explicit references to these global names:
   BUILTINS_THAT_READ_LOCALS = {'locals', 'vars', 'dir'}

but I haven't attempted to track these at the object level, just at the name level.

This means that if someone is passing these around bound to a different name, the results could be "wrong": the optimizations that I'm doing are synthesizing new local and global variables (in each case with a '__' prefix), and sometimes eliminating local variables.

However, it should work as expected for the simple and common case for code like this:

  def foo():

since the reference to "locals" can be seen at the name level.

But it won't work for something like this:

   def inlined_fn(a):

   def not_inlinable_for_some_reason(b):
       inlined_fn(b) # assume this callsite is inlined

   def some_other_fn_perhaps_in_another_module():
       hidden_ref_to_locals = locals

in that (if inlining happens) the local printed will be named "b", rather than "a".

But this seems to me like a contrived example: how often in real code do people pass around these builtins, rather than calling them directly?  I myself use them for debugging, but I only ever call them directly.

At a high level, what I'm interested in is providing additional benefits for python 3 relative to python 2 (to encourage migration), and I picked speed-ups.

I think it's reasonable to provide some kind of "optimized" mode that is allowed to take (limited) liberties with the language.

Note that with other languages, people seem to accept some mild degradation of debuggability in return for compiler optimizations.  I'd like to be able to provide a faster python under similar kinds of tradeoffs.

Currently, Python's optimized mode doesn't seem to do much (just compile away assertions iirc).  Perhaps these optimizations could be associated with the pre-existing optimizer flag; alternatively, perhaps a 3rd level could be provided?

This seems like a PEP-level request, but here's an outline of my thoughts here:

I'd like the highly-optimized mode to be permitted the following::
  - to create additional synthesized globals, with a "__" prefix; modification of these globals to lead to undefined behavior.
  - when querying locals under some circumstances (see below) for locals to not be present, or for additional locals to be present::
     - when using "inspect" and reading/modifying a Frame's f_locals
     - in "non-direct" invocations of "locals", "globals" and "dir" (however, when called directly by name, they must work)
     - as per globals, synthesized locals to have a prefix, and modification to lead to undefined behavior
  - a pony :)
  - more seriously, I'm planning to look next at inlining method calls (within one module), and type inference, and I may come up with additional requirements.

I believe that the above covers all of the places where my patch is taking liberties with existing Python semantics (modulo bugs): I'm still supporting runtime rebinding of other names, and (I believe), preserving existing behavior.  (please let me know if I'm missing something here!)

There seems to be some precedent for this: states:
> Note Because dir() is supplied primarily as a convenience for use at an
> interactive prompt, it tries to supply an interesting set of names more
> than it tries to supply a rigorously or consistently defined set of names,
> and its detailed behavior may change across releases. For example,
> metaclass attributes are not in the result list when the argument is a class.

and for locals() it warns:
> Note The contents of this dictionary should not be modified; changes may not
> affect the values of local and free variables used by the interpreter.

Does this sound sane? (obviously, actually approving this would be a matter for the BDFL).

Perhaps all synthesized vars could have a "__internal__" prefix to signify that they should be left alone.

The other PEP-level request would be for the highly-optimized mode to be permitted to take human-noticable time to generate its bytecode (rather than being "instantaneous").  I don't know if that's needed yet: the inliner is currently just a prototype, and I don't have any heuristics yet for deciding whether to inline a callsite (beyond an arbitrary cutoff I put in to stop bm_simple_call from "exploding"); I'm probably introducing various O(N^2) time/space behaviors (and worse) right now.
Date User Action Args
2010-11-20 21:11:21dmalcolmsetrecipients: + dmalcolm, jhylton, nnorwitz, brett.cannon, rhettinger, belopolsky, sdahlbac, orsenthil, titanstar, pitrou, thomaslee, benjamin.peterson, alex
2010-11-20 21:11:20dmalcolmsetmessageid: <>
2010-11-20 21:11:18dmalcolmlinkissue10399 messages
2010-11-20 21:11:18dmalcolmcreate