This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Build-out an AST optimizer, moving some functionality out of the peephole optimizer
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: 11682 Superseder:
Assigned To: Nosy List: Jeremy.Hylton, Trundle, benjamin.peterson, berker.peksag, daniel.urban, dmalcolm, eltoder, eric.snow, georg.brandl, isoschiz, jcon, mark.dickinson, mbdevpl, meador.inge, methane, nadeem.vawda, ncoghlan, pconnell, pitrou, pstch, rhettinger, santoso.wijaya, serhiy.storchaka, techtonik, terry.reedy, vstinner
Priority: normal Keywords: patch

Created on 2011-03-15 04:46 by eltoder, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
0_ast.patch eltoder, 2011-03-15 04:47
0_fold.patch eltoder, 2011-03-15 04:47
0_compile.patch eltoder, 2011-03-15 04:47
0_generated.patch eltoder, 2011-03-15 04:48
0_tests.patch eltoder, 2011-03-15 04:48
issue11549.patch serhiy.storchaka, 2016-05-11 08:22 Regenerated for review review
change-ast.patch methane, 2017-01-31 17:04 review
ast-docstring.patch methane, 2017-02-06 11:40 review
Messages (81)
msg130955 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-15 04:46
As pointed out by Raymond, constant folding should be done on AST rather than on generated bytecode. Here's a patch to do that. It's rather long, so overview first.

The patch replaces existing peephole pass with a folding pass on AST and a few changes in compiler. Feature-wise it should be on par with old peepholer, applying optimizations more consistently and thoroughly, but maybe missing some others. It passes all peepholer tests (though they seem not very extensive) and 'make test', but more testing is, of course, needed.

I've split it in 5 pieces for easier reviewing, but these are not 5 separate patches, they should all be applied together. I can upload it somewhere for review or split it in other ways, let me know. Also, patches are versus 1e00b161f5f5, I will redo against head.

TOC:
1. Changes to AST
2. Folding pass
3. Changes to compiler
4. Generated files (since they're checked in)
5. Tests

In detail:

1. I needed to make some changes to AST to enable constant folding. These are
a) Merge Num, Str, Bytes and Ellipsis constructors into a single Lit (literal) that can hold anything. For one thing, this removes a good deal of copy-paste in the code, since these are always treated the same. (There were a couple of places where Bytes ctor was missing for no apparent reason, I think it was forgotten.) Otherwise, I would have to introduce at least 4 more node types: None, Bool, TupleConst, SetConst. This seemed excessive.
b) Docstring is now an attribute of Module, FunctionDef and ClassDef, rather than a first statement. Docstring is a special syntactic construction, it's not an executable code, so it makes sense to separate it. Otherwise, optimizer would have to take extra care not to introduce, change or remove docstring. For example:

def foo():
  "doc" + "string"

Without optimizations foo doesn't have a docstring. After folding, however, the first statement in foo is a string literal. This means that docstring depends on the level of optimizations. Making it an attribute avoids the problem.
c) 'None', 'True' and 'False' are parsed as literals instead of names, even without optimizations. Since they are not redefineable, I think it makes most sense to treat them as literals. This isn't strictly needed for folding, and I haven't removed all the artefacts, in case this turns out controversial.

2. Constant folding (and a couple of other tweaks) is performed by a visitor. The visitor is auto-generated from ASDL and a C template. C template (Python/ast_opt.ct) provides code for optimizations and rules on how to call it. Parser/asdl_ct.py takes this and ASDL and generates a visitor, that visits only nodes which have associated rules (but visits them via all paths).
The code for optimizations itself is pretty straight-forward.
The generator can probably be used for symtable.c too, removing ~200 tedious lines of code. 

3. Changes to compiler are in 3 categories
a) Updates for AST changes.
b) Changes to generate better code and not need any optimizations. This includes tuple unpacking optimization and if/while conditions.
c) Simple peephole pass on compiler internal structures. This is a better form for doing this, than a bytecode. The pass only deals with jumps to jumps/returns and trivial dead code.
I've also made 'raise' recognized as a terminator, so that 'return None' is not inserted after it.

4, 5. No big surprises here.
msg131070 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2011-03-15 23:50
I'm confused.  Why aren't there review links?
msg131071 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-15 23:51
Because I don't know how to make them. Any pointers?
msg131072 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-15 23:55
> Because I don't know how to make them. Any pointers?

Martin is hacking on the tool these days... So it's no surprise it
doesn't work perfectly yet ;)
If you have a Google account you can upload these patches to
http://codereview.appspot.com/, though.
msg131074 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-16 00:16
Thanks. Review link: http://codereview.appspot.com/4281051
msg131083 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-16 02:52
The review links didn't come up automatically because 336137a359ae isn't a hg.python.org/cpython revision ID.
msg131084 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-16 02:55
I see. Should I attach diffs vs. some revision from hg.python.org?
msg131085 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-16 03:07
No need, since you manually created a review on appspot. The local Reitveld links are just a convenience that can avoid the need to manually create a review instance.
msg131166 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-16 20:13
Any comments on the code so far or suggestions on how we should move forward?
msg131172 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-16 20:34
I've been focusing on softer targets during the sprints - I'll dig into this once I'm back home and using my desktop machine again.
msg131210 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-17 02:02
I've updated patches on Rietveld with some small changes. This includes better code generation for boolops used outside of conditions and cleaned up optimize_jumps(). This is probably the last change before I get some feedback.
Also, I forgot to mention yesterday, patches on Rietveld are vs. ab45c4d0b6ef
msg131214 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-17 04:12
Just for fun I've run pystones. W/o my changes it averages to about 70k, with my changes to about 72.5k.
msg131392 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-03-19 05:27
A couple of somewhat related issues:
#10399 AST Optimization: inlining of function calls
#1346238 A constant folding optimization pass for the AST
Obviously, ast optimizers should work together and not duplicate.
Nice to see increased attention.
msg131396 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-19 06:02
AFAICT my patch has everything that #1346238 has, except BoolOps, which can be easily added (there's a TODO). I don't want to add any new code, though, until the current patch will get reviewed -- adding code will only make reviewing harder.

#10399 looks interesting, I will take a look.
msg131952 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-24 02:40
Is anyone looking or planing to look at the patch?
msg131953 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-03-24 02:45
I suspect someone will sometime. There is bit of a backlog of older issues.
msg132310 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-27 11:15
Finally got around to reviewing this (just a visual scan at this stage) - thanks for the effort. These are mostly "big picture" type comments, so I'm keeping them here rather than burying them amongst all the details in the code review tool.

The effect that collapsing Num/Str/Bytes into a single Lit node type has on ast.literal_eval bothered me initially, but looking more closely, I think those changes will actually improve the function (string concatenation will now work, and errors like "'hello' - 'world'" should give a more informative TypeError). (Bikeshed: We use Attribute rather than Attr for that node type, perhaps the full "Literal" name would be better, too)

Lib/test/disutil.py should really be made a feature of the dis module itself, by creating an inner disassembly function that returns a string, then making the existing "dis" and "disassembly" functions print that string (i.e. similar to what I already did in making dis.show_code() a thin wrapper around the new dis.code_info() function in 3.2). In the absence of a better name, "dis_to_str" would do.

Since the disassembly is interpreter specific, the new disassembly tests really shouldn't go directly in test_compile.py. A separate "test_ast_optimiser" file would be easier for alternate implementations to skip over. A less fragile testing strategy may also be to use the ast.PyCF_ONLY_AST flag and check the generated AST rather than the generated bytecode.

I'd like to see a written explanation for the first few changes in test_peepholer.py. Are those cases no longer optimised? Are they optimised differently? Why did these test cases have to change? (The later changes in that file look OK, since they seem to just be updating tests to handle the more comprehensive optimisation)

When you get around to rebasing the patch on 3.3 trunk, don't forget to drop any unneeded "from __future__" imports.

The generated code for the Lit node type looks wrong: it sets v to Py_None, then immediately checks to see if v is NULL again.

Don't use "string" as a C type - use "char *" (and "char **" instead of "string *").

There should be a new compiler flag to skip the AST optimisation step.

A bunch of the compiler internal changes seem to make the basic flow of the generated assembly not match the incoming source code. It doesn't seem like a good idea to conflate these with the AST optimisation patch. If that means leaving the peepholer in to handle them for now, that's OK - it's fine to just descope the peepholer without removing it entirely.
msg132312 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-27 11:39
I think the biggest thing to take out of my review is that I strongly encourage deferring the changes for 5(b) and 5(c).

I like the basic idea of using a template-based approach to try to get rid of a lot of the boilerplate code currently needed for AST visitors.

Providing a hook for optimisation in Python (as Dave Malcolm's approach does) is valuable as well, but I don't think the two ideas need to be mutually exclusive.

As a more general policy question... where do we stand in regards to backwards compatibility of the AST? The ast module docs don't have any caveats to say that it may change between versions, but it obviously *can* change due to new language constructs (if nothing else).
msg132313 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-27 11:44
> As a more general policy question... where do we stand in regards to
> backwards compatibility of the AST? The ast module docs don't have any
> caveats to say that it may change between versions, but it obviously
> *can* change due to new language constructs (if nothing else).

Yes, but can existing constructs produce a different structure from one
Python version to another?
It seems to me that the ast module is the recommended way to inspect the
structure of Python code (much more so that bytecode or concrete syntax
trees). Perhaps the optimizations can leave the initial ast intact?
Perhaps with an additional API to get the optimized ast as well?
msg132314 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-03-27 11:50
I would provide this via another compile flag a la PyCF_ONLY_AST.  If you give only this flag, you get the original AST.  If you give (e.g.)
PyCF_OPTIMIZED_AST, you get the resulting AST after the optimization stage (or the same, if optimization has been disabled).
msg132346 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-27 16:15
Thanks.

> string concatenation will now work, and errors like "'hello' - 'world'"
> should give a more informative TypeError
Yes, 'x'*5 works too.

> Bikeshed: We use Attribute rather than Attr for that node type,
> perhaps the full "Literal" name would be better
Lit seemed more in line with Num, Str, BinOp etc. No reason it can't be changed, of course.

> Lib/test/disutil.py should really be made a feature of the dis module
> itself
Agreed, but I didn't want to widen the scope of the patch. If this is something that can be reviewed quickly, I can make a change to dis. I'd add a keyword-only arg to dis and disassembly -- an output stream defaulting to stdout. dis_to_str then passes StringIO and returns the string. Sounds OK?

> Since the disassembly is interpreter specific, the new disassembly
> tests really shouldn't go directly in test_compile.py. A separate
> "test_ast_optimiser" file would be easier for alternate
> implementations to skip over.
New tests in test_compiler are not for the AST pass, but for changes to compile.c. I can split them out, how about test_compiler_opts?

> I'd like to see a written explanation for the first few changes in
> test_peepholer.py
Sure.
1) not x == 2 can be theoretically optimized to x != 2, while this test is for another optimization. not x is more robust.
2) Expression statement which is just a literal doesn't produce any code at all. This is now true for None/True/False as well. To preserve constants in the output I've put them in tuples.

> When you get around to rebasing the patch on 3.3 trunk, don't forget
> to drop any unneeded "from __future__" imports.
If you're referring to asdl_ct.py, that's actually an interesting question. asdl_ct.py is run by system installed python2, for obvious reasons. What is the policy here -- what is the minimum version of system python that should be sufficient to build python3? I tested my code on 2.6 and 3.1, and with __future__ it should work on 2.5 as well. Is this OK or should I drop 'with' so it runs on 2.4?

> The generated code for the Lit node type looks wrong: it sets v to
> Py_None, then immediately checks to see if v is NULL again.
Right, comment in asdl_c.py says:
# XXX: special hack for Lit. Lit value can be None and it
#  should be stored as Py_None, not as NULL.
If there's a general agreement on Lit I can certainly clean this up.

> Don't use "string" as a C type - use "char *" (and "char **" instead
> of "string *").
string is a typedef for PyObject that ASDL uses. I don't think I have a choice to not use it. Can you point to a specific place where char* could be used?

> There should be a new compiler flag to skip the AST optimisation step.
There's already an 'optimizations level' flag. Maybe we should make it more meaningful rather than multiplying the number of flags?

> A bunch of the compiler internal changes seem to make the basic flow
> of the generated assembly not match the incoming source code.
Can you give an example of what you mean?
The changes are basically 1) standard way of handling conditions in simple compilers 2) peephole.

> It doesn't seem like a good idea to conflate these with the AST
> optimisation patch. If that means leaving the peepholer in to handle
> them for now, that's OK - it's fine to just descope the peepholer
> without removing it entirely.
The reason why I think it makes sense to have this in a single change is testing. This allows to reuse all existing peephole tests. If I leave old peephole enabled there's no way to tell if my pass did something from disassembly. I can port tests to AST, but that seemed like more work than match old peepholer optimizations.
Is there any opposition to doing simple optimizations on compiler structures? They seem a good fit for the job. In fact, if not for stack representation, I'd say that they are better IR for optimizer than AST.

Also, can I get your opinion on making None/True/False into literals early on and getting rid of forbidden_name?

Antoine, Georg -- I think Nick's question is not about AST changing after optimizations (this can indeed be a separate flag), but the structure of AST changing. E.g. collapsing of Num/Str/Bytes into Lit.

Btw, if this is acceptable I'd make a couple more changes to make scope structure obvious from AST. This will allow auto-generating much of the symtable pass.
msg132348 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-27 16:25
> and with __future__ it should work on 2.5 as well.
Actually, seems that at least str.format is not in 2.5 as well. Still the question is should I make it run on 2.5 or 2.4 or is 2.6 OK (then __future__ can be removed).
msg132349 - (view) Author: Daniel Urban (daniel.urban) * (Python triager) Date: 2011-03-27 16:26
> not x == 2 can be theoretically optimized to x != 2, ...

I don't think it can:

>>> class X:
...     def __eq__(self, other):
...             return True
...     def __ne__(self, other):
...             return True
... 
>>> x = X()
>>> 
>>> not x == 2
False
>>> x != 2
True
>>>
msg132350 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-27 16:30
> I don't think it can:
That already doesn't work in dict and set (eq not consistent with hash), I don't think it's a big problem if that stops working in some other cases. Anyway, I said "theoretically" -- maybe after some conservative type inference.
msg132361 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-27 19:52
Also, to avoid any confusion -- currently my patch only runs AST optimizations before code generation, so compile() with ast.PyCF_ONLY_AST returns non-optimized AST.
msg132362 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-03-27 20:18
While I would not be happy to use class X above, the 3.2 manual explicitly says "There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. " .
msg132382 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-28 03:52
OK, I missed the fact that the new optimisation pass isn't run under PyCF_ONLY_AST.

However, as Eugene picked up, my concern is actually with the collapsing of Str/Num/Bytes/Ellipsis into the new Lit node, and the changes to the way None/True/False are handled. They're all changes that *make sense*, but would also likely cause a variety of AST manipulations to break. We definitely don't care when bytecode hacks break, but providing the ast module means that AST manipulation is officially supported.

However, the reason I bring up new constructs, is the fact that new constructs may break AST manipulation passes, even if the old structures are left intact - the AST visitor may miss (or misinterpret) things because it doesn't understand the meaning of the new nodes.

We may need to take this one back to python-dev (and get input from the other implementations as well). It's a fairly fundamental question when it comes to the structure of any changes.
msg132453 - (view) Author: Eugene Toder (eltoder) * Date: 2011-03-29 01:31
If we have to preserve backward compatibility of Python AST API, we can do this relatively easily (at the expense of some code complexity):
* Add 'version' argument to compile() and ast.parse() with default value of 1 (old AST). Value 2 will correspond to the new AST.
* Do not remove Num/Str/Bytes/Ellipsis Python classes. Make PyAST_obj2mod and PyAST_mod2obj do appropriate conversions when version is 1.
* Possibly emit a PendingDeprecationWarning when version 1 is used with the goal of removing it in 3.5

Alternative implementation is to leave Num/Str/etc classes in C as well, and write visitors (similar to folding one) to convert AST between old and new forms.

Does this sounds reasonable? Should this be posted to python-dev? Should I write a PEP (I'd need some help with this)?

Are there any other big issues preventing this to be merged?
msg132455 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-03-29 01:52
Eugene, I think you're doing great work here and would like to see you succeed.  In the near term, I don't have time to participate, but don't let that stop you.
msg132473 - (view) Author: anatoly techtonik (techtonik) Date: 2011-03-29 08:56
Is there any tool to see how it works step-by-step. The whole stuff is extremely interesting, but I can't fit all the details of AST processing in my head.
msg132476 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-03-29 10:47
Eugene: I suggest raising the question on python-dev. The answer potentially affects the PEP 380 patch as well (which adds a new attribute to the "Yield" node).

Anatoly: If you just want to get a feel for the kind of AST various pieces of code produce, then the "ast.dump" function (along with using the ast.PyCF_ONLY_AST flag in compile) may be enough.

You may also want to take a look at the AST -> dot file conversion code in Dave Malcolm's patches on #10399.
msg137785 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-06-07 03:36
Eugene raised the question of AST changes on python-dev [1] and the verdict was that so long as ast.__version__ is updated, AST clients will be able to cope with changes.

Benjamin Peterson made some subsequent changes to the AST (bringing the AST for try and with statements more in line with the concrete syntax, allowing source-to-source transforms to retain the original code structure).

This patch will probably need to be updated to be based on the latest version of the AST - I would be surprised if it applied cleanly to the current tip.

[1] http://mail.python.org/pipermail/python-dev/2011-April/110399.html
msg137788 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-06-07 05:53
Updated the title to reflect that the peephole optimizer will likely continue to exist but in a much simpler form.  Some complex peephole optimization such as constant folding can be handled more easily and more robustly at the AST level.

Other minor peephole optimizations such as jump-to-jump simplification as still bytecode level optimizations (ones that improve the quality of the generated code without visibility to higher level semantics).
msg137819 - (view) Author: Eugene Toder (eltoder) * Date: 2011-06-07 12:17
Nick, if there's an interest in reviewing the patch I can update the it. I doubt it needs a lot of changes, given that visitor is auto-generated.

Raymond, the patch contains a rewrite of low-level optimizations to work before byte code generation, which simplifies them a great deal.
msg137906 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-06-08 13:47
As Raymond noted though, some of the block stack fiddling doesn't make sense until after the bytecode has already been generated. It's OK to have multiple optimisers at different layers, each taking care of the elements that are best suited to that level.

And yes, an updated patch against the current tip would be good. Of my earlier review comments, the ones I'd still like to see addressed are:
- finish clearing out the now redundant special casing of None/True/False
- separating out the non-AST related compiler tweaks (i.e. 3b and 3c and the associated test changes) into their own patch (including moving the relevant tests into a separate @cpython_only test case)

I'm still not 100% convinced on that latter set of changes, but I don't 
want my further pondering on those to hold up the rest of the patch. (they probably make sense, it's just that the AST level changes are much easier to review than the ones right down at the bytecode generation level - reviewing the latter means getting back up to speed on precisely how the block stack works and it will be a while before I get around to doing that. It's just one of those things where the details matter, but diving that deep into the compiler is such a rare occurrence that I have to give myself a refresher course each time it happens).
msg138413 - (view) Author: Eugene Toder (eltoder) * Date: 2011-06-16 03:26
I found a problem in constant de-duplication, already performed by compiler, that needs to be fixed before this change can be merged. 

Compiler tries to eliminate duplicate constants by putting them into a dict. However, "duplicate" in this case doesn't mean just "equal", we need a stronger relationship, as there are many equal values that behave differently in some contexts, e.g. 0 and 0.0 and False or 0.0 and -0.0. To this end for each value we create a tuple of the value and it's type and have some logic for -0.0. This is handled in compiler_add_o in Python/compile.c.

This logic, however, only works for scalar values -- if we get a container with 0 and the same container with False we collapse them into one. This was not a problem before, because constant tuples were only created by peephole, which doesn't attempt de-duplication. If tuple folding is moved to AST we start hitting this problem:

>>> dis(lambda: print((0,1),(False,True)))
  1           0 LOAD_GLOBAL              0 (print)
              3 LOAD_CONST               1 ((0, 1))
              6 LOAD_CONST               1 ((0, 1))
              9 CALL_FUNCTION            2
             12 RETURN_VALUE

The cleanest solution seems to be to introduce a new rich comparison code: Py_EQUIV (equivalent) and implement it at least in types that we support in marshal. This will simplify compiler_add_o quite a bit and make it work for tuples and frozensets.
I'm open to other suggestions.
msg139395 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-06-29 06:21
Marking the PEP 380 implementation as a dependency, as I expect it to be easier to update this patch to cope with those changes than it would be the other way around.
msg160608 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-05-14 11:50
Bumping the target version to 3.4.

This is still a good long term idea, but it's a substantial enough change that we really want to land it early in a development cycle so we have plenty of time to hammer out any issues.
msg160662 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-05-14 20:28
Good call, Nick.
msg168146 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-08-13 21:29
In msg132312 Nick asked "where do we stand in regards to backwards compatibility of the AST?"

The current ast module chapter, second sentence, says ""The abstract syntax itself might change with each Python release;" this module helps to find out programmatically what the current grammar looks like."
where 'current grammar' is copied in 30.2.2. Abstract Grammar.

I do not know when that was written, but it clearly implies the the grammark, which defines node classes, is x.y version specific. I think this is the correct policy just so we can make changes, hopefully improvements, such as the one proposed here.
msg169894 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-09-05 23:48
I'm working on a AST optimizer for Python 2.6-3.3:
https://bitbucket.org/haypo/astoptimizer

It is implemented in Python and is able to optimize much more cases than the current bytecode peepholer.
msg169895 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-06 00:11
All of the optimisations that assume globals haven't been shadowed or rebound are invalid in the general case.

E.g. print(1.5) and print("1.5") are valid for *our* print function, but we technically have no idea if they're equivalent in user code.

In short, if it involves a name lookup and that name isn't reserved to the compiler (e.g. __debug__) then no, you're not allowed to optimise it at compile time if you wish to remain compliant with the language spec. Method calls on literals are always fair game, though (e.g. you could optimise "a b c".split())

Any stdlib AST optimiser would need to be substantially more conservative by default.
msg169896 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-09-06 00:25
> All of the optimisations that assume globals haven't been shadowed
> or rebound are invalid in the general case.

My main idea is that the developer of the application should be able to annotate functions and constants to declare them as "optimizable" (constant). I chose to expect builtins as not being overrided, but if it breaks applications, it can be converted to an option disabled by default.

There is a known issue: test_math fails because pow() is an alias to matH.pow() in doctests. The problem is that "from math import *" is called and the result is stored in a namespace, and then "pow(2,4)" is called in the namespace. astoptimizer doesn't detect that pow=math.pow because locals are only set when the code is executed (and not at compilation) with something like: exec(code, namespace). It is a limitation of the optimizer. A workaround is to disable optimizations when running tests.

It is possible to detect that builtins are shadowed (ex: print=myprint). astoptimizer has an experimental support of assignments, but it only works on trivial examples yet (like "x=1; print(x)") and is disabled by default (because it is buggy). I also plan to disable some optimizations if globals(), vars() or dir() is called.
msg169897 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-09-06 00:40
> Any stdlib AST optimiser would need to be substantially more conservative by default.

FYI The test suite of Python 2.7 and 3.3 pass with astoptimizer... except some "minor" (?) failures:

 * test_math fails for the reason explained above
 * test_pdb: it looks to be an issue with line number (debuggers don't like optimizers :-))
 * test_xml_etree and test_xml_etree_c: reference count of the None singleton

The test suite helped me to find bugs in my optimizer :-)

I also had to add some hacks (hasattr) for test_ast (test_ast generates invalid AST trees). The configuration should also be adapted for test_peepholer, because CPython peepholer uses a limit of 20 items, whereas astoptimizer uses a limit of 4096 bytes/characters for string by default. All these minor nits are now handled in a specific "cpython_tests" config.
msg169898 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-06 00:48
No, you're assuming global program analysis and *that is not a valid assumption*.

One of the key features of Python is that *monkeypatching works*. It's not encouraged, but it works.

You simply cannot play games with name lookups like this without creating something that is no longer Python.
msg169899 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-06 00:50
You also have to be very careful of the interface to tracing functions, such as profilers and coverage analysis tools.
msg169900 - (view) Author: Eugene Toder (eltoder) * Date: 2012-09-06 01:20
> Method calls on literals are always fair game, though (e.g. you could optimise "a b c".split())
What about optimizations that do not change behavior, except for different error messages? E.g. we can change
y = [1,2][x]
to
y = (1,2)[x]
where the tuple is constant and is stored in co_consts. This will, however, produce a different text in the exception when x is not 0 or 1. The type of exception is going to be the same.
msg169902 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-06 01:45
The peephole optimiser already makes optimisations like that in a couple of places (e.g. set -> frozenset):

>>> def f(x):
...    if x in {1, 2}: pass
...
>>> f.__code__.co_consts
(None, 1, 2, frozenset({1, 2}))

It's name lookup semantics that are the real minefield. It's one of the reasons PyPy's JIT can be so much more effective than a static optimiser - because it's monitoring real execution and inserting the appropriate guards it's not relying on invalid assumptions about name bindings.
msg169903 - (view) Author: Eugene Toder (eltoder) * Date: 2012-09-06 03:42
If I'm not missing something, changing
x in [1,2]
to
x in (1,2)
and
x in {1,2}
to
x in frozenset([1,2])
does not change any error messages.

Agreed that without dynamic compilation we can pretty much only track literals (including functions and lambdas) assigned to local variables.
msg185286 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-26 18:05
http://bugs.python.org/issue17515 might also play into this if it happens to go in.
msg193656 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-07-24 15:21
Just noting for the record (since it appears it was never brought back to the comments): it is expected that programs that manipulate the AST may require updates before they will work on a new version of Python. Preserving AST backwards compatbility is too limiting to the evolution of the language, so only source compatibility is promised.
msg193657 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-07-24 15:22
(That was the outcome of the suggested AST discussions on python-dev that were mentioned earlier)
msg265296 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-11 08:22
Regenerated for review.
msg265553 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-05-14 23:00
"issue11549.patch: serhiy.storchaka, 2016-05-11 08:22: Regenerated for review"

diff -r 1e00b161f5f5 PC/os2emx/python33.def
--- a/PC/os2emx/python33.def	Wed Mar 09 12:53:30 2011 +0100
+++ b/PC/os2emx/python33.def	Wed May 11 11:21:24 2016 +0300

The revision 1e00b161f5f5 is 4 years old. The patch looks very outdated :-/
msg265554 - (view) Author: Eugene Toder (eltoder) * Date: 2016-05-14 23:02
Fairly sure it's 5 years old.
msg265607 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-15 09:38
Yes, the patch is outdated, conflicts with current code (and would conflict even more after pushing the wordcode patch) and contains bugs. But it moved in right direction, I think your _PyCode_ConstantKey() could help to fix bugs. I'm going to revive this issue.
msg265613 - (view) Author: Eugene Toder (eltoder) * Date: 2016-05-15 13:17
Serhiy: Nice! Yes, _PyCode_ConstantKey solved the problem.

But #16619 went in the opposite direction of this patch, and introduced a new type of literal node instead of unifying the existing ones. Kind of a shame, since *this* patch, I believe, both fixes that bug and removes the unreachable code in the example :)

I also see that Victor has been doing some of the same work, e.g. #26146.
msg265621 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-05-15 15:35
> I also see that Victor has been doing some of the same work, e.g. #26146.

ast.Constant idea directly comes from your work. The implementatiln may ve
different.

It's a first step for AST optimizers.
msg286503 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-01-31 01:51
@haypo, how do you think about ast.Lit and ast.Constant?
Is this patch updated to use ast.Constant?
Or ast.Constant should be used only for some transform like constant folding?
msg286528 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-31 12:53
> @haypo, how do you think about ast.Lit and ast.Constant?

I already made two changes in Python 3.6 :-)

I added ast.Constant to Python 3.6. While it's not used by .py=>AST compiler, it is understood by the AST->bytecode compiler (ex: handle correctly special cases like docstrings).
https://docs.python.org/dev/whatsnew/3.6.html#ast
=> issue #26146

Moreover, I also modified the format of the co_lnotab structure to support negative line number delta. So a code transformer can move instructions and preserve the Python traceback feature.
https://docs.python.org/dev/whatsnew/3.6.html#changes-in-the-python-api
=> see the PEP 511
msg286530 - (view) Author: Hugo Geoffroy (pstch) * Date: 2017-01-31 13:33
I would like to point out that the changes in `ast.literal_eval` may have some security risk for code that do not expect this function to return an object with user-controlled length (for example, with `2**32*'X'`). AFAIK, this is not possible with the current version of `literal_eval`.

At least [this library](https://pypi.python.org/pypi/serpent) would have a serious risk of remote DoS :

> Because it only serializes literals and recreates the objects using ast.literal_eval(), the serialized data is safe to transport to other machines (over the network for instance) and de-serialize it there.

Sorry for the noise if this is a useless/incorrect consideration.
msg286531 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-31 14:05
Hugo Geoffroy added the comment:
> I would like to point out that the changes in `ast.literal_eval` may have some security risk for code that do not expect this function to return an object with user-controlled length (for example, with `2**32*'X'`). AFAIK, this is not possible with the current version of `literal_eval`.

Since the Python compiler doesn't produce ast.Constant, there is no
change in practice in ast.literal_eval(). If you found a bug, please
open a new issue.

> At least [this library](https://pypi.python.org/pypi/serpent) would have a serious risk of remote DoS :

I tried hard to implement a sandbox in Python and I failed:
https://lwn.net/Articles/574215/

I don't think that literal_eval() is safe *by design*.
msg286532 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-31 14:19
Good point Hugo. Yes, this should be taken into account when move constant folding to AST level. Thank you for the reminder.
msg286534 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-31 14:32
> Since the Python compiler doesn't produce ast.Constant, there is no
change in practice in ast.literal_eval(). If you found a bug, please
open a new issue.

Currently there is no a bug in ast.literal_eval() because the '**' operator is not accepted.

>>> ast.literal_eval("2**2**32")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/ast.py", line 85, in literal_eval
    return _convert(node_or_string)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 84, in _convert
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.BinOp object at 0xb6f2fa4c>

But if move the optimization to AST level this can add a vulnerability to DOS attack. The optimizer should do additional checks first than execute operators that can return too large value or take too much CPU time. Currently this vulnerability have place in the peephole optimizer.
msg286535 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-31 14:35
> Currently there is no a bug in ast.literal_eval() because the '**' operator is not accepted.

The doc says "This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing."
https://docs.python.org/dev/library/ast.html#ast.literal_eval

I don't think that it's a bug, but a deliberate design choice. a**b is an obvious trick to DoS a server (high CPU and memory usage).
msg286539 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-01-31 15:59
Hugo, Serhiy, and Victor: I think you're all agreeing with each other, but to make sure I'm understanding the point correctly:

1. ast.literal_eval() is currently safe from malicious code like "100000 ** 100000" or "1073741824 * 'a'" because it only traverses addition and subtraction nodes, so any such operations will just throw ValueError (As a point of interest: unary plus and minus are required to support positive and negative numeric literals, while binary addition and subtraction are required to support complex number literals. So the status quo isn't precisely the result of a conscious security decision, it's just a minimalist implementation of exactly what's necessary to support all of the builtin types, which also provides some highly desirable security properties when evaluating untrusted code)


2. an eager constant folding optimisation in the AST tier would happen *before* literal_eval filtered out the multiplication and exponentiation nodes, and hence would make literal_eval vulnerable to remote DOS attacks in cases where it is expected to be safe

However, that's not exactly how this patch works: if you pass "PyCF_ONLY_AST" as ast.parse does, it *doesn't* run the constant-folding step. Instead, the constant folding is run as an AST-to-AST transform during the AST-to-bytecode compilation step, *not* the initial source-to-AST step. (see http://bugs.python.org/issue11549#msg132361 )

This has a few nice properties:

- ast.literal_eval() remains safe
- source -> AST -> source transformation pipelines continue to preserve the original code structure
- externally generated AST structures still benefit from the AST optimisation pass
- we don't need a new flag to turn this optimisation pass off when generating the AST for a given piece of source code
msg286542 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-01-31 17:04
> 1. Changes to AST

I'm working on updating this part. There are some failing tests remains. 
But I doubt this stage is worth enough for now.

> a) Merge Num, Str, Bytes and Ellipsis constructors into a single Lit  (literal) that can hold anything. For one thing, this removes a good deal of copy-paste in the code, since these are always treated the same. (There were a couple of places where Bytes ctor was missing for no apparent reason, I think it was forgotten.) Otherwise, I would have to introduce at least 4 more node types: None, Bool, TupleConst, SetConst. This seemed excessive.

We have already Constant and NameConstant.  So it seems there are no need for
None, Bool, TupleConst, SetConst nodes.

I think converting Num, Str, Bytes, Ellipsis into Constant in folding stage
is easier than fixing all tests.


> b) Docstring is now an attribute of Module, FunctionDef and ClassDef, rather than a first statement. Docstring is a special syntactic construction, it's not an executable code, so it makes sense to separate it. Otherwise, optimizer would have to take extra care not to introduce, change or remove docstring.

Take docstring before constant folding isn't enough?
(I'm sorry if I'm wrong.  I haven't tried it yet.)


> c) 'None', 'True' and 'False' are parsed as literals instead of names, even without optimizations. Since they are not redefineable, I think it makes most sense to treat them as literals. This isn't strictly needed for folding, and I haven't removed all the artefacts, in case this turns out controversial.

They are all NameConstant already.
msg286588 - (view) Author: Eugene Toder (eltoder) * Date: 2017-02-01 05:08
> We have already Constant and NameConstant.  So it seems there are no need for
> None, Bool, TupleConst, SetConst nodes.
Yes, Constant is Victor's version of Lit.

> I think converting Num, Str, Bytes, Ellipsis into Constant in folding stage
> is easier than fixing all tests.
Fixing tests was fairly easy the last time. I think the question is what changes to the public API of AST are acceptable.

> Take docstring before constant folding isn't enough?
> (I'm sorry if I'm wrong.  I haven't tried it yet.)
It may be doable, but seems very messy. Instead of a clean pipeline text -> AST -> Optimized AST -> bytecode, you have to collect all docstrings, and pass them around in a side structure.

With the current code there can be a simple fix. If original string literals are Str, but constant-folded string constants are Constant, only treat Strs as docstrings. Then the optimizer needs a change to always preserve Str as a first statement in a function/module, and that's it.
I still think that having a dedicated docstring attribute in AST is cleaner, though.

> They are all NameConstant already.
Keep in mind this patch is 6 years old :)
msg286589 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-02-01 06:29
>> We have already Constant and NameConstant.  So it seems there are no need for
>> None, Bool, TupleConst, SetConst nodes.
> Yes, Constant is Victor's version of Lit.

Then, may I remove ast.Lit, and use Constant and NameConstant?


>> I think converting Num, Str, Bytes, Ellipsis into Constant in folding stage
>> is easier than fixing all tests.
> Fixing tests was fairly easy the last time. I think the question is what changes to the public API of AST are acceptable.

I think backward compatibility is not guaranteed.
But there are some usage of ast. (https://github.com/search?l=Python&p=2&q=ast.Num&type=Code&utf8=%E2%9C%93 )

So I think we should make change small as possible.


>> Take docstring before constant folding isn't enough?
>> (I'm sorry if I'm wrong.  I haven't tried it yet.)
> It may be doable, but seems very messy. Instead of a clean pipeline text -> AST -> Optimized AST -> bytecode, you have to collect all docstrings, and pass them around in a side structure.
>
> With the current code there can be a simple fix. If original string literals are Str, but constant-folded string  constants are Constant, only treat Strs as docstrings. Then the optimizer needs a change to always preserve Str as a first statement in a function/module, and that's it.
> I still think that having a dedicated docstring attribute in AST is cleaner, though.

OK.


>> They are all NameConstant already.
> Keep in mind this patch is 6 years old :)

I know. I want to move this patch forward, but I'm not frontend (parser, AST, and compiler) expert.
I can't make design decision without expert's advice.  Thanks for your reply.

Then, may I update the patch in following  direction?

* Remove ast.Lit.
* Keep docstring change.
msg286591 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-01 07:04
If you would like to implement constant folding at the AST level, I suggest
you to look at my fatoptimizer project:
https://github.com/haypo/fatoptimizer/blob/master/fatoptimizer/const_fold.py

The tricky part is to avoid operations when we know that it will raise an
exception or create an object too big according to our constraints.

I would prefer to implement an AST optimizer in Python, but converting C
structures to Python objects and then back to C structures has a cost. I'm
not sure that my optimizer implemented in Python is fast enough.

By the way, an idea would be to skip all optimizations in some cases like
for script.py when running python3 script.py.
msg286592 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-02-01 07:16
Before trying advanced optimizations, I want move suspended obvious optimizations forwards.

For example, removing unused constants is suspended because constant folding
should be moved from peephole to AST.  This is why I found this issue.

After that, I'm thinking about shrinking stacksize. frame_dealloc (scans whole stack) is one of hot functions.
msg286668 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-02-01 17:49
Dropping ast.Lit is fine. As for the docstring part, I'm torn. Yes it's nice as that will show up semantically in the Python code, but it's also easy to check for by just looking if the first statement is a Str (or Constant if that's what we make all strings). So I'll say I'm +0 on the docstring part.
msg286697 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-01 21:10
At the AST level, you have a wide range of possible optimizations. See the optimizations that I implemented in fatoptimizer (FAT Python) to have an idea:
http://fatoptimizer.readthedocs.io/en/latest/optimizations.html

FAT Python adds guards checked at runtime, something not possible (not wanted) here.

But if you start with constant folding, why not implementing constant propagation as well? What about loop unrolling?

Where is the limit? If you implement the AST optimizer in C, the limit will probably be your C skills and your motivation :-) In Python, the limit is more the Python semantics which is... hum... not well defined. For example, does it break the Python semantics to replace [i for i in (1, 2, 3)] with [1, 2, 3]? 

What if you use a debugger? Do yo expect a list comprehension or a literal list?

FYI I suspended my work on FAT Python because almost no other core developer was interested. I didn't get any support, whereas I need support to push core FAT Python features like function specialization and runtime checks (PEP 510, see also PEP 511). Moreover, I failed to show any significant speedup on non-trivial functions. I abandoned before investigating function inlining, even if FAT Python already has a basic support for function inlining.

This issue is open since 2011. The question is always the same: is it worth it?

An alternative is to experiment an AST optimizer outside CPython and come back later with more data to drive the design of such optimizer. With FAT Python, I chose to add hooks in the Python compiler, but different people told me that it's possible to do that without such hook but importlib (importlib hooks).

What do you think Naoki?
msg286725 - (view) Author: Eugene Toder (eltoder) * Date: 2017-02-02 00:32
Yes, doing optimizations on AST in CPython is unlikely to give any sizable speed improvements in real world programs. Python as a language is not suited for static optimization, and even if you manage to inline a function, there's still CPython's interpreted overhead and boxed types that dwarf the effect of the optimization.

The goal of this patch was never to significantly improve the speed. It was to replace the existing bytecode peephole pass with cleaner and simpler code, which also happens to produce slightly better results.
msg286726 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-02-02 00:56
My motivation is improve speed, reduce memory usage, and quicker
startup time for real world applications.  If some optimization in
FAT optimizer has significant speedup, I want to try it.

But this time, my motivation is I felt "everyone think constant folding
should go to AST from peephole, there is a patch about it, but unfortunately
it was suspended (because of lack of reviewers, maybe)."

As reading #28813, I think there are consensus about constant folding
should go AST.
msg286730 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-02 01:27
INADA Naoki added the comment:
> My motivation is improve speed,

Ah, if the motivation is performance, I would like to see benchmark
results :-) I understand that an AST optimizer would help to produce
more efficient bytecode, right?

> reduce memory usage,

I noticed an issue with the peephole optimizer: the constant folding
step keeps original constants. Moving constant folding to the AST
stage fixes this issue by design.

> and quicker startup time for real world applications.

You mean faster import time on precompiled .pyc files, right? It's
related to the hypothetical faster bytecode.

> If some optimization in FAT optimizer has significant speedup, I want to try it.

See http://fatoptimizer.readthedocs.io/en/latest/microbenchmarks.html#microbench

FYI it took me something like 2 months to build FAT Python
"infrastructure": fix CPython bugs, design guards, design the AST
optimizer, write unit tests, etc. I didn't spend much time on
efficient optimizations. But for my first rule was to not break the
CPython test suite! Not break the Python semantics, otherwise it would
be impossible to put enable the optimizer by default in CPython, which
is my long term goal.
msg287120 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-02-06 11:40
I've tried to update ast_opt.c[t] without changing AST.
But I can't find clear way to solve "foo" + "bar" docstring problem.

This patch adds only docstring to AST.
msg287121 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-06 11:50
Naoki: Can you please open a new issue for your ast-docstring.patch change? I like it, but this issue became too big, and I'm not sure that everyone in the nosy list is interested by this specific change.
msg287187 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-02-07 04:05
I submit new issues:

* #29463 for AST change (change docstring from first statement to attribute).
* #29469 for constant folding

Note that this issue contains more peephole -> AST optimization changes.
But I want to start these two patch to ease review and discussion.
msg287217 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-07 09:09
I created the issue #29471: "AST: add an attribute to FunctionDef to distinguish functions from generators and coroutines".
msg360721 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-01-26 06:07
I think this particular issue can be closed in now.  After it was written, the AST optimizer has been built-out and some peephole logic has moved.

Other adjustments and improvements can be continue to be done in the other open issues.
History
Date User Action Args
2022-04-11 14:57:14adminsetgithub: 55758
2020-01-26 06:07:31rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg360721

stage: resolved
2020-01-24 23:33:32brett.cannonsetnosy: - brett.cannon
2019-03-03 14:33:28serhiy.storchakaunlinkissue21074 superseder
2017-12-15 11:26:53serhiy.storchakaunlinkissue27169 dependencies
2017-10-05 07:48:41serhiy.storchakalinkissue27169 dependencies
2017-03-15 06:45:24mbdevplsetnosy: + mbdevpl
2017-02-07 09:09:18vstinnersetmessages: + msg287217
2017-02-07 04:05:55methanesetmessages: + msg287187
2017-02-06 12:11:03alexsetnosy: - alex
2017-02-06 11:50:42vstinnersetmessages: + msg287121
2017-02-06 11:40:31methanesetfiles: + ast-docstring.patch

messages: + msg287120
2017-02-02 01:35:29gregory.p.smithsetnosy: - gregory.p.smith
2017-02-02 01:27:34vstinnersetmessages: + msg286730
2017-02-02 00:56:04methanesetmessages: + msg286726
2017-02-02 00:32:29eltodersetmessages: + msg286725
2017-02-01 21:10:15vstinnersetmessages: + msg286697
2017-02-01 17:49:44brett.cannonsetmessages: + msg286668
2017-02-01 07:16:25methanesetmessages: + msg286592
2017-02-01 07:04:07vstinnersetmessages: + msg286591
2017-02-01 06:29:15methanesetmessages: + msg286589
2017-02-01 05:08:54eltodersetmessages: + msg286588
2017-01-31 17:04:48methanesetfiles: + change-ast.patch

messages: + msg286542
2017-01-31 15:59:23ncoghlansetmessages: + msg286539
2017-01-31 14:35:03vstinnersetmessages: + msg286535
2017-01-31 14:32:55serhiy.storchakasetmessages: + msg286534
2017-01-31 14:19:58serhiy.storchakasetmessages: + msg286532
2017-01-31 14:05:11vstinnersetmessages: + msg286531
2017-01-31 13:33:13pstchsetnosy: + pstch
messages: + msg286530
2017-01-31 12:53:54vstinnersetmessages: + msg286528
2017-01-31 01:51:50methanesetmessages: + msg286503
versions: + Python 3.7, - Python 3.6
2017-01-27 13:06:11methanesetnosy: + methane
2016-09-29 08:50:26serhiy.storchakalinkissue28308 dependencies
2016-09-29 08:50:19serhiy.storchakalinkissue28307 dependencies
2016-05-15 15:35:37vstinnersetmessages: + msg265621
2016-05-15 13:17:52eltodersetmessages: + msg265613
2016-05-15 09:38:38serhiy.storchakasetmessages: + msg265607
2016-05-14 23:02:51eltodersetmessages: + msg265554
2016-05-14 23:00:48vstinnersetmessages: + msg265553
2016-05-11 08:22:28serhiy.storchakasetfiles: + issue11549.patch
nosy: + serhiy.storchaka
messages: + msg265296

2015-06-02 00:44:25ncoghlansetversions: + Python 3.6, - Python 3.4
2014-03-30 19:47:10berker.peksagsetnosy: + berker.peksag
2014-03-30 06:04:57ncoghlanlinkissue21074 superseder
2013-07-24 15:22:31ncoghlansetmessages: + msg193657
2013-07-24 15:21:52ncoghlansetmessages: + msg193656
2013-04-19 21:24:41isoschizsetnosy: + isoschiz
2013-04-19 19:15:08pconnellsetnosy: + pconnell
2013-03-26 18:05:09brett.cannonsetmessages: + msg185286
2012-09-06 03:42:49eltodersetmessages: + msg169903
2012-09-06 01:45:36ncoghlansetmessages: + msg169902
2012-09-06 01:20:24eltodersetmessages: + msg169900
2012-09-06 00:50:25ncoghlansetmessages: + msg169899
2012-09-06 00:48:56ncoghlansetmessages: + msg169898
2012-09-06 00:40:49vstinnersetmessages: + msg169897
2012-09-06 00:25:24vstinnersetmessages: + msg169896
2012-09-06 00:11:42ncoghlansetmessages: + msg169895
2012-09-05 23:48:50vstinnersetnosy: + vstinner
messages: + msg169894
2012-08-13 21:29:50terry.reedysetmessages: + msg168146
2012-07-25 03:10:25meador.ingesetnosy: + meador.inge
2012-07-23 12:43:45Jeremy.Hyltonsetnosy: + Jeremy.Hylton
2012-07-19 04:55:41gregory.p.smithsetnosy: + gregory.p.smith
2012-05-14 20:28:35rhettingersetmessages: + msg160662
2012-05-14 11:50:14ncoghlansetmessages: + msg160608
versions: + Python 3.4, - Python 3.3
2011-07-09 20:55:28eric.snowsetnosy: + eric.snow
2011-06-29 20:58:43jconsetnosy: + jcon
2011-06-29 06:21:51ncoghlansetdependencies: + PEP 380 reference implementation for 3.3
messages: + msg139395
2011-06-16 03:26:03eltodersetmessages: + msg138413
2011-06-08 13:47:10ncoghlansetmessages: + msg137906
2011-06-07 12:17:43eltodersetmessages: + msg137819
2011-06-07 05:53:33rhettingersetmessages: + msg137788
title: Rewrite peephole to work on AST -> Build-out an AST optimizer, moving some functionality out of the peephole optimizer
2011-06-07 03:36:55ncoghlansetmessages: + msg137785
2011-03-29 10:47:01ncoghlansetmessages: + msg132476
2011-03-29 08:56:37techtoniksetnosy: + techtonik
messages: + msg132473
2011-03-29 01:52:54rhettingersetmessages: + msg132455
2011-03-29 01:31:38eltodersetmessages: + msg132453
2011-03-28 03:52:34ncoghlansetmessages: + msg132382
2011-03-27 20:18:19terry.reedysetmessages: + msg132362
2011-03-27 19:52:53eltodersetmessages: + msg132361
2011-03-27 16:30:11eltodersetmessages: + msg132350
2011-03-27 16:26:11daniel.urbansetmessages: + msg132349
2011-03-27 16:25:01eltodersetmessages: + msg132348
2011-03-27 16:15:43eltodersetmessages: + msg132346
2011-03-27 11:50:50georg.brandlsetmessages: + msg132314
2011-03-27 11:44:43pitrousetmessages: + msg132313
2011-03-27 11:39:44ncoghlansetmessages: + msg132312
2011-03-27 11:15:23ncoghlansetmessages: + msg132310
2011-03-24 02:45:03terry.reedysetmessages: + msg131953
2011-03-24 02:40:21eltodersetmessages: + msg131952
2011-03-19 19:06:56skip.montanarosetnosy: - skip.montanaro
2011-03-19 09:46:03georg.brandlsetnosy: + georg.brandl
2011-03-19 06:02:25eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, terry.reedy, mark.dickinson, ncoghlan, pitrou, nadeem.vawda, benjamin.peterson, alex, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131396
2011-03-19 05:27:41terry.reedysetnosy: + terry.reedy
messages: + msg131392
2011-03-19 05:23:26terry.reedylinkissue11462 superseder
2011-03-18 17:05:53nadeem.vawdasetnosy: + nadeem.vawda
2011-03-18 12:47:42mark.dickinsonsetnosy: + mark.dickinson
2011-03-17 04:12:22eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, alex, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131214
2011-03-17 02:02:56eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, alex, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131210
2011-03-16 23:32:15alexsetnosy: + alex
2011-03-16 20:34:47ncoghlansetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131172
2011-03-16 20:13:32eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131166
2011-03-16 03:07:42ncoghlansetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131085
2011-03-16 02:55:41eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131084
2011-03-16 02:52:00ncoghlansetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131083
2011-03-16 00:16:07eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131074
2011-03-15 23:55:43pitrousetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131072
2011-03-15 23:51:05eltodersetnosy: skip.montanaro, brett.cannon, rhettinger, ncoghlan, pitrou, benjamin.peterson, Trundle, dmalcolm, daniel.urban, santoso.wijaya, eltoder
messages: + msg131071
2011-03-15 23:50:08skip.montanarosetnosy: + skip.montanaro
messages: + msg131070
2011-03-15 22:20:36ncoghlansetnosy: + brett.cannon
2011-03-15 11:52:26ncoghlansetnosy: + dmalcolm, ncoghlan
2011-03-15 08:57:14daniel.urbansetnosy: + daniel.urban
2011-03-15 05:02:40pitrousetnosy: + benjamin.peterson
2011-03-15 05:00:57santoso.wijayasetnosy: + santoso.wijaya
2011-03-15 04:54:03Trundlesetnosy: + Trundle
2011-03-15 04:53:35eltodersetnosy: + rhettinger, pitrou
2011-03-15 04:48:11eltodersetfiles: + 0_tests.patch
2011-03-15 04:48:01eltodersetfiles: + 0_generated.patch
2011-03-15 04:47:46eltodersetfiles: + 0_compile.patch
2011-03-15 04:47:36eltodersetfiles: + 0_fold.patch
2011-03-15 04:47:18eltodersetfiles: + 0_ast.patch
keywords: + patch
2011-03-15 04:46:42eltodercreate