This is a big one:

* cleanup grammar; unifies listcomp/genexp grammar
which means that [x for x in 1, 2] is no longer valid

* cleanup comprehension compiling code (unifies all AST
code for the three comprehensions and most of the
compile.c code)

* add set comprehensions

This patch modifies list comprehensions to be
implemented more like generator expressions: in a
separate function, which means that the loop variables
will not leak any more.

Logged In: YES 
user_id=849994

The previously attached patch contains only the important
files. The FULL patch (attached now) also contains syntax
fixes in python files so that the test suite is mostly passing.

Note that the compiler package isn't ready yet.

Logged In: YES 
user_id=6380

Nice!

I see failures in 4 tests:

    test_compiler test_dis test_transformer test_univnewlines

test_univnewlines is trivial (it's deleting a variable
leaked out of a list comprehension); haven't looked at the
rest in detail

Logged In: YES 
user_id=849994

test_compiler and test_transformer fail because the compiler
package hasn't been updated yet.

test_dis fails because list comprehensions now generate
different bytecode.

Logged In: YES 
user_id=80475

Can you post a before and disassembly of some list and set
comprehensions.

Logged In: YES 
user_id=849994

Attaching slightly revised patch and bytecode comparison.

Logged In: YES 
user_id=80475

Would it be an oversimplfication for list and set comps to
keep everything in one code block and just hide the list
loop variables by renaming them:   x -->  __[x]

That approach would only require a minimal patch, and it
would make for a cleaner disassembly.

Logged In: YES 
user_id=6380

+1.

Would this cause problems for abominations like this though?

>>> a=[1]
>>> list(tuple(a) for a[0] in "abc")
[('a',), ('b',), ('c',)]
>>> a
['c']
>>>

Logged In: YES 
user_id=849994

Since you can put anything usable as an assignment target
after the "for" of a listcomp, just renaming might be
complicated.

Logged In: YES 
user_id=6380

Do you think this is ready to be checked in, or are you
still working on it?

Logged In: YES 
user_id=849994

It is complete, it works and it does not leak the loop
variable(s).

The question is whether it is okay for listcomps and
setcomps to be in their own anonymous function, which slows
listcomps down compared to the 2.x branch.

I don't know why the function approach was taken for
genexps, but I suspect it was because who implemented it
then saw this as the best way to hide the loop variable.

Perhaps somebody else more familiar with the internals and
the previous discussions can look over it.

Logged In: YES 
user_id=6380

I always assumed that the genexps *require* being a function
because that's the only way to create a generator.  But that
argument doesn't apply to listcomps.

That's about all I know of the implementation of these.. :-(

Have you asked python-dev?

Logged In: YES 
user_id=80475

Genexps necessarily need a separate stack frame to achieve
saved execution state (including the instruction pointer and
local variable).  Also, it was simplest to implement genexps
in terms of the existing and proven code for regular generators.

For list and set comps, I think you can take a simpler
approach and just rename the inner loop variable to
something invisible.  That will make it faster, make the
disassemby readable, and make it easier to follow in pdb. 
Also, we get to capitalize on proven code -- they only
difference is that the induction variable won't be visible
to surrounding code.

Since what you have works, I would say just check it in;
however, it would probably never get touched again and an
early, arbitrary design choice would get set in stone.  My
bet is that the renaming approach will result in a much
simpler patch.

There was some discussion on the py3k list about Raymond's suggestion.  Are you thinking of doing that?  I'd really like to see the syntactic changes and additions from this patch, but I agree that for list/set comps we can do without the extra stack frame.

This patch is now superseded by #1660500.