The attached patch adds the missing *-unpacking generalizations.
Specifically:

>> a, b, *c = range(5)

>>> *a, b, c = a, b, *c
>>> a, b, c
([0, 1, 2], 3, 4)
>>> [ *a, b, c ]
[0, 1, 2, 3, 4]
>>> L = [ a, (3, 4), {5}, {6: None}, (i for i in range(7, 10)) ]
>>> [ *item for item in L ]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Also, yielding everything from an iterator:

>>> def flatten(iterables):
...     for it in iterables:
...         yield *it
... 
>>> L = [ a, (3, 4), {5}, {6: None}, (i for i in range(7, 10)) ]
>>> flatten(L)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

This was discussed years ago and never got enough support:

http://mail.python.org/pipermail/python-dev/2005-October/057177.html

Didn't you say it does sets too? Does this work?
a = [1, 2, 3]
{1, *a, 0, 4} # {0, 1, 2, 3, 4}

How about dicts?
kwds = {'z': 0, 'w': 12}
{'x': 1, 'y': 2, **kwds} # {'x': 1, 'y': 2, 'z': 0, 'w': 12}

Also, now that we support

[*a, b, c]

shouldn't we also support

foo(*a, b, c)

?

On Sat, Mar 15, 2008 at 9:12 AM, Guido van Rossum <report@bugs.python.org>
wrote:

Guido van Rossum <guido@python.org> added the comment:

Didn't you say it does sets too? Does this work?
a = [1, 2, 3]
{1, *a, 0, 4} # {0, 1, 2, 3, 4}

Yes.

How about dicts?
kwds = {'z': 0, 'w': 12}
{'x': 1, 'y': 2, **kwds} # {'x': 1, 'y': 2, 'z': 0, 'w': 12}

Not yet.

Also, now that we support

[*a, b, c]

shouldn't we also support

foo(*a, b, c)

Sure. (And also 'foo(*a, *b, *c)'?) But have you taken a look lately at the
function definition grammar? I need some time to sort it out :)

Looking at the flatten() example I'm curious -- how come the output of

>> flatten(L)

is displayed as a list rather than as <generator at xxxxxx> ?

Also, do I understand correctly that

yield *(1, 2, 3)

is equivalent to

yield 1
yield 2
yield 3

? (That's really cool.)

On Sat, Mar 15, 2008 at 9:18 AM, Guido van Rossum <report@bugs.python.org>
wrote:

Guido van Rossum <guido@python.org> added the comment:

Looking at the flatten() example I'm curious -- how come the output of

>>> flatten(L)

is displayed as a list rather than as <generator at xxxxxx> ?

It's a typo. It should've been list(flatten(L)) :-) (see the tests included
in the patch.)

It looks like I completely missed PEP-3132. Sorry for a misleading
comment above.

At least I am not alone: "A little new feature in Python 3 that many
overviews don't tell you about: extended unpacking." http://pyside.blogspot.com/2007/10/new-in-python-3-extended-
unpacking.html

Just a nit: the syntax error message for invalid starred expressions
should be changed from "can use starred expression only as assignment
target".

This is fun, but needs more work (see python-3000 thread).

I'm setting the priority to low, since I won't hold up a release to get
this in (if there's even a rough consensus).

Updated patch: reworked some internals, and added generalization of
functioncalls after talking with Guido. *args is now considered just
another positional argument, and can occur anywhere in the positional
argument section. It can also occur more than once. Keyword arguments
now have to appear after *args, and **kwargs can now occur multiple
times at any position in the keyword argument list. test_extcall has
some examples.

(The opcodes are largely unaffected; just the order of '*args' and
keyword arguments is changed. Behind the scenes, anything after the
first '*args' argument is collapsed into a single *args, and everything
after the first '**kwargs' is likewise collapsed. The common case
(meaning any currently valid syntax, barring the 2to3 fix to swap *args
and keyword arguments) does not change in meaning or codepath, just the
complex cases are handled differently.)

This is still Work In Progress. To do: implement the dict unpacking
syntax (the mechanics are already there for keyword arguments to
functioncalls), make sure the precendence of * is correct, get more
complete test coverage, iron out the cosmetic bugs in the 2to3 fixer.

Bzr branch for this patch is
http://code.python.org/python/users/twouters/starunpack . There is also
a branch with just the functioncall changes (although the starunpack
changes are a small sprinkling on top of that branch, as it uses the
same new mechanics): http://code.python.org/python/users/twouters/funcargs .

What's dict unpacking? I hope it's not an implementation of this sad
idea posted recently:

{'a': x, 'b': y} = {'a': 42, 'b': 'hello'} # Same as x, y = 42, 'hello'

:-)

No, it's what you asked for in msg63551:

How about dicts?
kwds = {'z': 0, 'w': 12}
{'x': 1, 'y': 2, **kwds} # {'x': 1, 'y': 2, 'z': 0, 'w': 12}

(unpacking of dicts in dicts.)

I think we should either get this into the 3.0a5 release planned for May
7, or wait for 3.1. I'd prefer to see some kind of PEP discussion on
the python-3000 list, rather than just a BDFL approval in a tracker
issue. I think it's a useful feature (especially now we already have
PEP-3132) but we're getting close to the release, so I'd like to see
some more eyeballs on this code... I expect the PEP discussion will be
short and sweet -- either most folks like it, or we should not push
through at this point in time.

Agreed. A PEP was already on my TODO list (although I don't mind if
someone else picks it up :-) I implemented the
dict-unpacking-in-dict-literal syntax in the mean time; it's pushed to
the starunpack bzr branch, but I'll add some actual tests before I
upload the patch.

Thomas,

Could you add BUILD_UNPACK opcodes documentation to
Doc/library/dis.rst? It would also help if you modify CALL_FUNCTION
opcodes' documentation to explain how they will interact with unpacking
opcodes.

Do I understand correctly that non-starred arguments are packed into
intermediate tuples/sets in the presence of starred arguments so that
{a,b,c,d,e} is equivalent to {{a,b},c,{d,e}}? This should not be a
problem for tuples, but with sets, it means that {a,b,c} may behave
subtly differently from {a,*(b,c)}.

Do I understand correctly that non-starred arguments are packed into
intermediate tuples/sets in the presence of starred arguments so that
{a,b,c,d,e} is equivalent to {{a,b},c,{d,e}}? This should not be a
problem for tuples, but with sets, it means that {a,b,c} may behave
subtly differently from {a,*(b,c)}.

Can you show an example where this would be different?

On Mon, Apr 7, 2008 at 9:00 PM, Alexander Belopolsky <report@bugs.python.org>
wrote:

Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:

Thomas,

Could you add BUILD_UNPACK opcodes documentation to
Doc/library/dis.rst? It would also help if you modify CALL_FUNCTION
opcodes' documentation to explain how they will interact with unpacking
opcodes.

They don't interact. They're separate opcodes. The CALL_FUNCTION_* opcodes
are almost untouched, except the _VAR and _VAR_KW versions: previously, they
expected, on the stack, positional arguments followed by keyword/argument
pairs followed by the *args sequence followed by the **kwargs mapping (for
_VAR_KW.) I just changed the order so *args is before the keyword/argument
pairs. The change is not related to the BUILD__UNPACK opcodes, but rather
to Guido's request that the order of keyword arguments and *args in the
functioncall syntax changes. For the order of execution to remain sane, the
arguments need to be pushed on the stack in that order, and keeping the
_VAR opcode order the same would mean a large amount of ROT_* opcodes ;-P

Updating the docs is on the TODO list.

Do I understand correctly that non-starred arguments are packed into
intermediate tuples/sets in the presence of starred arguments so that
{a,b,c,d,e} is equivalent to {{a,b},c,{d,e}}? This should not be a
problem for tuples, but with sets, it means that {a,b,c} may behave
subtly differently from {a,*(b,c)}.

Yes, that's what happens, but only in the presence of *args. For
functioncalls, it only happens to everything after the first *args
(inclusive.) That means '{a, b, c}' does not change, and neither does
'func(a, b, c)' or 'func(a, b, *c)'. As for sets, I don't see why this would
be a problem; there is no difference in the set created by {a, b, c} and the
set created by {a, *{b, c}} or {a, *(b, c)}. The arguments are all
evaluated in the same order (left to right), and c replaces b, b replaces a
if they are considered equal by sets.

On Mon, Apr 7, 2008 at 3:07 PM, Guido van Rossum <report@bugs.python.org> wrote:

Can you show an example where this would be different?

Admittedly a contrived example, but

...    def __hash__(self):
...        print('hash', self)
...        return int.__hash__(self)
...
>>> a,b,c = map(X, range(3))
>>> {a,b,c}
hash 2
hash 1
hash 0
{0, 1, 2}
>>> {a,*(b,c)}
hash 0
hash 1
hash 2
{0, 1, 2}

I missed the first line copying the class definition into my previous
post. Class 'X' was defined as follows:

class X(int):
    def __hash__(self):
        print('hash', self)
        return int.__hash__(self)

I'm not sure how this matters. The order of evaluation is the same, the
BUILD_SET implementation just hashes the evaluated items in a different
order. You can't really rely on that particular order as it's tied
closely to the stack representation CPython uses. I also see no
practical reason -- or even practical *way* -- to abuse the hashing
order. But you have given me an idea on how to improve some of the code
in the BUILD_*_UNPACK opcodes, hmm.

I agree with Thomas. The order in which __hash__() is evaluated
shouldn't matter to your app; if __hash__() isn't a pure function (apart
from possible caching) you've got worse trouble.

On Mon, Apr 7, 2008 at 4:02 PM, Thomas Wouters <report@bugs.python.org> wrote:

.. The order of evaluation is the same, the
BUILD_SET implementation just hashes the evaluated items in a different
order. You can't really rely on that particular order as it's tied
closely to the stack representation CPython uses.

I agree and that's why I said the difference in behavior is "subtle"
and my example is "contrived." However, I believe Raymond expressed
a similar concern in msg63065.

I don't think the order in which the items are hashed is really what
Raymond was worried about. Rather, the size of the stack was, and the
effect of having all the items on the stack at the same time. I think
Raymond is wrong in this case; while the stack may grow relatively big,
we're only talking two pointers here. The items will all have to be
created anyway, and in the usual case the number of duplicate keys is low.

My patch actually includes pretty much the same change to BUILD_MAP,
because it greatly simplifies the compiler code and gets rid of a lot of
extra opcodes -- causing an overal speedup even in the face of large
dict literals. But I guess we should take it up with Raymond at some
point, perhaps as part of the PEP discussion.

What's the status of this? Is it still under consideration for 3.0 -- if
yes, that should get decided before the beta.

I was hoping this would make 3.1. Too late, I guess. What about 3.2?

Removed dead code. Awaiting code review! :)

All tests pass on my ubuntu 13.04 system.

Removed a confusing and outdated comment in Lib/importlib/_bootstrap.py

I have limited expertise in most of these areas, but I looked at starunpack40.diff and have these comments:

• tests look to have good coverage of the feature (can't speak to coverage of the parser/compiler code)
• parsermodule.c changes comprehension handling, but I thought we pulled this out?
• why was dictobject.c.h added? I don't understand clinic thoroughly, but it seems a lot of new code for what was changed in dictobject.c
• can the BUILD_(TUPLE|LIST)_UNPACK code in ceval.c share most of the processing? Or is there an unwanted perf impact to that?
• whoever applies the patch should regenerate importlib.h themselves - just a reminder

Otherwise, I didn't see anything that particularly scared me. Since we apparently don't have anyone willing and with the expertise to thoroughly check the patch, I'd vote for checking it in asap so it has more releases to bake before 3.5 final.

Hi Steve:

I have limited expertise in most of these areas, but I looked at starunpack40.diff and have these comments:

• tests look to have good coverage of the feature (can't speak to coverage of the parser/compiler code)
• parsermodule.c changes comprehension handling, but I thought we pulled this out?

There was some code taken common in various places, but there should be no code for unpacking comprehensions left in. Do you have a question about a specific line?

• why was dictobject.c.h added? I don't understand clinic thoroughly, but it seems a lot of new code for what was changed in dictobject.c

They weren't added. They were moved by someone else. The only changes are exposing a method.

• can the BUILD_(TUPLE|LIST)_UNPACK code in ceval.c share most of the processing? Or is there an unwanted perf impact to that?

Good idea. I'll make that change.

• whoever applies the patch should regenerate importlib.h themselves - just a reminder

Otherwise, I didn't see anything that particularly scared me. Since we apparently don't have anyone willing and with the expertise to thoroughly check the patch, I'd vote for checking it in asap so it has more releases to bake before 3.5 final.

Thanks!

All tests pass. All reviewer comments addressed. Please let me know if anything else needs to be done from my end.

The latest patch looks good to me. No need to do the additional AST refactoring if it's going to make PEP-492's implementor's life harder (but I do read Guido's comment as a reason to check this in sooner rather than later :>) So, unless anyone objects I'll check in the latest patch on Monday.

Thanks for the review Thomas! And yes, that's what I meant. :-)

It certainly would be nice to have documentation.

Yeah, but the docs don't need to be committed in time for beta 1. The
source code should go in ASAP, especially since the PEP-492 changes will
have to be merged in on top of them. @thomas: which Monday were you
shooting for? I had hoped yesterday...

On Sat, May 2, 2015 at 6:52 PM, Benjamin Peterson <report@bugs.python.org>
wrote:

Benjamin Peterson added the comment:

It certainly would be nice to have documentation.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue2292\>

---

(To clarify, the PEP itself probably serves as enough documentation in the
interim.)

On Tue, May 5, 2015 at 4:47 PM, Guido van Rossum <guido@python.org> wrote:

Yeah, but the docs don't need to be committed in time for beta 1. The
source code should go in ASAP, especially since the PEP-492 changes will
have to be merged in on top of them. @thomas: which Monday were you
shooting for? I had hoped yesterday...

On Sat, May 2, 2015 at 6:52 PM, Benjamin Peterson <report@bugs.python.org>
wrote:

>
> Benjamin Peterson added the comment:
>
> It certainly would be nice to have documentation.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue2292\>
> _______________________________________
>

--
--Guido van Rossum (python.org/~guido)

On Tue, May 5, 2015, at 19:48, Guido van Rossum wrote:

Guido van Rossum added the comment:

Yeah, but the docs don't need to be committed in time for beta 1. The
source code should go in ASAP, especially since the PEP-492 changes will
have to be merged in on top of them. @thomas: which Monday were you
shooting for? I had hoped yesterday...

I suppose I can just do it.

a65f685ba8c0

Thanks Benjamin!
On May 5, 2015 5:17 PM, "Benjamin Peterson" <report@bugs.python.org> wrote:

Benjamin Peterson added the comment:

a65f685ba8c0

----------
resolution: -> fixed
status: open -> closed

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue2292\>

---

FYI, I meant last Monday, but I forgot it was May 4th (Dutch Memorial day) and May 5th (Dutch Liberation day), so that got in the way :P

Should we keep this bug open for docs changes, or is there a separate issue for that?

On Wed, May 6, 2015, at 09:15, Thomas Wouters wrote:

Thomas Wouters added the comment:

FYI, I meant last Monday, but I forgot it was May 4th (Dutch Memorial
day) and May 5th (Dutch Liberation day), so that got in the way :P

Should we keep this bug open for docs changes, or is there a separate
issue for that?

bpo-24136

I get a test failure in Cython's compatibility tests which seems to be attributable to this change:

    """
    >>> def sideeffect(x):
    ...     L.append(x)
    ...     return x
    >>> def unhashable(x):
    ...     L.append(x)
    ...     return [x]

    >>> L = []
    >>> {1:2, sideeffect(2): 3, 3: 4, unhashable(4): 5, sideeffect(5): 6}    # doctest: +ELLIPSIS
    Traceback (most recent call last):
    TypeError: ...unhashable...
    >>> L
    [2, 4]
    """

Instead, L ends up being [2, 4, 5]. Is this intended? Or acceptable?

There is a change as part of this to make dict building more like list and set building, which both have this behaviour.

The same changes have likely occurred before whenever BUILD_LIST and BUILD_SET were introduced, and this behaviour seems particularly undefined.

That said, I did overlook the difference. Hopefully there's agreement that it doesn't matter.

I think it's fine. It collects all the keys and values and then calls
BUILD_MAP (a new opcode), rather than calling STORE_MAP for each key/value
pair. I think this is a reasonable strategy for compiling a dict display.

On Thu, May 7, 2015 at 11:40 AM, Joshua Landau <report@bugs.python.org>
wrote:

Joshua Landau added the comment:

There is a change as part of this to make dict building more like list and
set building, which both have this behaviour.

The same changes have likely occurred before whenever BUILD_LIST and
BUILD_SET were introduced, and this behaviour seems particularly undefined.

That said, I did overlook the difference. Hopefully there's agreement that
it doesn't matter.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue2292\>

---

This could likely stand to be clarified in the language reference, though
(as well as in the 3.5 porting notes)

Do we need to make lib2to3 compatible with the new grammar?

Yes we should. I'd consider it a bug if it wasn't supported in 3.5.0 and we could fix that bug in 3.5.1.

I created bpo-25969 to track the lib2to3 update.

the docs don't need to be committed in time for beta 1.

10 months and 2 releases after the code patch, the doc patches in bpo-24136 are incomplete (I believe), uncommitted, untouched since July, and forgotten about. The issue needs more help.

bpo-26213 was opened for documenting bytecode changes. But 21 months and 3 releases after the code patch the documentation is still not updated.