classification
Title: Missing *-unpacking generalizations
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: twouters Nosy List: Jeff.Kaufman, Joshua.Landau, Rosuav, SpaghettiToastBook, akuchling, andybuckley, belopolsky, eric.araujo, eric.snow, ezio.melotti, fdrake, fhahn, georg.brandl, giampaolo.rodola, gvanrossum, ncoghlan, pconnell, pmoore, r.david.murray, terry.reedy, twouters, zbysz
Priority: normal Keywords: patch

Created on 2008-03-15 15:41 by twouters, last changed 2014-02-07 22:42 by pconnell.

Files
File name Uploaded Description Edit
starunpack.diff twouters, 2008-04-05 23:59
starunpack2.diff fhahn, 2013-11-17 15:15 review
Messages (39)
msg63548 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-03-15 15:41
The attached patch adds the missing *-unpacking generalizations.
Specifically:

>>> a, b, *c = range(5)

>>> *a, b, c = a, b, *c
>>> a, b, c
([0, 1, 2], 3, 4)
>>> [ *a, b, c ]
[0, 1, 2, 3, 4]
>>> L = [ a, (3, 4), {5}, {6: None}, (i for i in range(7, 10)) ]
>>> [ *item for item in L ]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Also, yielding everything from an iterator:

>>> def flatten(iterables):
...     for it in iterables:
...         yield *it
... 
>>> L = [ a, (3, 4), {5}, {6: None}, (i for i in range(7, 10)) ]
>>> flatten(L)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
msg63550 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-03-15 16:09
This was discussed years ago and never got enough support:

http://mail.python.org/pipermail/python-dev/2005-October/057177.html
msg63551 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-03-15 16:12
Didn't you say it does sets too?  Does this work?
a = [1, 2, 3]
{1, *a, 0, 4}   # {0, 1, 2, 3, 4}

How about dicts?
kwds = {'z': 0, 'w': 12}
{'x': 1, 'y': 2, **kwds}  # {'x': 1, 'y': 2, 'z': 0, 'w': 12}

Also, now that we support

[*a, b, c]

shouldn't we also support

foo(*a, b, c)

?
msg63552 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-03-15 16:14
On Sat, Mar 15, 2008 at 9:12 AM, Guido van Rossum <report@bugs.python.org>
wrote:

>
> Guido van Rossum <guido@python.org> added the comment:
>
> Didn't you say it does sets too?  Does this work?
> a = [1, 2, 3]
> {1, *a, 0, 4}   # {0, 1, 2, 3, 4}

Yes.

>
>
> How about dicts?
> kwds = {'z': 0, 'w': 12}
> {'x': 1, 'y': 2, **kwds}  # {'x': 1, 'y': 2, 'z': 0, 'w': 12}

Not yet.

>
>
> Also, now that we support
>
> [*a, b, c]
>
> shouldn't we also support
>
> foo(*a, b, c)
>

Sure. (And also 'foo(*a, *b, *c)'?) But have you taken a look lately at the
function definition grammar? I need some time to sort it out :)
msg63553 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-03-15 16:18
Looking at the flatten() example I'm curious -- how come the output of

>>> flatten(L)

is displayed as a list rather than as <generator at xxxxxx> ?

Also, do I understand correctly that

yield *(1, 2, 3)

is equivalent to

yield 1
yield 2
yield 3

? (That's really cool.)
msg63554 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-03-15 16:22
On Sat, Mar 15, 2008 at 9:18 AM, Guido van Rossum <report@bugs.python.org>
wrote:

>
> Guido van Rossum <guido@python.org> added the comment:
>
> Looking at the flatten() example I'm curious -- how come the output of
>
> >>> flatten(L)
>
> is displayed as a list rather than as <generator at xxxxxx> ?
>

It's a typo. It should've been list(flatten(L)) :-) (see the tests included
in the patch.)
msg63557 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-03-15 16:55
It looks like I completely missed PEP 3132.  Sorry for a misleading 
comment above.

At least I am not alone: "A little new feature in Python 3 that many 
overviews don't tell you about: extended unpacking." http://pyside.blogspot.com/2007/10/new-in-python-3-extended-
unpacking.html
msg63563 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-03-15 20:15
Just a nit: the syntax error message for invalid starred expressions
should be changed from "can use starred expression only as assignment
target".
msg63859 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-03-18 03:02
This is fun, but needs more work (see python-3000 thread).

I'm setting the priority to low, since I won't hold up a release to get
this in (if there's even a rough consensus).
msg65012 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-04-05 23:59
Updated patch: reworked some internals, and added generalization of
functioncalls after talking with Guido. *args is now considered just
another positional argument, and can occur anywhere in the positional
argument section. It can also occur more than once. Keyword arguments
now have to appear after *args, and **kwargs can now occur multiple
times at any position in the keyword argument list. test_extcall has
some examples.

(The opcodes are largely unaffected; just the order of '*args' and
keyword arguments is changed. Behind the scenes, anything after the
first '*args' argument is collapsed into a single *args, and everything
after the first '**kwargs' is likewise collapsed. The common case
(meaning any currently valid syntax, barring the 2to3 fix to swap *args
and keyword arguments) does not change in meaning or codepath, just the
complex cases are handled differently.)

This is still Work In Progress. To do: implement the dict unpacking
syntax (the mechanics are already there for keyword arguments to
functioncalls), make sure the precendence of * is correct, get more
complete test coverage, iron out the cosmetic bugs in the 2to3 fixer.

Bzr branch for this patch is
http://code.python.org/python/users/twouters/starunpack . There is also
a branch with just the functioncall changes (although the starunpack
changes are a small sprinkling on top of that branch, as it uses the
same new mechanics): http://code.python.org/python/users/twouters/funcargs .
msg65018 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-04-06 04:34
What's dict unpacking? I hope it's not an implementation of this sad
idea posted recently:

{'a': x, 'b': y} = {'a': 42, 'b': 'hello'}  # Same as x, y = 42, 'hello'

:-)
msg65023 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-04-06 06:38
No, it's what you asked for in msg63551:

> How about dicts?
> kwds = {'z': 0, 'w': 12}
> {'x': 1, 'y': 2, **kwds}  # {'x': 1, 'y': 2, 'z': 0, 'w': 12}

(unpacking of dicts in dicts.)
msg65079 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-04-07 17:45
I think we should either get this into the 3.0a5 release planned for May
7, or wait for 3.1.  I'd prefer to see some kind of PEP discussion on
the python-3000 list, rather than just a BDFL approval in a tracker
issue.  I think it's a useful feature (especially now we already have
PEP 3132) but we're getting close to the release, so I'd like to see
some more eyeballs on this code...  I expect the PEP discussion will be
short and sweet -- either most folks like it, or we should not push
through at this point in time.
msg65089 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-04-07 18:28
Agreed. A PEP was already on my TODO list (although I don't mind if
someone else picks it up :-) I implemented the
dict-unpacking-in-dict-literal syntax in the mean time; it's pushed to
the starunpack bzr branch, but I'll add some actual tests before I
upload the patch.
msg65095 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-04-07 19:00
Thomas,

Could you add BUILD_*_UNPACK opcodes documentation to
Doc/library/dis.rst?   It would also help if you modify CALL_FUNCTION_*
opcodes' documentation to explain how they will interact with unpacking
opcodes.

Do I understand correctly that non-starred arguments are packed into
intermediate tuples/sets in the presence of starred arguments so that
{a,b,*c,d,e} is equivalent to {*{a,b},*c,*{d,e}}? This should not be a
problem for tuples, but with sets, it means that {a,b,c} may behave
subtly differently from {a,*(b,c)}.
msg65096 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-04-07 19:07
>  Do I understand correctly that non-starred arguments are packed into
>  intermediate tuples/sets in the presence of starred arguments so that
>  {a,b,*c,d,e} is equivalent to {*{a,b},*c,*{d,e}}? This should not be a
>  problem for tuples, but with sets, it means that {a,b,c} may behave
>  subtly differently from {a,*(b,c)}.

Can you show an example where this would be different?
msg65100 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-04-07 19:28
On Mon, Apr 7, 2008 at 9:00 PM, Alexander Belopolsky <report@bugs.python.org>
wrote:

>
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
>
> Thomas,
>
> Could you add BUILD_*_UNPACK opcodes documentation to
> Doc/library/dis.rst?   It would also help if you modify CALL_FUNCTION_*
> opcodes' documentation to explain how they will interact with unpacking
> opcodes.

They don't interact. They're separate opcodes. The CALL_FUNCTION_* opcodes
are almost untouched, except the _VAR and _VAR_KW versions: previously, they
expected, on the stack, positional arguments followed by keyword/argument
pairs followed by the *args sequence followed by the **kwargs mapping (for
_VAR_KW.) I just changed the order so *args is before the keyword/argument
pairs. The change is not related to the BUILD_*_UNPACK opcodes, but rather
to Guido's request that the order of keyword arguments and *args in the
functioncall syntax changes. For the order of execution to remain sane, the
arguments need to be pushed on the stack in that order, and keeping the
_VAR* opcode order the same would mean a large amount of ROT_* opcodes ;-P

Updating the docs is on the TODO list.

>
> Do I understand correctly that non-starred arguments are packed into
> intermediate tuples/sets in the presence of starred arguments so that
> {a,b,*c,d,e} is equivalent to {*{a,b},*c,*{d,e}}? This should not be a
> problem for tuples, but with sets, it means that {a,b,c} may behave
> subtly differently from {a,*(b,c)}.
>

Yes, that's what happens, but only in the presence of *args. For
functioncalls, it only happens to everything after the first *args
(inclusive.) That means '{a, b, c}' does not change, and neither does
'func(a, b, c)' or 'func(a, b, *c)'. As for sets, I don't see why this would
be a problem; there is no difference in the set created by {a, b, c} and the
set created by {a, *{b, c}} or {a, *(b, c)}.  The arguments are all
evaluated in the same order (left to right), and c replaces b, b replaces a
if they are considered equal by sets.
msg65109 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-04-07 19:53
On Mon, Apr 7, 2008 at 3:07 PM, Guido van Rossum <report@bugs.python.org> wrote:

>  Can you show an example where this would be different?

Admittedly a contrived example, but

...    def __hash__(self):
...        print('hash', self)
...        return int.__hash__(self)
...
>>> a,b,c = map(X, range(3))
>>> {a,b,c}
hash 2
hash 1
hash 0
{0, 1, 2}
>>> {a,*(b,c)}
hash 0
hash 1
hash 2
{0, 1, 2}
msg65110 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-04-07 19:58
I missed the first line copying the class definition into my previous
post.  Class 'X' was defined as follows:

class X(int):
    def __hash__(self):
        print('hash', self)
        return int.__hash__(self)
msg65111 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-04-07 20:02
I'm not sure how this matters. The order of evaluation is the same, the
BUILD_SET implementation just hashes the evaluated items in a different
order. You can't really rely on that particular order as it's tied
closely to the stack representation CPython uses. I also see no
practical reason -- or even practical *way* -- to abuse the hashing
order. But you have given me an idea on how to improve some of the code
in the BUILD_*_UNPACK opcodes, hmm.
msg65116 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-04-07 20:14
I agree with Thomas.  The order in which __hash__() is evaluated
shouldn't matter to your app; if __hash__() isn't a pure function (apart
from possible caching) you've got worse trouble.
msg65117 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-04-07 20:15
On Mon, Apr 7, 2008 at 4:02 PM, Thomas Wouters <report@bugs.python.org> wrote:
> .. The order of evaluation is the same, the
>  BUILD_SET implementation just hashes the evaluated items in a different
>  order. You can't really rely on that particular order as it's tied
>  closely to the stack representation CPython uses.

I agree and that's why I said the difference in behavior is "subtle"
and my example is "contrived."   However, I believe Raymond expressed
a similar concern in msg63065.
msg65125 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2008-04-07 21:33
I don't think the order in which the items are hashed is really what
Raymond was worried about. Rather, the size of the stack was, and the
effect of having all the items on the stack at the same time. I think
Raymond is wrong in this case; while the stack may grow relatively big,
we're only talking two pointers here. The items will all have to be
created anyway, and in the usual case the number of duplicate keys is low. 

My patch actually includes pretty much the same change to BUILD_MAP,
because it greatly simplifies the compiler code and gets rid of a lot of
extra opcodes -- causing an overal speedup even in the face of large
dict literals. But I guess we should take it up with Raymond at some
point, perhaps as part of the PEP discussion.
msg67465 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-05-28 21:29
What's the status of this? Is it still under consideration for 3.0 -- if
yes, that should get decided before the beta.
msg88533 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2009-05-29 20:39
I was hoping this would make 3.1.  Too late, I guess.  What about 3.2?
msg88537 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-05-29 20:52
> I was hoping this would make 3.1.  Too late, I guess.  What about 3.2?

Here's what I said before:

"""
I think we should either get this into the 3.0a5 release planned for May
7, or wait for 3.1.  I'd prefer to see some kind of PEP discussion on
the python-3000 list, rather than just a BDFL approval in a tracker
issue.  I think it's a useful feature (especially now we already have
PEP 3132) but we're getting close to the release, so I'd like to see
some more eyeballs on this code...  I expect the PEP discussion will be
short and sweet -- either most folks like it, or we should not push
through at this point in time.
"""

I still think this is the way to do it.  I'm +0 myself.  We might decide
not to do it just because py3k needs stability more than it needs more
features.
msg100293 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2010-03-02 14:24
Updating version; does someone want to revive this for 3.2?
msg100295 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-03-02 14:31
I believe this would be blocked by the moratorium.  Correct me if I'm wrong.
msg100296 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-03-02 15:22
fix inadvertent reassignment.
msg100304 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-03-02 19:12
I would like to see this revisited when the moratorium is lifted.
Added 3.3 as a surrogate for that.
As per GvR above, it will need a PEP to pin down details, alternatives, and use cases and discussion on python-ideas list.
msg100305 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-03-02 19:15
Whoops, just noticed newish 'after moratorium' keyword. Good idea.
msg116416 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-14 18:05
Does

 yield *it

mean

 yield iter(tuple(it))

or

 for i in it:
     yield i
?
msg149588 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-16 01:27
In the python-ideas thread "list / array comprehensions extension"
Guido replied in reference to an earlier quote from him:
"I think that -0 was contextual (too many moving parts for the original Py3k release). Today I am +1."

There are others in favor, pending a PEP that specifies all the details.
msg151143 - (view) Author: Zbyszek Jędrzejewski-Szmek (zbysz) * Date: 2012-01-12 18:03
#11682 will likely be merged. The part of this patch about "yielding everything from an iterator" becomes obsolete:
>>> def flatten(iterables):
...    for it in iterables:
...      yield from it
... 
>>> L = [ [0,1,2], (3, 4), {5}, {6: None}, (i for i in range(7, 10)) ]
>>> list(flatten(L))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The rest is of course still valid and useful.
msg186099 - (view) Author: Jeff Kaufman (Jeff.Kaufman) Date: 2013-04-05 18:29
What would it take to get this moving again?
msg186101 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-05 19:14
Someone needs to write the PEP.
msg191754 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-06-24 12:11
Since this question just came up on python-ideas again, here's a summary of the current status:

1. The current patch is known to be outdated due to the inclusion of PEP 380 in Python 3.3 ("yield from itr" eliminates any need for "yield *itr")

2. Since this is a syntax change, a PEP is needed to elaborate on the exact details of what will be added (see PEP 3132 as an example of a PEP that covered a similar change for assignment *targets*)
msg192984 - (view) Author: Joshua Landau (Joshua.Landau) Date: 2013-07-13 00:49
http://www.python.org/dev/peps/pep-0448/ is out; see what you think.

See http://mail.python.org/pipermail/python-ideas/2013-July/021872.html for all the juicy discussion so far.
msg203191 - (view) Author: (fhahn) Date: 2013-11-17 15:15
I've updated the patch to apply to the current tip.

But this patch breaks *-unpacking, I'll try to take a closer look during the next week.
History
Date User Action Args
2014-02-07 22:42:27pconnellsetnosy: + pconnell
2014-01-30 23:58:38Rosuavsetnosy: + Rosuav
2013-11-17 15:15:53fhahnsetfiles: + starunpack2.diff
nosy: + fhahn
messages: + msg203191

2013-07-13 00:49:13Joshua.Landausetnosy: + Joshua.Landau
messages: + msg192984
2013-06-24 21:26:50SpaghettiToastBooksetnosy: + SpaghettiToastBook
2013-06-24 12:11:37ncoghlansetmessages: + msg191754
2013-04-05 19:14:02r.david.murraysetmessages: + msg186101
2013-04-05 18:29:58Jeff.Kaufmansetnosy: + Jeff.Kaufman
messages: + msg186099
2012-09-26 17:51:51ezio.melottisetkeywords: - after moratorium
versions: + Python 3.4, - Python 3.3
2012-01-16 08:55:42ezio.melottisetnosy: + ncoghlan
2012-01-12 18:03:34zbyszsetnosy: + zbysz
messages: + msg151143
2011-12-16 01:27:09terry.reedysetmessages: + msg149588
2011-12-16 00:52:05eric.snowsetnosy: + eric.snow
2011-12-15 23:16:07pmooresetnosy: + pmoore
2011-12-15 17:18:39ezio.melottisetpriority: low -> normal
2011-04-22 14:29:25fdrakesetnosy: + fdrake
2011-03-21 19:47:13ezio.melottisetnosy: gvanrossum, twouters, akuchling, georg.brandl, terry.reedy, belopolsky, giampaolo.rodola, ezio.melotti, eric.araujo, andybuckley, r.david.murray
versions: + Python 3.3
2011-01-13 23:23:55giampaolo.rodolasetnosy: + giampaolo.rodola
2010-09-14 18:05:05eric.araujosetnosy: + eric.araujo
messages: + msg116416
2010-09-14 14:32:09andybuckleysetnosy: + andybuckley
2010-06-26 23:31:35ezio.melottisetkeywords: patch, patch, after moratorium
nosy: + ezio.melotti
2010-03-02 19:15:18terry.reedysetkeywords: patch, patch, after moratorium

messages: + msg100305
versions: - Python 3.3
2010-03-02 19:12:32terry.reedysetkeywords: patch, patch, after moratorium

messages: + msg100304
versions: + Python 3.3
2010-03-02 15:22:21r.david.murraysetnosy: - pedronis
messages: + msg100296

assignee: pedronis -> twouters
keywords: patch, patch, after moratorium
2010-03-02 14:31:57r.david.murraysetassignee: twouters -> pedronis
versions: - Python 3.2
keywords: + after moratorium
nosy: + r.david.murray, pedronis

messages: + msg100295
stage: patch review
2010-03-02 14:24:55akuchlingsetversions: + Python 3.2, - Python 3.0
nosy: + akuchling

messages: + msg100293

keywords: patch, patch
2009-05-29 20:52:30gvanrossumsetkeywords: patch, patch

messages: + msg88537
2009-05-29 20:41:07terry.reedylinkissue6100 superseder
2009-05-29 20:39:38terry.reedysetkeywords: patch, patch
nosy: + terry.reedy
messages: + msg88533

2008-05-28 21:29:55georg.brandlsetkeywords: patch, patch
messages: + msg67465
2008-04-07 21:33:30twouterssetkeywords: patch, patch
messages: + msg65125
2008-04-07 20:15:11belopolskysetmessages: + msg65117
2008-04-07 20:14:31gvanrossumsetkeywords: patch, patch
messages: + msg65116
2008-04-07 20:02:09twouterssetkeywords: patch, patch
messages: + msg65111
2008-04-07 19:58:37belopolskysetmessages: + msg65110
2008-04-07 19:53:34belopolskysetmessages: + msg65109
2008-04-07 19:34:08twouterssetfiles: - unnamed
2008-04-07 19:28:01twouterssetfiles: + unnamed
messages: + msg65100
2008-04-07 19:07:51gvanrossumsetmessages: + msg65096
2008-04-07 19:00:14belopolskysetmessages: + msg65095
2008-04-07 18:28:32twouterssetkeywords: patch, patch
messages: + msg65089
2008-04-07 17:45:11gvanrossumsetkeywords: patch, patch
messages: + msg65079
2008-04-06 06:38:07twouterssetkeywords: patch, patch
messages: + msg65023
2008-04-06 04:34:30gvanrossumsetkeywords: patch, patch
messages: + msg65018
2008-04-06 00:02:48twouterssetfiles: - morestar.diff
2008-04-06 00:00:00twouterssetkeywords: patch, patch
files: + starunpack.diff
messages: + msg65012
2008-03-18 03:02:20gvanrossumsetpriority: low
assignee: gvanrossum -> twouters
messages: + msg63859
keywords: patch, patch
2008-03-15 20:15:53georg.brandlsetkeywords: patch, patch
nosy: + georg.brandl
messages: + msg63563
2008-03-15 16:55:09belopolskysetmessages: + msg63557
2008-03-15 16:25:56gvanrossumsetfiles: - unnamed
2008-03-15 16:25:51gvanrossumsetfiles: - unnamed
2008-03-15 16:22:37twouterssetfiles: + unnamed
messages: + msg63554
2008-03-15 16:18:09gvanrossumsetkeywords: patch, patch
messages: + msg63553
2008-03-15 16:14:25twouterssetfiles: + unnamed
messages: + msg63552
2008-03-15 16:12:12gvanrossumsetkeywords: patch, patch
messages: + msg63551
2008-03-15 16:09:27belopolskysetnosy: + belopolsky
type: enhancement
messages: + msg63550
2008-03-15 15:41:19twouterscreate