classification
Title: Evaluation order of dictionary display is different from reference manual.
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.4, Python 3.3, Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, eric.araujo, ezio.melotti, georg.brandl, gvanrossum, ncoghlan, nedbat, r.david.murray, rhettinger, sandro.tosi, swamiyeswanth, takayuki, terry.reedy
Priority: low Keywords: patch

Created on 2011-02-13 09:39 by takayuki, last changed 2013-03-28 13:23 by terry.reedy.

Files
File name Uploaded Description Edit
compile.diff skip.montanaro, 2011-02-13 13:32 review
Messages (21)
msg128482 - (view) Author: (takayuki) Date: 2011-02-13 09:39
Running the following code shows "2 1 4 3", but in reference manual
http://docs.python.org/reference/expressions.html#expression-lists
the evaluation order described as
{expr1: expr2, expr3: expr4}

def f(i):
    print i
    return i

{f(1):f(2), f(3):f(4)}

I found some of past discussions about this problem, for example,
http://mail.python.org/pipermail/python-dev/2002-November/030461.html
http://mail.python.org/pipermail/python-dev/2002-November/030458.html
http://bugs.python.org/issue448679
http://svn.python.org/view?view=rev&revision=30148
, but current implementation seems not to reflect these accomplishment.

In present, which behaviour is legal?
msg128487 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 13:04
Here's a patch.
msg128489 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 13:08
Working on a test case too.
msg128490 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 13:14
It's not so easy as first appeared.
msg128492 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 13:32
Okay, this looks better.  I was confusing myself with leftover .pyc
files I think.  Test included.
msg128493 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 13:37
Georg,  I think this might need to sneak into 3.2.
msg128494 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-13 14:13
I don't think so -- it's a very minor deviation from the spec and not a critical bug.
msg128498 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 14:26
3.2 and earlier versions are all frozen, but for 3.3 it might make sense to bump the version number of the bytecode and change STORE_MAP to take the key and value in the opposite order, thus allowing to remove the ROT_TWO I had to add to make this work.
msg128500 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-02-13 14:35
Interesting.  I would actually have expected the observed behavior.  I think of the : in a dictionary literal as an assignment.
msg128502 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-02-13 14:56
BTW, it would be nice to know if this behavior was consistent with the docs at any time (the merge of the AST branch in 2.5 might be an obvious candidate where it was broken).

Also interesting would be what other implementations of Python do.
msg128507 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 15:17
As Georg suggested, it is correct in 2.4, wrong in 2.5.
msg128520 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-02-13 21:51
<sigh> If only a test had been checked-in eight years ago ...

It looks like a number of things have to be changed in order to bring behavior back to what the docs promise.  Dict literals and dict comprehensions need to be fixed in both the compile.py and compile.c.  To avoid introducing a ROT_TWO, the store STORE_MAP and MAP_ADD opcodes need minor modifications (just switch the u and w variable assignments).
msg128521 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-02-13 21:52
takayuki, a special thanks to you for submitting such a well-researched bug report :-)
msg128522 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2011-02-13 21:54
> To avoid introducing a ROT_TWO, the store STORE_MAP and MAP_ADD
> opcodes need minor modifications (just switch the u and w variable
> assignments).

Either of which would not be possible in anything other that 3.3 or 
later, right?
msg128530 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-02-13 23:41
How much to change and how far to backport may make for a good python-dev discussion.  

ISTM that changing generated code goes hand-in-hand with changing opcode semantics (as long as the magic number gets bumped).  OTOH, none of this may be worth backporting at all since no one seems to have noticed or cared for eight years.   

I don't have any feelings about it one way or the other, but it would great to see Py3.2.1 get fixed properly.
msg138223 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-06-12 22:35
If I understand, since the present patch uses present opcode semantics, adding a rotate, it could go into 2.7 and 3.2. Correct?

3.3 could get an improved patch that instead changed opcodes to expect key and value in the other order.
msg175120 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-11-07 19:18
Today in pydev thread "chained assignment weirdity" Guido said
http://permalink.gmane.org/gmane.comp.python.devel/135746
"I agree that we should be *very* conservative in changing the
meaning of existing opcodes (adding new one is a different story)."
...
"Hm. I really don't think that is a good development for Python to
compromise in the area of expression evaluation order where side
effects are involved."
...

"I haven't looked at the proposed fixes, but I think correctness is more important than saving an extra bytecode (OTOH keeping the set of opcodes the same trumps both). I can't imagine that this extra opcode will be significant in many cases."

To which Nick C. replied
"Since you've indicated the implementation is in the wrong here and you
also want to preserve opcode semantics, I think Skip's patch is
correct, but also needs to be applied to dict comprehensions (now we
have them). The extra bytecode is only ROT_TWO, which is one of the
cheapest we have kicking around."

To which Guido said "Ok, somebody go for it! (Also please refer to my pronouncement in the bug"
msg178133 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-12-25 14:58
I came across the same problem in #16777.

IMHO the current behavior is better, and the documentation should be fixed instead, for the following reasons:
 1) it's consistent with assignments, where the RHS is evaluated before the LHS (see also msg128500).  This should also be the behavior of the dict(k=v) syntax, and what I personally expect;
 2) changing it back is not backward-compatible with any code written during the last 10 years;
 3) keeping the current behavior and fixing the docs is simpler than fixing the code to match the docs;

In addition, I would avoid writing code with side-effects in a dict literal, even if the order was documented and guaranteed.  The fact that we don't see many reports about this seems to indicate that people don't write such code, or if they do they rely on the current order.
msg179585 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-01-10 19:21
I am sticking with my opinion from before: the code should be fixed.  It doesn't look like assignment to me.

I am fine with making this a "feature" only fixed in 3.4.  (You can even fix the docs in 3.3 as long as you fix them back for 3.4.)

Minor note for Ezio: dict(k=v, ...) doesn't have this problem because k is just a keyword, not an expression.
msg185438 - (view) Author: Ned Batchelder (nedbat) * Date: 2013-03-28 12:09
Since this is documented in the Python Language Reference, it doesn't make much sense to have it describe one way for 3.3 and another for 3.4, does it?  By definition, doesn't that make this an implementation dependency?  We should update the docs to say that pairs in dicts will be evaluated left-to-right, but that the order of key and value is implementation dependent, since it actually is.
msg185446 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-28 13:23
I strongly agree with Guido that we should fix the code. To me, defined left-to-right evaluation of sub-expressions within an expression is a feature. C's 'implementation defined' is a nuisance.

I do not think we should temporarily change the doc. In other cases where we have only applied a bugfix to a future version, we have not temporarily doc'ed the bug as 'correct'. The change *should* be in 
What's New.
History
Date User Action Args
2013-03-28 13:23:58terry.reedysetmessages: + msg185446
2013-03-28 12:09:01nedbatsetnosy: + nedbat
messages: + msg185438
2013-01-10 19:21:43gvanrossumsetmessages: + msg179585
2012-12-25 14:58:49ezio.melottisetnosy: + gvanrossum, ezio.melotti
type: behavior
messages: + msg178133
2012-12-25 14:41:49ezio.melottilinkissue16777 superseder
2012-11-07 19:18:21terry.reedysetmessages: + msg175120
2012-11-07 15:16:04ncoghlansetpriority: high -> low
nosy: + ncoghlan

versions: + Python 3.4
2012-06-24 20:12:44pitrousetnosy: + benjamin.peterson
2012-06-23 07:15:54terry.reedysetcomponents: + Interpreter Core, - None
2012-01-09 21:24:11sandro.tosisetnosy: + sandro.tosi
2012-01-09 20:02:42amaury.forgeotdarclinkissue13729 superseder
2011-06-12 22:35:21terry.reedysetversions: - Python 3.1
nosy: + terry.reedy

messages: + msg138223

stage: patch review
2011-03-19 21:22:28rhettingersetpriority: normal -> high
nosy: georg.brandl, rhettinger, eric.araujo, r.david.murray, swamiyeswanth, takayuki
2011-03-19 19:09:24skip.montanarosetnosy: - skip.montanaro
2011-02-18 19:19:14eric.araujosetnosy: + eric.araujo
2011-02-13 23:41:53rhettingersetnosy: skip.montanaro, georg.brandl, rhettinger, r.david.murray, swamiyeswanth, takayuki
messages: + msg128530
2011-02-13 21:54:39skip.montanarosetnosy: skip.montanaro, georg.brandl, rhettinger, r.david.murray, swamiyeswanth, takayuki
messages: + msg128522
2011-02-13 21:52:17rhettingersetnosy: skip.montanaro, georg.brandl, rhettinger, r.david.murray, swamiyeswanth, takayuki
messages: + msg128521
2011-02-13 21:51:07rhettingersetnosy: skip.montanaro, georg.brandl, rhettinger, r.david.murray, swamiyeswanth, takayuki
messages: + msg128520
2011-02-13 15:20:54pitrousetnosy: + rhettinger
2011-02-13 15:17:10skip.montanarosetnosy: skip.montanaro, georg.brandl, r.david.murray, swamiyeswanth, takayuki
messages: + msg128507
2011-02-13 14:56:29georg.brandlsetnosy: skip.montanaro, georg.brandl, r.david.murray, swamiyeswanth, takayuki
messages: + msg128502
2011-02-13 14:35:33r.david.murraysetnosy: + r.david.murray
messages: + msg128500
2011-02-13 14:26:54skip.montanarosetnosy: skip.montanaro, georg.brandl, swamiyeswanth, takayuki
messages: + msg128498
2011-02-13 14:13:52georg.brandlsetnosy: skip.montanaro, georg.brandl, swamiyeswanth, takayuki
messages: + msg128494
2011-02-13 13:37:42skip.montanarosetnosy: + georg.brandl
messages: + msg128493
2011-02-13 13:32:38skip.montanarosetfiles: + compile.diff
nosy: skip.montanaro, swamiyeswanth, takayuki
messages: + msg128492
2011-02-13 13:14:57skip.montanarosetnosy: skip.montanaro, swamiyeswanth, takayuki
messages: + msg128490
2011-02-13 13:14:33skip.montanarosetfiles: - compile.diff
nosy: skip.montanaro, swamiyeswanth, takayuki
2011-02-13 13:08:52skip.montanarosetnosy: skip.montanaro, swamiyeswanth, takayuki
messages: + msg128489
2011-02-13 13:04:09skip.montanarosetfiles: + compile.diff
versions: + Python 3.1, Python 3.2, Python 3.3
nosy: + skip.montanaro

messages: + msg128487

keywords: + patch
2011-02-13 12:28:50swamiyeswanthsetnosy: + swamiyeswanth
2011-02-13 09:39:12takayukicreate