classification
Title: Docs missing the behavior of += (in-place add) for lists.
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.4, Python 3.3, Python 3.2, Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: benrg, docs@python, ezio.melotti, montysinngh, r.david.murray
Priority: normal Keywords:

Created on 2012-12-16 20:41 by montysinngh, last changed 2013-01-24 18:40 by benrg.

Messages (9)
msg177627 - (view) Author: Ashwini Chaudhary (montysinngh) Date: 2012-12-16 20:41
I think the python docs are missing the behavior of += for lists. It actually calls list.extend() but can't find that anywhere in docs expect in source code, http://hg.python.org/cpython/file/2d2d4807a3ed/Objects/listobject.c#l892.
msg177631 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-12-16 21:44
Well, it is effectively documented by the text here:

   http://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-statements

since "a + b" is logically equivalent to a.extend(b) when a is being updated "in-place".  The fact that it is in fact implemented using extend is an implementation detail.

That said, it would be logical to add an entry for the augmented assignment to the table here:

   http://docs.python.org/3/library/stdtypes.html#mutable-sequence-types

There also may be other places in that chapter where augmented assignment deserves mention.
msg180501 - (view) Author: (benrg) Date: 2013-01-24 01:27
This is bizarre:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> x = y = [1, 2]
>>> x += [3]
>>> y
[1, 2, 3]
>>> x = y = {1, 2}
>>> x -= {2}
>>> y
{1}
>>>

Since when has this been standard behavior? The documentation says:

"An augmented assignment expression like x += 1 can be rewritten as x = x + 1 to achieve a similar, but not exactly equal effect. In the augmented version, x is only evaluated once. Also, when possible, the actual operation is performed in-place, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead."

What is "when possible" supposed to mean here? I always thought it meant "when there are known to be no other references to the object". If op= is always destructive on lists and sets, then "where possible" needs to be changed to "always" and a prominent warning added, like "WARNING: X OP= EXPR DOES NOT BEHAVE EVEN REMOTELY LIKE X = X OP EXPR IN PYTHON WHEN X IS A MUTABLE OBJECT, IN STARK CONTRAST TO EVERY OTHER LANGUAGE WITH A SIMILAR SYNTAX."
msg180504 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-24 03:37
> What is "when possible" supposed to mean here?

Generally it means "when the object is mutable":
>>> l = [1,2,3]
>>> id(l)
3074713484
>>> l += [4]
>>> id(l)
3074713484
>>> t = (1,2,3)
>>> id(t)
3074704004
>>> t += (4,)
>>> id(t)
3075304860

Tuples are not mutable, so it's not possible to modify them in place, and a new tuple needs to be created.
Note that while most mutable objects in the stdlib that support += do indeed modify the object rather than creating a new one, I don't think this is strictly required.
IOW that paragraph is already warning you that (with mutable objects) the object might be reused, depending on the implementation.  Maybe this should be clarified?
(IIRC in CPython it could be possible that in some situations an immutable object still has the same id after an augmented assignment, but, if it really happens, it is an implementation detail and shouldn't affect semantics.)
msg180507 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-24 04:10
If you really want to freak out, try this:

>>> x = ([],)
>>> x[0] += [1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> x
([1],)

but to answer your question, it has *always* worked that way, from the time augmented assignment was introduced (see, eg, issue 1306777, which was reported against python 2.4).  

Remember, Python names refer to pointers to objects, they are not variables in the sense that other languages have variables.

Guido resisted augmented assignment for a long time.  These confusions speak to why.

As far as I know Ezio is correct, "when possible" means "when the target is mutable".  The documentation should probably be clarified on that point.  I'm not sure it is practical to let whether or not the target is mutated be an implementation detail. IMO the behavior must be clearly defined for each type that is built in to Python.
msg180508 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-24 04:59
To clarify, with "depends on the implementation" I meant the way a particular class is implemented (i.e. a class might decide to return a new object even if it's mutable).
The behavior of built-in types is well defined and should be the same across all the Python implementations.
Regarding the comment about immutable types, it's something specific to CPython (I don't remember the specific details though, so I might be wrong), and somewhat similar to:
>>> 'a'*20 is 'a'*20
True
>>> 'a'*25 is 'a'*25
False
This shouldn't be a problem though, so if you e.g. do "x = y = immutableobj;  y += 1", 'x' should never be affected.
msg180510 - (view) Author: (benrg) Date: 2013-01-24 05:18
> As far as I know Ezio is correct, "when possible" means "when the target is mutable".  The documentation should probably be clarified on that point.

Yes, it needs to be made very, very clear in the documentation. As I said, I'm not aware of any other language in which var op= expr does not mean the same thing as var = var op expr. I'm actually amazed that neither of you recognize the weirdness of this behavior (and even more amazed that GvR apparently didn't). I'm an experienced professional programmer, and I dutifully read the official documentation cover to cover when I started programming in Python, and I interpreted this paragraph wrongly, because I interpreted it in the only way that made sense given the meaning of these operators in every other language that has them. Python is designed to be unsurprising; constructs generally mean what it looks like they mean. You need to explain this unique feature of Python in terms so clear that it can't possibly be mistaken for the behavior of all of the other languages.

> Remember, Python names refer to pointers to objects, they are not variables in the sense that other languages have variables.

That has nothing to do with this. Yes, in Python (and Java and Javascript and many other languages) all objects live on the heap, local variables are not first-class objects, and var = expr is a special form. That doesn't change the fact that in all of those other languages, var += expr means var = var + expr. In C++ local variables are first-class objects and var += expr means var.operator+=(expr) or operator+=(var, expr), and this normally modifies the thing on the left in a way that's visible through references. But in C++, var = var + expr also modifies the thing on the left, in the same way.

In Python and Java and Javascript and ..., var = value never visibly mutates any heap object, and neither does var = var + value (in any library that defines a sane + operator), and therefore neither should var += value (again, in any sanely designed library). And it doesn't. Except in Python.
msg180511 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-24 06:22
> Python is designed to be unsurprising; constructs generally mean
> what it looks like they mean.

AFAIK in C "x += 1" is equivalent to "x++", and both are semantically more about incrementing (mutating) the value of x than about creating a new value that gets assigned to x.
Likewise it seems to me more natural to interpret "x += y" as "add the value of y to the object x" than "add x and y together and save the result in x".
Clearly if you are used to other languages with different semantics you might expect a different behavior, but you could say the same about the fact that int/int gives float on Python 3: it's surprising if you are used to other languages like C, but otherwise it's more natural.

> I interpreted this paragraph wrongly, because I interpreted it in the
> only way that made sense given the meaning of these operators in 
> every other language that has them.

It seems to me that the documentation doesn't leave much room for interpretation regarding the fact that the object is mutated in place; the only problem is that it doesn't specify clearly what are the objects that do this.
msg180541 - (view) Author: (benrg) Date: 2013-01-24 18:40
> AFAIK in C "x += 1" is equivalent to "x++", and both are semantically
> more about incrementing (mutating) the value of x than about creating a
> new value that gets assigned to x. Likewise it seems to me more natural
> to interpret "x += y" as "add the value of y to the object x" than "add
> x and y together and save the result in x".

Look, it's very simple: in C, ++x and x += 1 and x = x + 1 all mean the same thing. You can argue about how to describe the thing that they do, but there's only one thing to describe. Likewise, in every other language that borrows the op= syntax from C, it is a shorthand for the expanded version with the bare operator. As far as I know, Python is the only exception. If you know of another exception please say so.


> >>> x = ([],)
> >>> x[0] += [1]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: 'tuple' object does not support item assignment
> >>> x
> ([1],)

I actually knew about this. It's an understandably difficult corner case, since the exception is raised after __iadd__ returns, so there's no chance for it to roll back its changes.

At least, I thought it was a difficult corner case back when I thought the in-place update was a mere optimization. But if += really means .extend() on lists, this should not raise an exception at all. In fact there's no sense in having __iadd__ return a value that gets assigned anywhere, since mutable objects always mutate and return themselves and immutable objects don't define __iadd__. It looks like the interface was designed with the standard semantics in mind but the implementation did something different, leaving a vestigial assignment that's always a no-op. What a disaster.
History
Date User Action Args
2013-01-24 18:40:16benrgsetmessages: + msg180541
2013-01-24 06:22:22ezio.melottisetmessages: + msg180511
2013-01-24 05:18:42benrgsetmessages: + msg180510
2013-01-24 04:59:57ezio.melottisetmessages: + msg180508
2013-01-24 04:10:02r.david.murraysetmessages: + msg180507
2013-01-24 03:37:15ezio.melottisetmessages: + msg180504
2013-01-24 01:27:29benrgsetnosy: + benrg
messages: + msg180501
2013-01-02 18:23:42ezio.melottisetnosy: + ezio.melotti

stage: needs patch
2012-12-16 21:44:54r.david.murraysetnosy: + r.david.murray

messages: + msg177631
versions: - Python 3.1, Python 3.5
2012-12-16 20:41:13montysinnghcreate