classification
Title: nested list produced with multiplication is linked to the same list
Type: Stage:
Components: Interpreter Core Versions: Python 3.5, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Matthew Tanous, abarry, benjamin.peterson
Priority: normal Keywords:

Created on 2016-05-27 05:25 by Matthew Tanous, last changed 2016-05-27 20:59 by abarry. This issue is now closed.

Messages (6)
msg266475 - (view) Author: Matthew Tanous (Matthew Tanous) Date: 2016-05-27 05:25
If I produce a list in this fashion:

l = [[x] * n] * n

I would expect that I would obtain a matrix-like structure.  Instead, I get a list of the *same* list, such that the statement:

l[x][y] = z

would change, in essence, every value in "column" y.  This is different from the case where the list contains a string or integer value, where the new list points to separate instances of the internal values.

In my view, this is strange and counter-intuitive behavior.  If I want to create a matrix-like set to None to start, instead of using:

 mat = [[None] * N] * N

I have to use:

 mat = [[None] * N for i in range(N)]

If there is any possibility for it to be changed, I think that would improve things considerably.  In almost no cases, in my opinion, would I want a list of lists that are forced to be the same.
msg266477 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-05-27 05:42
That's just how Python works. https://docs.python.org/3/faq/programming.html#how-do-i-create-a-multidimensional-list
msg266482 - (view) Author: Matthew Tanous (Matthew Tanous) Date: 2016-05-27 06:57
I'm aware that it's how Python works at present. My point was that this is awkward and counter-intuitive, with no purpose I can see worth serving.

"That's just how it works" seems to me a rather insufficient answer, especially for such a seemingly trivial change.

So, no, it's not technically a bug. But I think it's worth some discussion.

> On May 26, 2016, at 11:42 PM, Benjamin Peterson <report@bugs.python.org> wrote:
> 
> 
> Benjamin Peterson added the comment:
> 
> That's just how Python works. https://docs.python.org/3/faq/programming.html#how-do-i-create-a-multidimensional-list
> 
> ----------
> nosy: +benjamin.peterson
> resolution:  -> not a bug
> status: open -> closed
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27135>
> _______________________________________
msg266491 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-05-27 12:30
The reason why Python behaves like that in this case is a side-effect of the way the data model is implemented. Lists are a mutable data type, which means that referencing the same list - which is what you are doing - as opposed to creating a new one each time, does exactly what is expected of the data model, which is to keep a reference to the original list. Consider:

x = ??
y = x

The object bound to `y` refers to the same object as the one bound to `x`, such that `x is y`, for every possible value of `x`. As `=` is the assignment operator (and not the copy operator, is such a thing existed), `y` merely has a reference to `x`.

There are very valid use cases where you will want to get a reference to the original list to modify it, for example in a function that is meant to alter the original. If you want a copy, use `l[:]` (or `l.copy()` in more recent versions).

In Python, every object and operation is evaluated once, when the interpreter comes across it (in a for or while loop, it will come over the same point multiple times, so it will evaluate it multiple times). By doing:

[[x] * y] * n

You are essentially doing this:

l_1 = [x] # one-element list
l_2 = l_1 * y # list with `y` times `x` in it (references to `x`, not copies)
l_3 = [l_2] # a new list with one element, `l_2`
l_4 = l_3 * n # list with `n` times `l_3` in it (references to `l_3`, not copies)

As you can see, it makes no sense to have multiple different lists by doing that. Doing a list comprehension, such as:

[[None] * n for _ in range(N)]

Will evaluate a new list each time the interpreter comes across it, which will be N times for this case. Thus, you get a different list everytime.

In other words, the reason that "That's just how it works" is because this behaviour is a side-effect, not something that's out of the way. Creating a new special case for this particular case seems like a very big stretch. Remember, "Special cases aren't special enough to break the rules."

This change that is "seemingly trivial" to you is actually asking to break the Python semantics in some weird and convoluted way, for no gain since we have list comprehensions (that you seem to already be acquainted with).

Did that seem like a sufficient answer? ;-)
msg266510 - (view) Author: Matthew Tanous (Matthew Tanous) Date: 2016-05-27 20:12
It makes sense, except for this case is not true when "x" is an immutable data type - it appears as though something like [5] * n creates a list of totally separate elements (even if they start as the same thing).

It is difficult to see a case where I would not want separate elements from a mutable type as well. It would seem to me that having the "*" operator function as the list comprehension does now would be more logical and straightforward.

This would change the default list in a way that is mostly unseen: [5] * 3 would still result in [5, 5, 5] while treating mutable nesting in a way that I believe would be more understandable from the apparent intent of the syntax in question.

This would also resolve the problem of being unable to use the list multiplication to create a list of mutable objects. If I have a mutable object Foo and wish to create a list of different instances of them, I cannot use the straightforward syntax [Foo()] * n at present to do this, as there will only be one actual Foo instance in that list. I think that this is confusing to developers who would generally expect that this shortcut (and this is the only purpose of the operator for lists) would create a list of different instances.

> On May 27, 2016, at 6:30 AM, Emanuel Barry <report@bugs.python.org> wrote:
> 
> 
> Emanuel Barry added the comment:
> 
> The reason why Python behaves like that in this case is a side-effect of the way the data model is implemented. Lists are a mutable data type, which means that referencing the same list - which is what you are doing - as opposed to creating a new one each time, does exactly what is expected of the data model, which is to keep a reference to the original list. Consider:
> 
> x = ??
> y = x
> 
> The object bound to `y` refers to the same object as the one bound to `x`, such that `x is y`, for every possible value of `x`. As `=` is the assignment operator (and not the copy operator, is such a thing existed), `y` merely has a reference to `x`.
> 
> There are very valid use cases where you will want to get a reference to the original list to modify it, for example in a function that is meant to alter the original. If you want a copy, use `l[:]` (or `l.copy()` in more recent versions).
> 
> In Python, every object and operation is evaluated once, when the interpreter comes across it (in a for or while loop, it will come over the same point multiple times, so it will evaluate it multiple times). By doing:
> 
> [[x] * y] * n
> 
> You are essentially doing this:
> 
> l_1 = [x] # one-element list
> l_2 = l_1 * y # list with `y` times `x` in it (references to `x`, not copies)
> l_3 = [l_2] # a new list with one element, `l_2`
> l_4 = l_3 * n # list with `n` times `l_3` in it (references to `l_3`, not copies)
> 
> As you can see, it makes no sense to have multiple different lists by doing that. Doing a list comprehension, such as:
> 
> [[None] * n for _ in range(N)]
> 
> Will evaluate a new list each time the interpreter comes across it, which will be N times for this case. Thus, you get a different list everytime.
> 
> In other words, the reason that "That's just how it works" is because this behaviour is a side-effect, not something that's out of the way. Creating a new special case for this particular case seems like a very big stretch. Remember, "Special cases aren't special enough to break the rules."
> 
> This change that is "seemingly trivial" to you is actually asking to break the Python semantics in some weird and convoluted way, for no gain since we have list comprehensions (that you seem to already be acquainted with).
> 
> Did that seem like a sufficient answer? ;-)
> 
> ----------
> nosy: +ebarry
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27135>
> _______________________________________
msg266513 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-05-27 20:59
You do raise a valid point, sequence repetition (*) on lists of mutable objects serves very little purpose. However, as I stated, it is a side-effect of the data model and changing it would be yet another special case, which aren't special enough to break the rules, and this applies here.

Keep in mind that, currently, this is only confusing to newbies. Your idea would make this confusing for long-time Python programmers.

If you still wish to propose such a change, I recommend to take this to the Python-ideas mailing list (point to this issue in your email). Be warned though that most core developers (and most non-core devs such as myself) will probably prefer the statu quo, again because the change would be yet another special case, and probably view this as an attractive nuisance ("Why does [[x] * y] * n create separate lists but x = y = [] refer to the same list?").
History
Date User Action Args
2016-05-27 20:59:09abarrysetmessages: + msg266513
2016-05-27 20:12:18Matthew Tanoussetmessages: + msg266510
2016-05-27 12:30:52abarrysetnosy: + abarry
messages: + msg266491
2016-05-27 06:57:59Matthew Tanoussetmessages: + msg266482
2016-05-27 05:42:04benjamin.petersonsetstatus: open -> closed

nosy: + benjamin.peterson
messages: + msg266477

resolution: not a bug
2016-05-27 05:25:00Matthew Tanouscreate