classification
Title: add to "looping techniques" tutorial a note about modifying sequence
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Ian, chris.jerdonek, docs@python, georg.brandl, jcea, python-dev, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2012-10-14 07:27 by Ian, last changed 2012-10-16 03:04 by chris.jerdonek. This issue is now closed.

Files
File name Uploaded Description Edit
issue-16225-1-default.patch chris.jerdonek, 2012-10-14 09:06 review
issue-16225-2-default.patch chris.jerdonek, 2012-10-14 21:56 review
issue-16225-3-default.patch chris.jerdonek, 2012-10-14 22:10 review
Messages (18)
msg172855 - (view) Author: Ian Carr-de Avelon (Ian) Date: 2012-10-14 07:27
I'm new to Python and I've hit what appears to me to be a bug, but may be a "feature", so a tutorial bug. 
I tried to loop through the items in a list, test each and remove those which fail the test.

Simplifying to illustrate:
>>> print test
[1, 2, 3, 4, 5]
>>> for item in test:
...     print item
...     test.remove(item)
... 
1
3
5
>>> print test
[2, 4]

Whereas I would expect to see all items looped through and non left in the list.
I have worked with languages where you are explicitly warned that you must not mess with the loop variable, or where the behaviour you will get is explained in detail, so you can use it. Not having anything flagged up in eg 5.6. Looping Techniques on
http://docs.python.org/tutorial/datastructures.html
I assumed that the behaviour would be safe and intuative.
Yours
Ian
msg172856 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 07:36
> I have worked with languages where you are explicitly warned that you must not mess with the loop variable

There is a warning in this part of the tutorial:

"It is not safe to modify the sequence being iterated over in the loop..."

(from http://docs.python.org/dev/tutorial/controlflow.html#for-statements )

But it may be good to add a note to the section you reference as well.
msg172861 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 09:06
Attached is a simple way of addressing this (essentially copying the verbiage and example from the other page).  If we want, we could make the sample code different so that the reader doesn't see the same thing twice.
msg172875 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-14 11:24
It is safe to modify a sequence during iteration if it's size not increased.

>>> words = ['cat', 'window', 'defenestrate']
>>> for i, w in enumerate(words):
...     if len(w) > 6:
...         words[i] = w[:5] + '…'
... 
>>> words
['cat', 'window', 'defen…']
msg172897 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 18:14
> It is safe to modify a sequence during iteration if it's size not increased.

What do you mean by "safe"?  The example given by the original commenter does not increase the size either.  I believe "safe" is meant in the sense of avoiding possibly unexpected behavior.  I can clarify the language if you feel it is ambiguous.
msg172899 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-10-14 18:31
Well, I guess Serhiy meant "neither increase nor decrease".

In the end, the exact behavior will never be clear if you don't state explicitly how it's implemented: a counter that starts at 0 and is increased on __next__ until it's equal to len(list) (as checked at every iteration step).  But in general the advice should be: if you want to insert or remove elements during iteration, iterate over a copy.
msg172906 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 19:38
> But in general the advice should be: if you want to insert or remove elements during iteration, iterate over a copy.

I would expand this to cover changing the list in any way.  I think the point being made is that iteration doesn't implicitly make a copy.  There are cases where modifying the list in place can also yield unexpected results.  For example (naive list reversal):

>>> words = ['cat', 'window', 'defenestrate']
>>> for i, word in enumerate(words):
...     words[-i-1] = word
... 
>>> words
['cat', 'window', 'cat']
msg172926 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 21:56
Attaching revised patch.
msg172930 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 22:10
Reattaching.  I duplicated a variable definition that was defined previously.
msg172932 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-14 22:22
I mean it does not lead to crash, hang, etc. Even growing list during iteration can be safe you move forward faster than list grows or if you known where to stop.

Unexpected behavior for one people can be expected for others.
msg172933 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 22:34
> I mean it does not lead to crash, hang, etc.

I agree.  I removed the word "safe" in the patch I attached to reduce ambiguity.  Regarding unexpected behavior, remember that the tutorial is for beginners/newcomers.
msg172934 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-14 23:00
> I agree.  I removed the word "safe" in the patch I attached to reduce
> ambiguity.

Yes, so much the better.

It will be nice somewhere in deep clarify for experts what happens with list 
iterator if the list changed.

And iterating over modifyed (if you insert/remove keys, but not when you only 
update values) dict or set really yield unexpected results (i.e. they expected 
in them undefinability). This is really not recommended.
msg172935 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-14 23:04
> It will be nice somewhere in deep clarify for experts what happens with list iterator if the list changed.

There is a note somewhat to this effect here:

http://docs.python.org/dev/reference/compound_stmts.html#the-for-statement

"Note There is a subtlety when the sequence is being modified by the loop...."
msg173007 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-10-16 01:24
> > It will be nice somewhere in deep clarify 
> > for experts what happens with list iterator if the list changed.

Resist the urge to over-specify.  Unless the behavior is tested and known to be consistent across all implementations, I'm content with the current docs:

"Note There is a subtlety when the sequence is being modified by the loop...."

Chris's currently patch seems reasonable to me and I don't think anything further is a good idea.

With nearly any data structure in any language, most programmers learn to be cautious about and generally avoid looping over a structure while mutating it.  Adding yet more documentation details won't make the issue go away.

I recommend posting the current patch and closing this issue to be done with it.
msg173008 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-16 01:30
Thanks, Raymond.  I will be doing that later today.
msg173011 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-16 02:51
New changeset 1f1bf6a3abbc by Chris Jerdonek in branch '3.2':
Issue #16225: Add additional note to tutorial about changing sequence while looping.
http://hg.python.org/cpython/rev/1f1bf6a3abbc

New changeset 8cb14494d33c by Chris Jerdonek in branch '3.3':
Issue #16225: Merge from 3.2: Add additional note to tutorial about looping.
http://hg.python.org/cpython/rev/8cb14494d33c

New changeset e0a407d41af5 by Chris Jerdonek in branch 'default':
Issue #16225: Merge from 3.3: Add additional note to tutorial about looping.
http://hg.python.org/cpython/rev/e0a407d41af5
msg173012 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-16 03:02
New changeset dc006b6212e7 by Chris Jerdonek in branch '2.7':
Issue #16225: Backport from 3.2: Add additional note to tutorial about looping.
http://hg.python.org/cpython/rev/dc006b6212e7
msg173013 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-10-16 03:04
Committed.  Thanks for reporting the suggestion, Ian.
History
Date User Action Args
2012-10-16 03:04:55chris.jerdoneksetstatus: open -> closed
resolution: fixed
messages: + msg173013

stage: patch review -> resolved
2012-10-16 03:02:25python-devsetmessages: + msg173012
2012-10-16 02:51:35python-devsetnosy: + python-dev
messages: + msg173011
2012-10-16 01:30:20chris.jerdoneksetmessages: + msg173008
2012-10-16 01:24:42rhettingersetnosy: + rhettinger
messages: + msg173007
2012-10-16 00:24:21chris.jerdoneksettitle: list.remove in for loop -> add to "looping techniques" tutorial a note about modifying sequence
type: enhancement
versions: + Python 3.2, Python 3.3, Python 3.4
2012-10-15 02:43:39jceasetnosy: + jcea
2012-10-14 23:04:25chris.jerdoneksetmessages: + msg172935
2012-10-14 23:00:23serhiy.storchakasetmessages: + msg172934
2012-10-14 22:34:39chris.jerdoneksetmessages: + msg172933
2012-10-14 22:22:13serhiy.storchakasetmessages: + msg172932
2012-10-14 22:10:10chris.jerdoneksetfiles: + issue-16225-3-default.patch

messages: + msg172930
2012-10-14 21:56:28chris.jerdoneksetfiles: + issue-16225-2-default.patch

messages: + msg172926
2012-10-14 19:38:42chris.jerdoneksetmessages: + msg172906
2012-10-14 18:31:14georg.brandlsetnosy: + georg.brandl
messages: + msg172899
2012-10-14 18:14:30chris.jerdoneksetmessages: + msg172897
2012-10-14 11:24:05serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg172875
2012-10-14 09:06:25chris.jerdoneksetfiles: + issue-16225-1-default.patch
keywords: + patch
messages: + msg172861

stage: patch review
2012-10-14 07:36:46chris.jerdoneksetnosy: + docs@python, chris.jerdonek
messages: + msg172856

assignee: docs@python
components: + Documentation
2012-10-14 07:27:28Iancreate