classification
Title: Mutating a list while iterating: clarify the docs
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Stefan Pochmann, docs@python, rhettinger, terry.reedy, tim.peters, xgdomingo
Priority: normal Keywords:

Created on 2018-02-04 19:27 by tim.peters, last changed 2018-02-11 03:02 by tim.peters.

Messages (6)
msg311614 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2018-02-04 19:27
This has come up repeatedly, and the docs should be updated to resolve it:

https://stackoverflow.com/questions/48603998/python-iterating-over-a-list-but-i-want-to-add-to-that-list-while-in-the-loop/48604036#48604036

Seemingly the only relevant documentation is in the reference manual, but it's flawed:

https://docs.python.org/3.7/reference/compound_stmts.html#the-for-statement

- The behavior it's describing is specific to list iterators, but it pretends to apply to "mutable sequences" in general (which may or may not mimic list iterators in relevant respects).

- It's not clear that the "length of the sequence" (list!) is evaluated anew on each iteration (not, e.g., captured once at the start of the `for` loop).

- While it describes things that can go wrong, it doesn't describe the common useful case:  appending to a list during iteration (for example, in a breadth-first search).
msg311700 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2018-02-06 02:08
Some other points to consider:  I don't think this belongs in the `for` statement docs at all.  Instead they should merely _note_ that what happens if a sequence being iterated over is mutated is up to the sequence iterator.

Then, e.g., the `list` docs should spell out the behavior currently (but, as already noted, incompletely) described.  From StackOverflow experience, I may be the only person in the world who knows that list behavior here is documented under the `for` statement - everyone else gives up after not finding a word about it in the `list` docs ;-)

`collections.deque` docs, in contrast, may wish to document the behavior of its iterators.  They in fact work hard to raise an exception if a size-changing mutation occurs during iteration, but that doesn't appear to be documented.  It's worth debating whether it should be.

Are there other mutable sequence types in the standard distribution?
msg311735 - (view) Author: Stefan Pochmann (Stefan Pochmann) Date: 2018-02-06 18:08
Yes, there's also `array.array`.
msg311791 - (view) Author: Stefan Pochmann (Stefan Pochmann) Date: 2018-02-07 15:46
And `bytearray`.
msg311932 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-10 03:31
Replace 'sequence' with 'collection' and I agree.  The for loop code just calls iter() and it.next() until an exception.  The behavior of it.next for builtins should be documented with the builtins.  (I think there is something already about dict iteration in the dict entry.)

I agree we should say that the list iterator checks the internal index against the current .__len__() on each .next call.  I had not thought about the usefullness of doing that for breadth-first search, so yes, mention that.
msg311977 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2018-02-11 03:02
Stefan, yup!  Thank you.  `array.array` and `bytearray` iterators appear to work the same way as `list` iterators here.

Terry, the note in the `for` statement docs was written before there _was_ an iterator protocol.  For example, here are the 1.5.1 docs:

https://docs.python.org/release/1.5.1/ref/ref-9.html#HEADING9-27

Back then, the expression_list had to evaluate to a "string, tuple or list".  Or so the docs said - I don't remember the implementation details.

Regardless, I think we all agree the note is at best out of date ;-)
History
Date User Action Args
2018-02-11 03:02:24tim.peterssetmessages: + msg311977
2018-02-11 02:36:29xgdomingosetnosy: + xgdomingo
2018-02-10 03:31:53terry.reedysetnosy: + terry.reedy

messages: + msg311932
versions: - Python 2.7, Python 3.4, Python 3.6, Python 3.7, Python 3.8
2018-02-07 15:46:00Stefan Pochmannsetmessages: + msg311791
2018-02-06 18:08:56Stefan Pochmannsetnosy: + Stefan Pochmann
messages: + msg311735
2018-02-06 02:08:01tim.peterssetnosy: + rhettinger
messages: + msg311700
2018-02-04 19:27:00tim.peterscreate