Title: Dict fails to notice addition and deletion of keys during iteration
Type: behavior Stage:
Components: Documentation Versions: Python 2.6, Python 2.5
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: benjamin.peterson, georg.brandl, r.david.murray, steven.daprano, terry.reedy
Priority: normal Keywords:

Created on 2009-05-14 05:43 by steven.daprano, last changed 2009-05-17 08:24 by georg.brandl. This issue is now closed.

Messages (10)
msg87729 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2009-05-14 05:42
I'm not sure if this is a documentation bug or a behaviour bug, or 
possibly both.

The documentation warns about adding or deleting items from a dict 
while iterating over it:

"Using iteritems() while adding or deleting entries in the dictionary 
will raise a RuntimeError."

Same for other dict iterators.

However, you can add and delete items, so long as the overall size of 
the dict doesn't change. Consequently, some modifications to the dict 
aren't caught, leading to various misbehaviour in (at least) Python 
2.5 and 2.6.

Some dicts appear to "run too long":

>>> d = dict(x=3, y=4)  # Two items
>>> it = d.iteritems()
>>>  # One
('y', 4)
>>> del d['y']
>>> d['z'] = 5
>>>  # Two
('x', 3)
>>>  # Three
('z', 5)

While others run too short:

>>> d = {-1: 'aa', -2: 'bb'}  # Two items
>>> it = d.iteritems()
>>>  # One
(-2, 'bb')
>>> del d[-1]
>>> d[0] = 'cc'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
msg87735 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-05-14 11:50
This is correct; dict iterators check to see if the size of the dict is
different. However, fixing this problem would require tracking dict
contents during iteration. That strikes me as too inefficient and too
much code for this little case.
msg87855 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2009-05-16 04:20
The OP reported a real mismatch between doc and behavior.  If the
behavior is not changed, I think the doc should be.  Other implementors,
reading the doc, might think that they do have to write code to track
changes.  From the doc, I thought that CPython did that.

So I suggest changing reopening and changing the doc to say
"Changing the net size of the dictionary while using iteritems() will
raise a RuntimeError."  Same for iterkeys() and itervalues()[sp?]

Or remove the warning, as happened in the Py3 changeover to views, or
was that a mistake?
msg87859 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2009-05-16 07:13
I agree with Terry Reedy. I'm re-opening it as a documentation bug (if 
I can -- if I can't, I'll just request somebody who can do so).
msg87875 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-05-16 10:42
I don't think it would be better to change the documentation to "will
raise a RuntimeError or result in undefined behavior."  It already tells
you not to do this strongly enough.
msg87893 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-05-16 13:52
The precisionist in me insists that at a minimum 'will' should be changed
to 'may'.  Otherwise either the docs are lying or the implementation
has a bug.

Or perhaps we could add a footnote about the intentionally divergent
behavior of the CPython implementation?

(As Terry pointed out these docs may be used by other implementors as a
prescriptive guide, just as the language reference is.)
msg87895 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2009-05-16 14:02
With respect Georg, given that the behaviour won't be changed, the 
documentation is simply *wrong*. It's not a matter of telling 
people "don't do this" -- somebody, somewhere, is going to rely on the 
documented behaviour. The docs make the clear promise that, and I 
quote, "Using iteritems() while adding or deleting entries in the 
dictionary will raise a RuntimeError", but that's not what happens. 
The actual behaviour is undefined.
msg87899 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-05-16 15:18
I wouldn't call it *wrong* as much as *not the whole truth*.  It is true
that if just one key is added or removed, a RuntimeError will be raised.
 There are probably lots of places in our docs where the whole truth
isn't told, but in a way that works if you do the sensible thing.

In this case I can neither see anyone relying on RuntimeError being
raised (except for dict's own test suite), nor an implementor looking
only at these docs, not the source, to implement one of Python's most
crucial object types.

Anyway, if you find a new wording that isn't too clumsy, I'll change it.
msg87943 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2009-05-16 23:28
I would be happy enough to change 'will' to 'may'.  There are a lot of
undocumented exception-raising circumstances, and this one already be
undocumented in Py3.
msg87964 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-05-17 08:24
OK, I now changed it to "may raise ... or fail to iterate over all
entries" in r72708.
Date User Action Args
2009-05-17 08:24:54georg.brandlsetmessages: + msg87964
2009-05-16 23:28:26terry.reedysetmessages: + msg87943
2009-05-16 15:18:16georg.brandlsetmessages: + msg87899
2009-05-16 14:02:40steven.dapranosetmessages: + msg87895
2009-05-16 13:52:06r.david.murraysetnosy: + r.david.murray
messages: + msg87893
2009-05-16 10:42:56georg.brandlsetstatus: open -> closed

messages: + msg87875
2009-05-16 07:13:57steven.dapranosetstatus: closed -> open

messages: + msg87859
components: - Interpreter Core
nosy: georg.brandl, terry.reedy, benjamin.peterson, steven.daprano
2009-05-16 04:20:02terry.reedysetnosy: + terry.reedy
messages: + msg87855
2009-05-14 11:50:03benjamin.petersonsetstatus: open -> closed

nosy: + benjamin.peterson
messages: + msg87735

resolution: wont fix
2009-05-14 05:43:58steven.dapranocreate