Title: dictkeysobject: Add maximum iteration check for .values() and .items()
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, pitrou, rhettinger, serhiy.storchaka, thomas.perl
Priority: normal Keywords: patch

Created on 2019-03-29 14:16 by thomas.perl, last changed 2019-04-02 09:30 by inada.naoki. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12619 merged thomas.perl, 2019-03-29 14:19
Messages (4)
msg339115 - (view) Author: Thomas Perl (thomas.perl) * Date: 2019-03-29 14:16
On top of issue 36452, I noticed some other corner cases that are still not handled. For one, the patch (Github PR 12596) only handles iterating over keys, but not iterating over values or items:

a = {0: 0}
it = iter(a.values())
print('Length hint:', it.__length_hint__())
print('Length hint:', it.__length_hint__())
del a[0]
a[1] = 99
print('Length hint:', it.__length_hint__())

Replace a.values() there with a.items() -- same issue. Note that PR 12596 fixes the a.keys() case (same as iterating over "a" directly).

Applying the "di->len == 0" check in dictiter_iternextvalue() and dictiter_iternextitem() would fix those two cases above, but would still not fix the following case:

a = {0: 'a', 1: 'b', 2: 'c'}

it = iter(a)
i = next(it)
print('got first:', i)
del a[1]
a[1] = 'd'
i = next(it)
print('got second:', i)
i = next(it)
print('got third:', i)

    i = next(it)
    raise RuntimeError(f'got fourth: {i}')
except StopIteration:
    print('stop iteration')

The reason for this is that the iteration count (3 in this case) isn't modified, but the dict's keys are still changed, and the iteration order is as follows:

got first: 0
got second: 2
got third: 1
stop iteration

Note that the key 1 there is first deleted and then set.

I'll add a Github PR that tries to solve these corner cases too by tracking dict keys modification.
msg339117 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-03-29 14:37
This is a duplicate of issue6017, issue19332 and issue29420.
msg339253 - (view) Author: Thomas Perl (thomas.perl) * Date: 2019-03-31 09:54
Repurposing this as per:
msg339324 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-04-02 09:30
New changeset b8311cf5e5d72f8a8aa688b7da1760d6a74a4d72 by Inada Naoki (Thomas Perl) in branch 'master':
bpo-36473: add maximum iteration check for dict .values() and .items() (GH-12619)
Date User Action Args
2019-04-02 09:30:30inada.naokisetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-04-02 09:30:12inada.naokisetmessages: + msg339324
2019-03-31 09:54:37thomas.perlsetmessages: + msg339253
title: Detect all dictionary changes during iteration -> dictkeysobject: Add maximum iteration check for .values() and .items()
2019-03-29 14:37:11serhiy.storchakasetnosy: + rhettinger, serhiy.storchaka, pitrou
messages: + msg339117
2019-03-29 14:36:08xtreaksetnosy: + inada.naoki
2019-03-29 14:19:55thomas.perlsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12554
2019-03-29 14:16:18thomas.perlcreate