classification
Title: Class __dict__ iteration order changing due to type instance key-sharing
Type: Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, abarry, amaury.forgeotdarc, barry, beazley, benjamin.peterson, ionelmc, ncoghlan, pingebretson, rhettinger, serhiy.storchaka, yselivanov
Priority: normal Keywords:

Created on 2016-01-09 12:56 by ncoghlan, last changed 2020-09-11 23:13 by brett.cannon.

Files
File name Uploaded Description Edit
ns_reordering_bug.py ncoghlan, 2016-01-09 12:56 Reproducer used for hg bisect
Messages (4)
msg257824 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-01-09 12:56
Dave Beazley found some odd behaviour in Python 3.4.1+, where the order of the keys in a class dictionary can be changed by assigning a new value to an existing key: https://gist.github.com/dabeaz/617a5b0542d57e003433

Dave's original reproducer showed a case where iterating over class attributes replacing some of them with new values worked correctly as a class decorator on a normal instance of type, but was unreliable when the same operation was called from a metaclass __new__ or __init__ method.

Further investigation showed that it wasn't the timing of the assignment that mattered, but rather the use of a subclass of type rather than type itself as the metaclass.

Checking between 3.4.0 and 3.4.1 with hg bisect using the simpler attached script as the reproducer identified the enabling of key sharing with subclass instances in #20637 as the apparent culprit.

My current theory is that from 3.3.0 to 3.4.0, keys weren't being shared between instances of type and instances of type subclasses at all, and changing that in 3.4.1 broke a subtle assumption somewhere in type_new.
msg258107 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-01-12 16:28
Is this a bug though?  No guarantees of dict order exists, regardless of whether the dict changes or not, right?  Even if the implementation used to, or in some circumstances still does, appear to preserve iteration order, you shouldn't count on it.
msg258112 - (view) Author: Ionel Cristian Mărieș (ionelmc) Date: 2016-01-12 18:01
As I understood it the issue is not with the order but with the iteration being "unstable" (eg: same key appears multiple times). Yes, the dict is mutated while it's being iterated on, but no keys are added or removed, only values are changed.
msg258113 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2016-01-12 18:02
Indeed. Here is another version of the script, it crashes when I set PYTHONHASHSEED=1 and passes with PYTHONHASHSEED=3::

class Meta(type):
    def __new__(meta, clsname, bases, methods):
        cls = super(Meta, meta).__new__(meta, clsname, bases, methods)
        count = 0
        for name in vars(cls):
            if name.startswith('f_'):
                print('decorate', name)
                count += 1
                setattr(cls, name, getattr(cls, name))
        assert count == 8
        return cls

class Spam2(metaclass=Meta):
    def f_1(self): pass
    def f_2(self): pass
    def f_3(self): pass
    def f_4(self): pass
    def f_5(self): pass
    def f_6(self): pass
    def f_7(self): pass
    def f_8(self): pass
History
Date User Action Args
2020-09-11 23:13:21brett.cannonsetnosy: - brett.cannon
2016-01-12 18:02:50amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg258113
2016-01-12 18:01:05ionelmcsetmessages: + msg258112
2016-01-12 16:28:45barrysetnosy: + barry
messages: + msg258107
2016-01-10 00:16:44abarrysetnosy: + abarry
2016-01-09 22:11:52yselivanovsetnosy: + yselivanov
2016-01-09 20:06:02ionelmcsetnosy: + ionelmc
2016-01-09 18:04:25brett.cannonsetnosy: + beazley
2016-01-09 18:03:36brett.cannonsetnosy: + brett.cannon
2016-01-09 13:01:40serhiy.storchakasetnosy: + rhettinger, benjamin.peterson, Mark.Shannon, pingebretson, serhiy.storchaka
2016-01-09 12:56:11ncoghlancreate