This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Regression in memory use of instances due to dictionary ordering
Type: Stage: resolved
Components: Versions:
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, brandtbucher, rhettinger
Priority: normal Keywords: 3.9regression

Created on 2021-07-29 10:28 by Mark.Shannon, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (4)
msg398475 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-07-29 10:28
class C:
    def __init__(self, cond):
        if cond:
            self.a = 1
        self.b = 2

c1 = C(True)
c2 = C(False)

   
In Python 3.5, the dictionary keys are shared
---------------------------------------------

>>> sys.getsizeof(c2)
56
>>> sys.getsizeof(c1.__dict__)
96
>>> sys.getsizeof(c2.__dict__)
96

In Python 3.9, the keys are not shared
--------------------------------------

>>> sys.getsizeof(c2)
48
>>> sys.getsizeof(c1.__dict__)
272
>>> sys.getsizeof(c2.__dict__)
232

This represents an increase of memory use for c1 of 110%. (48+272)/(56+96) == 2.1

With compact object layout (https://github.com/faster-cpython/ideas/issues/69), 
any failure to share keys will result in a tripling of memory use per object.

It is not an uncommon pattern for some attributes to initialized lazily,
which causes also prevents key-sharing.

Not only does it increase memory use, it prevents optimizations that attempt to specialize attribute access for instances of a class, as different instances will have different layouts.

The purpose of compact dicts was to save memory, but in this case we are using a lot more memory by preventing PEP 412 from working.
msg398857 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-08-04 01:55
ISTM this was mostly a net win.  We get compactness and sharing most of the time, and lose sharing only in cases like this where different instances of the same class happen to have different attributes.  Personally, I would not have expected sharing to occur in those cases.
msg398876 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-08-04 09:30
Raymond,
When you say "this was mostly a net win" do you mean the more compact layout or ordering?

The compact layout is obviously a win, and doesn't conflict with sharing. The problem is that ordering conflicts with sharing.

As long as instances all have attributes that are a subset of the shared keys stored on the class, they can share attributes, or least they could in 3.5.
msg399505 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-08-13 08:37
Duplicate of https://bugs.python.org/issue40116
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 88935
2021-08-13 08:37:01Mark.Shannonsetstatus: open -> closed
resolution: duplicate
messages: + msg399505

stage: resolved
2021-08-04 20:40:13brandtbuchersetnosy: + brandtbucher
2021-08-04 09:30:36Mark.Shannonsetmessages: + msg398876
2021-08-04 01:55:06rhettingersetnosy: + rhettinger
messages: + msg398857
2021-07-29 10:28:09Mark.Shannoncreate