classification
Title: Asyncio classes missing __slots__
Type: Stage:
Components: asyncio Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Bluenix2, andrei.avk, asvetlov, josh.r, serhiy.storchaka, yselivanov
Priority: normal Keywords:

Created on 2021-06-05 14:58 by Bluenix2, last changed 2021-06-18 04:55 by andrei.avk.

Messages (10)
msg395165 - (view) Author: Bluenix (Bluenix2) Date: 2021-06-05 14:58
Most of asyncio's classes are missing a __slots__ attribute - for example Lock, Event, Condition, Semaphore, BoundedSemaphore all from locks.py; or Future, FlowControlMixin, Queue, BaseSubprocessTransport from various other parts of asyncio.
msg395166 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-06-05 15:15
What is the problem with this?

Setting __slots__ is needed if we want to save memory for creating a lot of instances. It can also be used for preventing adding arbitrary attributes and/or making weak references. Setting __slots__ in base class is required if you want to get a benefit of setting __slots__ in any of subclasses.
msg395170 - (view) Author: Bluenix (Bluenix2) Date: 2021-06-05 15:55
> What is the problem with this?

The problem is that asyncio *is not* defining __slots__.

> Setting __slots__ in base class is required if you want to get a benefit of setting __slots__ in any of subclasses.

That is my use-case for this.
msg395171 - (view) Author: wyz23x2 (wyz23x2) * Date: 2021-06-05 16:21
OK, so:
>>> (1).__slots__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__slots__'
>>> 4.5.__slots__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'float' object has no attribute '__slots__'
>>> complex(5, 2).__slots__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'complex' object has no attribute '__slots__'
>>> 'Hello'.__slots__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__slots__'
>>> b'50'.__slots__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'bytes' object has no attribute '__slots__'
>>> [2.72, 3.14].__slots__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute '__slots__'
>>> 

Many many more.
So these *all* need __slots__???
That a major change into Python 5000.
msg395176 - (view) Author: Bluenix (Bluenix2) Date: 2021-06-05 18:22
>>> (1).__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__dict__'
>>> 4.5.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'float' object has no attribute '__dict__'
>>> 'Hello'.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__dict__'
>>> b'50'.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'bytes' object has no attribute '__dict__'
>>> [2.72, 3.14].__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute '__dict__'


> __slots__ allow us to explicitly declare data members (like properties) and deny the creation of __dict__ and __weakref__ (unless explicitly declared in __slots__ or available in a parent.)

From https://docs.python.org/3/reference/datamodel.html

They don't have __slots__, nor a __dict__ or __weakref__:

>>> (1).__weakref__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__weakref__'
>>> 4.5.__weakref__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'float' object has no attribute '__weakref__'
>>> 'Hello'.__weakref__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__weakref__'
>>> b'50'.__weakref__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'bytes' object has no attribute '__weakref__'
>>> [2.72, 3.14].__weakref__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute '__weakref__'

They're essentially C extension classes: https://docs.python.org/3/extending/newtypes_tutorial.html

I am not sure what this argument was meant to prove.


What would be the downside to adding __slots__ to asyncio's classes? Other than that someone can no longer arbitrarily add attributes?
msg395969 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-06-17 03:05
Bluenix, can you describe your use case in more detail?
msg395990 - (view) Author: Bluenix (Bluenix2) Date: 2021-06-17 10:37
My exact use-case is that I am subclassing asyncio.Semaphore to change some functionality (override `release()` to do nothing and set up tasks to schedule calls to reset the counter). I am expecting *a lot* of these instances so (like Serhiy Storchaka nicely put it) I would like to reduce the memory footprint of these classes by using __slots__. The issue now becomes that asyncio.Semaphore (like most other asyncio classes) have not defined __slots__, this prohibits my subclass from taking advantage of __slots__.
msg396030 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-06-18 03:46
The size of an instance of Semaphore is 48, of an empty tuple is 40, of a small int is 28, of an instance of a normal class with a single slot in __slots__ is also 40, are you use 48 byte size will really be an issue for you?
msg396031 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2021-06-18 03:57
Andrei: The size of an instance of Semaphore is 48 bytes + 104 more bytes for the __dict__ containing its three attributes (ignoring the cost of the attributes themselves). A slotted class with three attributes only needs 56 bytes of overhead per-instance (it has no __dict__, so the 56 is the total cost). Dropping overhead of the instances by >60% can make a difference if you're really making many thousands of them.

Personally, I think Python level classes should generally default to using __slots__ unless the classes are explicitly not for subclassing; not using __slots__ means all subclasses have their hands tied by the decision of the parent class. Perhaps explicitly opting in to __weakref__ (which __slots__ removes by default) to allow weak referencing, but it's fairly rare a class *needs* to otherwise allow the creation of arbitrary attributes.
msg396033 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-06-18 04:55
My mistake, I forgot the size of the dict itself is not included in getsizeof().
History
Date User Action Args
2021-06-18 04:55:40andrei.avksetmessages: + msg396033
2021-06-18 03:57:38josh.rsetnosy: + josh.r
messages: + msg396031
2021-06-18 03:46:53andrei.avksetmessages: + msg396030
2021-06-17 10:37:21Bluenix2setmessages: + msg395990
2021-06-17 03:05:17andrei.avksetnosy: + andrei.avk
messages: + msg395969
2021-06-05 18:22:40Bluenix2setmessages: + msg395176
2021-06-05 16:23:16wyz23x2setnosy: - wyz23x2
2021-06-05 16:21:38wyz23x2setnosy: + wyz23x2
messages: + msg395171
2021-06-05 15:55:41Bluenix2setmessages: + msg395170
2021-06-05 15:15:26serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg395166
2021-06-05 14:58:52Bluenix2create