classification
Title: __slots__ for subclasses of variable length types
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: arigo Nosy List: BreamoreBoy, ajaksu2, martin.panter, mwh, rhettinger, ronaldoussoren, terry.reedy
Priority: normal Keywords:

Created on 2005-03-30 17:09 by mwh, last changed 2015-03-12 05:25 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
var-sized-slots-1.diff mwh, 2005-03-30 17:09 mwh's patch #1
Messages (12)
msg48094 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2005-03-30 17:09
This is a first, rough cut at allowing subclasses of variable length 
types to have __slots__ of all flavours, not just __dict__.

The motivation is trying to understand and document what's going on 
in typeobject.c, and the less special cases knocking around the 
better.

This patch also allows instances of such classes to be weakly 
referenced.

What is missing: tests, lots of tests, documentation.  Also, the code 
is a bit hackish at various points; a degree of clean up can certainly 
be acheived.

Also, I think my code probably fails to cope with code like:

class A(str):
 pass # implicitly adds __dict__, __weakref__
class B(A):
 __slots__ = ["a", "b"]

b = B()
b.c = 1

Hmm, yes.  Oh well, no time to fix today (I don't think it's that big a 
deal).
msg48095 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2005-04-03 14:11
Logged In: YES 
user_id=4771

I'm confused: the rule for negative slot offsets appear to be different to the one for tp_dictoffset, which only increases the amount of obscurity around here.

tp_dictoffset counts relative to the end of the object, whereas in your patch negative slot offsets are a different trick to mean "relative to the start but skipping the varsized part".  The difference shows up when subclassing increases tp_basicsize.  This should be resolved one way or the other -- and I think that a clear picture of the various parts of the object and how they are measured would be a good start.

That's also related to your proposed change to extra_ivars(), which would become slightly more permissive; I strongly suspect that it would allow more strange segfaulting cases to sneak in undetected...
msg48096 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2005-04-03 14:32
Logged In: YES 
user_id=6656

> I'm confused: the rule for negative slot offsets appear to be
> different to the one for tp_dictoffset

Yes.  I think this is actually necessary.

Consider:

class S(str):
    __slots__ = ['a']

you'd except S.__dict__['a'].__offset__ (well, if the attribute existed) to be 
-4.

Then

class T(S):
    __slots__ = ['b']

then using the 'from the end of the object' rule for  T().a would actually find 
T.b.  (IOW, T.__dict__['b'].__offset__ == T.__dict__['a'].__offset__ == -4).  
The alternative would be to somehow override all the slots in S when T is 
defined -- and this doesn't seem wise.

__dict__ indeed works differently, because 
instance.__class__.__dictoffset__ is updated on subclassing.  You could 
make __dict__ work like the slots mentioned above, but then you'd have to 
find the '__dict__' descriptor every time you wanted to access an 
instance's dictionary, and that would be slow (and might even not really 
work, but I don't want to risk brain-explosion by thinking about it too hard)

> which only increases the amount of obscurity around here.

Yeah, sorry about that.

I think something I've realised over the past few days is that __dict__ 
really is special.  I'm not sure __weakref__ is (though I guess it's special in 
that you want to be able to access it from C without any risk of executing 
Python level code, i.e. replacing Py_GETWEAKREFLIST(ob) with 
PyOjbect_GetAttrString(ob, "__weakref__") would be unfortunate).

> This should be resolved one way or the other 

See above -- don't think you can.

> -- and I think that
> a clear picture of the various parts of the object and how they
> are measured would be a good start.

No kidding here!

> That's also related to your proposed change to extra_ivars(),
> which would become slightly more permissive; I strongly suspect
> that it would allow more strange segfaulting cases to sneak in
> undetected...

Almost certainly!
msg48097 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2005-04-03 15:27
Logged In: YES 
user_id=4771

I think it's still possible to give slot.offset the same meaning as tp_dictoffset, even given the additional constrain that it can't change upon subclassing.  In your example classes S and T, we can put 'b' before 'a' in memory, so that a.offset==-4 (for both S and T) and b.offset==-8.
msg48098 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2005-04-03 16:19
Logged In: YES 
user_id=6656

Heh, yes that works, and completely hadn't occurred to me.
msg82173 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-02-15 22:15
Patch has tests.
msg110432 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-16 12:44
How much rework if any is needed to get this patch into py3k?
msg113454 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-09 18:55
I believe this is covered by the PEP3003 3.2 change moratorium.
msg113615 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-08-11 19:16
ISTM the space saving of value of __slots__ isn't typically needed in the context of variable length built-in types.

Guido has long regarded __slots__ as a confusing hack.  That should warn us away for extending its functionality.

Unless there are some compelling use cases, a simple and clean patch, and a clear one sentence explanation for users, I recommend this feature request be closed.

Michael, do you still want this?
msg113630 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2010-08-11 21:58
Well, I can think of some counters to that -- surely it's _more_ confusing if slots only works some of the time? -- but realistically I'm not going to work on this any further.
msg113631 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-08-11 22:03
Declaring YAGNI and closing.
Thanks Michael.
msg184641 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-03-19 15:41
I do have a usecase for this: subclasses of int. 

Having slots would be nice for a reasonably efficient implementation of named constants (as recently discussed on python-ideas), and I'm already using a subclass of int of PyObjC to attach a single other value to a Python integer. In both cases the overhead of the __dict__ is pretty large.
History
Date User Action Args
2015-03-12 05:25:41martin.pantersetnosy: + martin.panter
2013-03-19 16:20:16arigosetnosy: - arigo
2013-03-19 15:41:01ronaldoussorensetnosy: + ronaldoussoren

messages: + msg184641
versions: + Python 3.4, - Python 3.3
2013-03-19 15:31:32ronaldoussorenlinkissue17295 superseder
2011-02-16 07:32:11orsenthilsetkeywords: - after moratorium
nosy: mwh, arigo, rhettinger, terry.reedy, ajaksu2, BreamoreBoy
resolution: wont fix
2010-08-11 22:03:01rhettingersetstatus: open -> closed

messages: + msg113631
2010-08-11 21:58:37mwhsetmessages: + msg113630
2010-08-11 19:16:36rhettingersetnosy: + rhettinger
messages: + msg113615
2010-08-09 18:55:22terry.reedysetversions: + Python 3.3, - Python 3.2
nosy: + terry.reedy

messages: + msg113454

keywords: + after moratorium, - patch
2010-07-16 12:44:54BreamoreBoysetnosy: + BreamoreBoy

messages: + msg110432
versions: + Python 3.2, - Python 2.7
2009-02-15 22:15:31ajaksu2settype: enhancement
stage: patch review
messages: + msg82173
nosy: + ajaksu2
versions: + Python 2.7, - Python 2.5
2005-03-30 17:09:14mwhcreate