Issue38262
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-09-24 10:11 by acue, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
tests.tar.gz | acue, 2019-09-24 10:11 | call script and 4 test cases | ||
mixin_tests.py | steven.daprano, 2019-09-24 12:34 | |||
mixin_tests2.py | steven.daprano, 2019-09-24 13:07 | |||
mixin_C_is_B0A1.py | acue, 2019-09-24 20:54 | |||
mixin_C_is_B0A1.py | acue, 2019-09-25 03:45 |
Messages (10) | |||
---|---|---|---|
msg353062 - (view) | Author: Arno-Can Uestuensoez (acue) | Date: 2019-09-24 10:11 | |
The attached examples evaluate inheritance by mixins with different arameters and in different orders. The 'super()' call propagates the call resolution by definition to the next level, and cares about multiple inheritance as well as mixins. Where mixins are the superior case as they represent multiple inheritance in any case due to the common top class 'object'. So far perfect. The oddity I came around is the simple case of mixins with different parameters, where both are derived from 'object'. It is obvious, that such a class has to comply to the parents call interface - here object - when to be used standalone. BUT when I ignore this and set intentionally a parameter - for test-purposes of mixins only - the resolution algorithm seems to normalize both for the final call without parameters - as soon as one of the mixins is a non-parameter call. This is independent from the inheritance order for the derived class. I expect for the rightmost of the inherited classes to pass the parameters literally. When both classes have parameters, the parameter is passed to 'object' and correctly raises an exception. #---------------------------------------------------------- [acue@lap001 mixin-order-with-different-params]$ ./call.sh ************************ different parameters for mixin classes at diffeent positions 0: super() method signature has no param 1: super() method signature has param final top-class is 'object()' which does not know the parameter - here 'propagate' ************************ #--------------- # # CALL: mixin_C_is_A1B0.py # B:C A:C C:C #--------------- # # CALL: mixin_C_is_A1B1.py # Traceback (most recent call last): File "mixin_C_is_A1B1.py", line 24, in <module> c=C(True) File "mixin_C_is_A1B1.py", line 20, in __init__ super(C, self).__init__(propagate) File "mixin_C_is_A1B1.py", line 4, in __init__ super(A, self).__init__(propagate) File "mixin_C_is_A1B1.py", line 12, in __init__ super(B, self).__init__(propagate) TypeError: object.__init__() takes no parameters #--------------- # # CALL: mixin_C_is_B0A1.py # A:C B:C C:C #--------------- # # CALL: mixin_C_is_B1A1.py # Traceback (most recent call last): File "mixin_C_is_B1A1.py", line 25, in <module> c=C(True) File "mixin_C_is_B1A1.py", line 20, in __init__ super(C, self).__init__(propagate) File "mixin_C_is_B1A1.py", line 12, in __init__ super(B, self).__init__(propagate) File "mixin_C_is_B1A1.py", line 4, in __init__ super(A, self).__init__(propagate) TypeError: object.__init__() takes no parameters [acue@lap001 mixin-order-with-different-params]$ #---------------------------------------------- Is this intentional? If so, what is the explanation? - Because it potentially hides bugs for later debugging... |
|||
msg353075 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2019-09-24 11:42 | |
Are your tests so huge that you need to gzip them before uploading? That makes them extremely difficult to read from the browser, and makes me extremely suspicious that this could be some form of zip bomb or other malware. Couldn't you upload a single .py file instead a shell script? You say that super() "cares about multiple inheritance as well as mixins" and then say "It is obvious, that such a class has to comply to the parents call interface - here object - when to be used standalone." I don't understand what you mean by "standalone" here, but to be clear, Python mixins are just regular classes. There is no difference between ordinary classes and mixin classes in Python, and they use the same multiple-inheritance method resolution order whether you think of them as "mixins" or "multiple inheritance". You say: "BUT when I ignore this and set intentionally a parameter - for test-purposes of mixins only - the resolution algorithm seems to normalize both for the final call without parameters - as soon as one of the mixins is a non-parameter call. This is independent from the inheritance order for the derived class." I am sorry, I cannot work out what you are saying here. |
|||
msg353084 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2019-09-24 12:34 | |
I took the health of my PC into my hands *smiles* and unzipped your test cases. I don't know .sh scripts well, so I didn't run that (and Windows users won't be able to run it), but I looked at the Python scripts, and I think I have duplicated your test cases in a way that I find easier to understand. (Fewer files, more descriptive names, less code duplication.) |
|||
msg353090 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2019-09-24 13:07 | |
I'm not an expert on super and multiple inheritance, but let me see if I can explain what is going on. (Hopefully an expert will correct me if I get something wrong.) 1. The class C(A_With_Arg, B_With_Arg) is straight-forward: the super call inside A passes the True parameter to B, which passes it to object, which raises. 2. The class C(B_With_Arg, A_With_Arg) is the same, except B passes the parameter to A, which passes it to object, which raises. 3. The class C(A_With_Arg, B_Without_Arg) is also straight-forward: the super call inside A passes the True parameter to B, but B ignores it and passes nothing to object. 4. The class C(B_Without_Arg, A_With_Arg) is the tricky case. The super call inside B passes no arguments to A, so A gets the default value of False for its parameter. Since A's propagate parameter is False, it doesn't call super at all, object's __init__ method doesn't get called, and there's no exception. I've added another version, mixin_tests2.py, which hopefully makes it a bit easier to see what is happening. You said: "The 'super()' call propagates the call resolution by definition to the next level". I think this means that you expect that in class C(A, B), A.__init__ will call object.__init__, and B.__init__ will call object.__init__ as well. But that's not what happens. That's what Python 2 "classic classes" did, but it was buggy, because superclasses would be called *twice*. Perhaps it will help to read this: https://www.python.org/download/releases/2.3/mro/ So I believe that the behaviour you are seeing is correct, there is no error or bug. Do you agree? If so, we can close this issue. Otherwise, if you disagree and still believe this is a bug, please explain why you think it is a bug. |
|||
msg353118 - (view) | Author: Arno-Can Uestuensoez (acue) | Date: 2019-09-24 20:54 | |
Hey, first of all thank you for the fast reply. Your examples reinforce exactly my issue. I prefer the KISS principle in providing isolated use-cases for transparency without extras. Which is seperated reduced flat calls only. The other point is, I develop and test shared code for Python2.7(which will be out there for a while...) and Python3.5+, thus prefer shared syntax whenever possible. The "super()" interface has changed, using the shared syntax from 2.7.16: - Python2.7.16: super(type[, object-or-type]) - Python3.7.4: super([type[, object-or-type]]) The test of your code with Python 2.7.16 requires modification. You are right with the case 4, this is the actual issue related to the MRO and the parameter passing. I did not ask for the number of calls of the common base classes, but the passing of the parameters. The call order/MRO is OK, but I doubt the passed parameter signatures. My basic assumption here is that the call routing by MRO should or even must not be intermixed with the actually passed call parameters. So I added the output of your example for "mixin_tests2.py" for Python3.6.5, tested for Python3.7.0 too. #-------------- (3.6.5) [acue@lap001 test-modified]$ python mixin_tests2.py ============================== C MRO: ['C', 'A_With_Arg', 'B_With_Arg', 'MyBase', 'object'] C:C True A_With_Arg:C True call super().__init__ with argument B_With_Arg:C True call super().__init__ with argument MyBase received 1 arguments TypeError('object.__init__() takes no parameters',) ============================== C MRO: ['C', 'B_With_Arg', 'A_With_Arg', 'MyBase', 'object'] C:C True B_With_Arg:C True call super().__init__ with argument A_With_Arg:C True call super().__init__ with argument MyBase received 1 arguments TypeError('object.__init__() takes no parameters',) ============================== C MRO: ['C', 'A_With_Arg', 'B_Without_Arg', 'MyBase', 'object'] C:C True A_With_Arg:C True call super().__init__ with argument B_Without_Arg:C True call super().__init__ without argument MyBase received 0 arguments success ============================== C MRO: ['C', 'B_Without_Arg', 'A_With_Arg', 'MyBase', 'object'] C:C True B_Without_Arg:C True call super().__init__ without argument A_With_Arg:C False success (3.6.5) [acue@lap001 test-modified]$ #----------------------- Or reduced to the focus of the actual issue and depicted the actual passed parameters. See the following "Case 4 - same as mixin_C_is_B0A1" for the essential question. ============================== Case 0 - same as mixin_C_is_A1B1 A_With_Arg:C True B_With_Arg:C True TypeError('object.__init__() takes no parameters',) Resulting in: C.__init__(True) A_With_Arg.__init__(True) B_With_Arg.__init__(True) MyBase.__init__(True) object.__init__(True) Comment: The TypeError is expected and OK. ============================== Case 1 - same as mixin_C_is_B1A1 B_With_Arg:C True A_With_Arg:C True TypeError('object.__init__() takes no parameters',) Resulting in: C.__init__(True) B_With_Arg.__init__(True) A_With_Arg.__init__(True) MyBase.__init__(True) object.__init__(True) Comment: The TypeError is expected and OK. ============================== Case 2 - same as mixin_C_is_A1B0 A_With_Arg:C True B_Without_Arg:C True MyBase received 0 arguments success Comment: The success is expected and OK. The actual call due to MRO is: C(A_With_Arg(), B_Without_Arg()) Resulting in: C.__init__(True) A_With_Arg.__init__(True) B_Without_Arg.__init__(True) MyBase.__init__(False) object.__init__() Remark: I assume, that the single call of "object.__init__()" is here intentional. ============================== Case 4 - same as mixin_C_is_B0A1 B_Without_Arg:C True A_With_Arg:C False success Comment: The success is NOT expected and as far as I can see should be treated as an Error so NOK. The actual call is: C(B_Without_Arg(), A_With_Arg()) Resulting in: Seemingly the behaviour is: C.__init__(True) B_Without_Arg.__init__(True) A_With_Arg.__init__(False) This means the call of "super(B, self).__init__()" within "B.__init__()" defines the call parameters of "A.__init__()". Even though I expect the call order as routed by the MRO, I did not expected the parameters of A to be passed from the sibling class B, but as defined by the base class C. Thus the following: C.__init__(True) B_Without_Arg() A_With_Arg.__init__(True) object.__init__(True) # this is the behaviour I expect # due to the MRO and the # parameters passed from the # base class Remark: I assume, that the single call of "object.__init__()" is here intentional. #----------------------- The essential part I doubt is given by the case 4 / same as mixin_C_is_B0A1, where the call parameter of A is defined by the sibling B, instead of by the base class C. The routing of the call order is OK. My remark with "standalone call" depicts the fact, that instanciating e.g. only your class "A_With_Arg(True)" will raise in any case a TypeError, but using it in the context "C(A_With_Arg(), B_Without_Arg())" may not because the call of the "object" class follows the right-most mixin. But anyhow, this means different parameters by the "suppressed" left-side classes/alls would be silently suppressed. Resulting from the case 4, I tend to see the aspect of the parameter passing as erroneous. |
|||
msg353125 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2019-09-25 00:17 | |
Apologies for the Python 2.7 incompatible syntax, I hadn't noticed that you had selected 2.7. But since 2.7 (and 3.8) are in feature freeze, any behavoural changes can only apply to 3.9 or later. I'm afraid that you haven't explained why you think the code is buggy. Does the behaviour differ from the documented behaviour in any way? "My basic assumption here is that the call routing by MRO should or even must not be intermixed with the actually passed call parameters." I'm afraid I don't understand this. "Remark: I assume, that the single call of "object.__init__()" is here intentional." Of course it is intentional, as calling a superclass method twice is incorrect behaviour. You cannot expect that all superclass methods are idempotent. Obviously, in the case of ``object`` itself, there's no harm in calling __init__ twice but think about superclasses that have side-effects, or do expensive processing, and aren't idempotent. If you call them twice in the inheritance chain, bad things can happen. Which is why languages that support multiple inheritance generally require that the MRO is linearized so that each superclass is called once and exactly once. That was the problem with Python 2 classic classes and fixing it was one of the motivations of introducing so-called "new style classes" and object. See the link I provided in my previous comment, and also: https://courses.cs.northwestern.edu/325/readings/inheritance.php Unless you can explain how and why this behaviour differs from the documented behaviour, I'm going to close this as "Not a bug". If you want to argue that the C3 linearization algorithm is "wrong", you probably should take that to the Python-Dev mailing list. The argument by KISS is probably not going to be enough: it is more important for code to be *correct* than to be simple, and the simple multiple inheritance algorithm is demonstrably incorrect. |
|||
msg353131 - (view) | Author: Raymond Hettinger (rhettinger) * | Date: 2019-09-25 01:29 | |
> The oddity I came around is the simple case of mixins with > different parameters, where both are derived from 'object'. > It is obvious, that such a class has to comply to the parents > call interface - here object - when to be used standalone. BUT when > I ignore this and set intentionally a parameter - for test-purposes > of mixins only - the resolution algorithm seems to normalize both for the > final call without parameters - as soon as one of the mixins is a > non-parameter call. This is independent from the inheritance order > for the derived class. Sorry, this isn't a bug. In may be inconvenient but it is intrinsic to how cooperative multiple inheritance works. FWIW, the techniques for managing parameter passing are covered in this blog post: https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ |
|||
msg353136 - (view) | Author: Arno-Can Uestuensoez (acue) | Date: 2019-09-25 03:45 | |
Hi, first of all thank you both for the fast reply. The point is here, that the call routing and the parameter passing are intermixed. The MRO defines the call order, but does not define a signature change for the call within one inheritance layer. All articles I found including the mentioned blog-post(https://rhettinger.wordpress.com/2011/05/26/super-considered-super/) deal with the call order. But as far as I can see all imply silently the correctness of the call signatures defined by the derived classes - see also https://en.wikipedia.org/wiki/Object-oriented_programming#Polymorphism. Well, this is so common for me since about 1990, that I even have to think about how to explain. The simple diagram of the basic 4-class diamond depicts the call routing and the call signatures - see https://en.wikipedia.org/wiki/Multiple_inheritance#The_diamond_problem. The class hierarchy is A | +---+---+ | | B C | | +---+---+ | D with the classes A or more precise A(object) B(A) C(A) D(B, C) The resulting call routing MRO/C3 linearization - https://en.wikipedia.org/wiki/C3_linearization - is: D -> B -> C -> A See https://en.wikipedia.org/wiki/Multiple_inheritance#Mitigation, and https://www.python.org/download/releases/2.2.3/descrintro/#mro. This basic diamond structure represents the call signatures: I: for D II: for B and C III: for D The call signature of a given method interface for the implementation of class D has to be equally defined by the classes B *AND* C! Which means has to be the same - ignoring defaults here. This is how OO works. Though the called signatures of the classes B *AND* C has to be identical too! Which is completely independent from the dynamic call order - but defined by the static inheritance graph. This is in particular true due to the fact, that the single inheritance call signatures has to be identical too D(B).mthX() == D(C).mthX() or in general - related to the call signatures the following interface calls are defined to have equal call parameters(ignoring defaults here): D(B,C).mthX() == D(C,D).mthX() == D(B).mthX() == D(C).mthX() Anything else is an error. The current implementation changes the routed method calls to the following call signatures: I: for D II: for B III: for C and D So this is not just an inconvenience as you mentioned. It is simply a bug. |
|||
msg353137 - (view) | Author: Arno-Can Uestuensoez (acue) | Date: 2019-09-25 03:51 | |
Fix two typos in my previous post: #>>> This basic diamond structure represents the call signatures: I: for D II: for B and C III: for A #<<< and #>>> The current implementation changes the routed method calls to the following call signatures: I: for D II: for B III: for C and A #<<< |
|||
msg353167 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2019-09-25 10:55 | |
On Tue, Sep 24, 2019 at 08:54:49PM +0000, Arno-Can Uestuensoez wrote: > Or reduced to the focus of the actual issue and depicted the actual > passed parameters. See the following "Case 4 - same as > mixin_C_is_B0A1" for the essential question. > Case 4 - same as mixin_C_is_B0A1 > > B_Without_Arg:C True > A_With_Arg:C False > > success > > Comment: > The success is NOT expected and as far as I can see should be treated as an Error so NOK. Why would it be an error when you override object.__init__ completely? object.__init__ doesn't get called at all, because A overrides it: class A(object): def __init__(self, propagate=False): if propagate: super(A, self).__init__(propagate) # this section is implied else: pass # override the superclass B_Without_Arg calls ``super().__init__`` with no arguments, so ``A.__init__`` fills in the default argument, propagate=False. [...] > This means the call of "super(B, self).__init__()" within > "B.__init__()" defines the call parameters of "A.__init__()". Yes, that's how it works in cooperative multiple inheritance. It may help to read this: https://www.artima.com/weblogs/viewpost.jsp?thread=281127 The very badly named essay "Super considered harmful" is also worth reading. It is badly named because it is not *super* which is painful, and the author eventually was forced to admit that super is the best available solution to the complications of multiple inheritance. http://fuhm.net/super-harmful/ Somewhere on Youtube you should be able to find a video of a talk given by Raymond where he discusses ways to solve multiple inheritance problems. Sorry I don't remember what it is called. > Even though I expect the call order as routed by the MRO, > I did not expected the parameters of A to be passed from the > sibling class B, but as defined by the base class C. I think I've made that same mistake too. Perhaps the documentation should make this aspect of multiple inheritance more clear. I think that we have established that, whether you expected it or not, the behaviour you show is the way Python multiple inheritance has worked since version 2.3 and is working as designed. If you want to debate that design, the bug tracker is not the right place for it, and you should take it to the Python-Dev mailing list. It might be more productive to ask the mailing list why that design choice was made, before asserting that it is wrong. If you aren't fond of mailing lists, you could try the Python Discuss: https://discuss.python.org/ but either way, I think that this issue should be closed. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:20 | admin | set | github: 82443 |
2019-09-25 10:55:16 | steven.daprano | set | messages: + msg353167 |
2019-09-25 03:51:51 | acue | set | messages: + msg353137 |
2019-09-25 03:45:13 | acue | set | files:
+ mixin_C_is_B0A1.py resolution: not a bug -> postponed messages: + msg353136 |
2019-09-25 01:29:27 | rhettinger | set | status: open -> closed nosy: + rhettinger messages: + msg353131 resolution: not a bug stage: resolved |
2019-09-25 00:17:23 | steven.daprano | set | messages:
+ msg353125 versions: + Python 3.9, - Python 2.7, Python 3.6 |
2019-09-24 22:14:19 | ppperry | set | type: compile error -> behavior |
2019-09-24 20:54:48 | acue | set | files:
+ mixin_C_is_B0A1.py messages: + msg353118 |
2019-09-24 13:07:58 | steven.daprano | set | files:
+ mixin_tests2.py messages: + msg353090 |
2019-09-24 12:34:34 | steven.daprano | set | files:
+ mixin_tests.py messages: + msg353084 |
2019-09-24 11:42:08 | steven.daprano | set | nosy:
+ steven.daprano messages: + msg353075 |
2019-09-24 10:11:33 | acue | create |