object.__init__ shouldn't allow args/kwds #44742
Comments
|
object.__init__ currently allows any amount of args and keywords even though they're ignored. This is inconsistent with other built-ins, like list, that are stricter about what they'll accept. It's also inconsistent with object.__new__, which does throw if any are provided (if the default __init__ is to be used). To reproduce: object.__init__(object(), foo, bar=3) This is a slight irritation when using cooperative super calling. I'd like each class' __init__ to cherry-pick keyword params it accepts and pass the remaining ones up the chain, but right now I can't rely on object.__init__ to throw if there are remaining keywords by that point. |
blakeross
mannequin
added
the
interpreter-core
blakeross
mannequin
added
the
interpreter-core
|
I don't really understand either why object_new() checks the arguments, not object_init(): """
static int
object_init(PyObject *self, PyObject *args, PyObject *kwds)
{
return 0;
}
/* If we don't have a tp_new for a new-style class, new will use this one.
Therefore this should take no arguments/keywords. However, this new may
also be inherited by objects that define a tp_init but no tp_new. These
objects WILL pass argumets to tp_new, because it gets the same args as
tp_init. So only allow arguments if we aren't using the default init, in
which case we expect init to handle argument parsing. */
static PyObject *
object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
if (type->tp_init == object_init && (PyTuple_GET_SIZE(args) ||
(kwds && PyDict_Check(kwds) && PyDict_Size(kwds)))) {
PyErr_SetString(PyExc_TypeError,
"default __new__ takes no parameters");
return NULL;
}
return type->tp_alloc(type, 0);
}
""" |
|
I'll try to explain why I did it this way. I was considering the single inheritance case implementing an Immutable object, which overrides __new__ but has no need to override __init__ (since it's too late to do anything in __init__ for an Immutable object). Since the __init__ still gets called it would be annoying to have to override it just to make the error go away if there was a check in __init__. The other case is overriding __init__ without overriding __new__, which is the most common way of doing Mutable objects; here you wouldn't want __new__ to complain about extra args. So the only time when you'd want complaints is if both __new__ and __init__ are the defaults, in which case it doesn't really matter whether you implement this in __init__ or in __new__, so I arbitrarily chose __new__. I wasn't thinking of your use case at the time though (cooperative super calls to __init__, which still isn't something I engage in on a day-to-day basis). I wonder if the right thing to do wouldn't be to implement the same check both in __init__ and in __new__. Am I makign sense? |
|
Makes sense. I don't think we can ever be completely correct here since we're inferring intent from the presence of __init__/new that's liable to be wrong in some cases, but it's likely correct often enough that it's worth doing. If I understand correctly, we want to be more forgiving iff one of the two methods is used, so it seems like we should be complaining if both are used *or* if neither is used. After all, I could add a __new__ to my coop use case and I'd still want object to complain. If that's the case, both object_new and object_init should be complaining if ((tp->tp_new == object_new && tp->tp_init == object_init) || (tp->tp_new != object_new && tp->tp_init != object_init)). Of course, for the paranoid, there's always the risk that __new__ will modify these class functions and change the outcome :) For instance, if a class had a __new__ and no __init__ and its __new__ changed __new__ back to object.__new__, object_init on that run would be fooled into thinking it's using the defaults for both and would complain. I think this could only be fixed in type_call, which is rather ugly...but then, this *is* a special case of the "call __init__ after __new__" behavior, and we're trying to solve it within the methods themselves. Perhaps this last point is academic enough to be ignored...I don't know why anyone would do this, although the language makes it possible. |
|
Attached is a patch that implements this proposal, adding copious commentary. It doesn't seem to break anything in the test suite. I wonder if we should even make the check more rigid: check the argument list if either the current method *is* overridden or the other one *is not* overridden. This would make super calls check the arguments even if the other method is overridden. What do you think? |
|
This smells enough like a new feature that it couldn't go into 2.5. |
|
I think making the check more rigid is a good idea, since this should throw: class a(object):
def __init__(self, foo):
super(a, self).__init__(foo)
def __new__(cls, foo):
return object.__new__(cls)
a(1)(minor typo in the patch: "solution it" -> "solution is") |
|
Here's a stricter version. Unfortunately it breaks a couple of standard modules; this is a confirmation of my doubts whether the style of cooperative super calling of __init__ that you use is really the most common or "best practice". So far I have only fixed string.py (which would otherwise prevent extensions from being built); I haven't looked into why the other tests fail: test_array, test_cpickle, test_descr, test_pickle (and maybe more?). My conclusion: this would probably break too much code to be worth it. So I'll have to revert to the previous version. But anyway, here it is for your perusal. |
|
I should mention that if we can't get the strict version of this in 2.6, we should be able to get it into 3.0. |
|
Looks good. I skimmed briefly the tests you mentioned. The issue with test_array appears to be exactly the kind of bug this is intended to identify: it calls array.__init__(...), but array doesn't have its own initializer, so object's is used. I'd guess that the others are failing due to whatever the problem with pickling is (test_descr uses pickling). I haven't looked into that yet. I'm sure cooperative super calling of __init__ isn't all that common (it seems like the mechanism itself isn't used much yet, and may not be until it's done via keyword) but it doesn't seem like such a bad practice, especially when mixins are in the picture. There doesn't seem to be a great alternative. |
|
Well, but since it's been like this for a long time, I don't want to gratuitously break code. At least not in 2.6. So I'm rejecting the stricter patch for 2.6. (However, if you want to submit patches that would fix these breakages anyway, be my guest.) |
|
Holding the strict version for 3 makes sense to me. Let me know if you need anything more on my end... thanks for the fast turnaround. |
|
I ask myself, what should I expect from the documentation... >>> object.__init__.__doc__
'x.__init__(...) initializes x; see x.__class__.__doc__ for signature'
>>> object.__class__
<type 'type'>
>>> type.__doc__
"type(object) -> the object's type\ntype(name, bases, dict) -> a new type"and I still don't know ;-). |
|
I think the avoidance of super() is largely *because* of this kind of leniency. Consider this snippet on python 2.3: class Base(object):
def __init__(self, foo=None, *args, **kwargs):
super(Base, self).__init__(foo, *args, **kwargs)
Base(foo='bar')
Base('bar', 42)
Base('bar', 42, x=7)All pass silently, no error checking. Now consider a python 3.0 version, with a strict object.__init__: class Base:
def __init__(self, foo=None, *, y='hi', **kwargs):
super(Base, self).__init__(**kwargs)
Base(foo='bar') # Valid, accepted
Base('bar', 42) # Raises exception
Base('bar', x=7) # Raises exceptionThe error checking added by this bug/patch and the error checking added by PEP-3102 (keyword-only arguments) make super a lot more sane. I think it would also help if calling a method via super() didn't allow positional arguments. If the base class's arguments can't be given as keyword args then you probably should call it explicitly, rather than relying on super()'s MRO. I was also going to suggest super() should automagically create empty methods your parent classes don't have, but then I realized you really should have a base class that asserts the lack of such an automagic method: class Base(object):
def mymethod(self, myonlyarg='hello world'):
assert not hasattr(super(Base, self), 'mymethod')By the time you reach this base class you will have stripped off any extra arguments that your subclasses added, leaving you with nothing to pass up (and nothing to pass to). Having two mixins with the same method name and without a common parent class is just not sane. |
That's absurd, *except* for __init__(), where it could make sense depending on the style of cooperation used. But not enough to enforce this in the language; in Py3k you will be able to enforce this on a per-class basis.
Right. This is a cornerstone of cooperative multiple inheritance that sometimes seems to be forgotten; there is a big difference between defining a method and extending a method, and only extending methods can make super calls. The __init__ case is an exception, because there's no requirement that a subclass have a signature compatible with the superclass (if you don't get this, read up on constructors in C++ or Java). |
The vast majority of "positional" arguments can also be given via name. The rare exceptions (primarily C functions) may not cooperate well anyway, so you're trading a relatively obscure limitation for better error detection. Perhaps not that important though, since it could be taught as bad style unless absolutely needed.
I understand the desire for it to be an exception, I fail to see how it actually is one. The namespace/signature conflicts exist just the same. The only way I can see to handle incompatible signatures is to add a flag that says "I am the *ONLY* class allowed to subclass X" (triggering an error if violated), have super() entirely bypass it, and then call X.__init__() directly. Even that doesn't handle X's superclasses being subclassed more than once, and it looks pretty complicated/obscure anyway. |
|
Committed revision 54539. The committed version issues warnings rather than errors when both methods are overridden, to avoid too much breakage. The string.py change was necessary to avoid spurious warnings (with no module/lineno!) and breakage of test_subprocess.py. Something fishy's going on -- is string.Template() used by the warnings module or by site.py??? I'm leaving this bug open but changing the category to Py3k so remind me it needs to be merged and then changed there. |
|
FWIW, this change will be somewhat pervasive and will affect anything inheriting object.__init__ including immutable builtins (like tuple, float, and frozenset) as well as user-defined new-style classes that do not define their own __init__ method (perhaps using new instead). Here are the warnings being thrown-off by the current test suite: /py26/Lib/test/test_array.py:731: DeprecationWarning: object.__init__() takes no parameters /py26/Lib/copy_reg.py:51: DeprecationWarning: object.__init__() takes no parameters /py26/Lib/test/test_descr.py:2308: DeprecationWarning: object.__init__() takes no parameters |
|
That's one way of looking at it. You could also say that it found two legitimate problems:
The copy_reg warning is more subtle, and needs a work-around. I've checked in all three fixes. |
|
Can this be closed? |
|
FYI - A discussion on why this change may have been a bad idea and breaks peoples existing code: |
|
What exactly is the correct solution with Python 2.6 to avoid this warning? My use case is something like class myunicode(unicode):
def __init__(self, *args, **kwargs):
unicode.__init__(self, *args, **kwargs)
self.someattribute = calculate_attribute_once()Shall I overwrite __new__ rather than __init__? Or what :-) |
|
@dauerbaustelle |
|
New changeset 25b71858cb14 by Benjamin Peterson in branch 'default': |
|
Please don't add python-dev@python.org to the nosy list. |
|
python-dev is just the name of the robot which notes records changesets. |
|
For reference, I encountered an issue due to this change and didn't quite understand what was going on. I distilled the problem down and posted a question on stack overflow: The answer led me here, so now I understand. I wanted to share this use-case for posterity. I didn't find anything in the "what's new" documents for Python 3.3 or 3.0. Was this fundamental signature change to all objects documented anywhere? Any objection if I draft a change to the docs? |
|
First, What's New " explains the new features in Python". This issue is a bugfix. AFAIK, object() has always been documented as having no parameters. The fact that passing extra args should raise a TypeError is no secret. Second, this *is* documented. The third sentence of For 3.2, the What's New sentence was "For full details, see the Misc/NEWS file" and the link went to the raw repository file. In any case, *any* bugfix breaks code that depends on the bug. Hence the decision to make the full changelog more available and more readable. I realize that the change to the header for What's New is hard to miss. But what are we to do? Add a new What's New in What's New doc for one release? Put the change in flashing red type? |
|
Aah. Indeed, that's where I should have looked. Thanks for the pointer. |
|
I recently ran into this error again. I was writing this class to provide backward-compatible context manager support for zipfile.ZipFile on Python 2.6 and 3.1: class ContextualZipFile(zipfile.ZipFile):
"""
Supplement ZipFile class to support context manager for Python 2.6
"""
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.close()
def __new__(cls, *args, **kwargs):
"""
Construct a ZipFile or ContextualZipFile as appropriate
"""
if hasattr(zipfile.ZipFile, '__exit__'):
return zipfile.ZipFile(*args, **kwargs)
return super(ContextualZipFile, cls).__new__(cls, *args, **kwargs)At the point where super is called, the author is unaware of the details of the function signature for zipfile.ZipFile.__new__, so simply passes the same arguments as were received by the derived class. However, this behavior raises a DeprecationWarning on Python 2.6 and 3.1 (and would raise an error on Python 3.2 if the code allowed it). What's surprising is that the one cannot simply override a constructor or initializer without knowing in advance which of those methods are implemented (and with what signature) on the parent class. It seems like the construction (calling of __new__) is special-cased for classes that don't implement __new__. What is the proper implementation of ContextualZipFile.__new__? Should it use super but omit the args and kwargs? Should it call object.__new__ directly? Should it check for the existence of __new__ on the parent class (or compare it to object.__new__)? |
|
If you don't know enough about the base class you shouldn't be subclassing it. In this particular case you should be overriding __init__, not __new__. |
|
From what I see, you do not need to change either __new__ or __init__, just add __enter__ and __exit__ , and you only need to do that in 2.6. Since Zipfile is written in Python, you could monkey-patch instead of subclassing, if that is easier in your particular case. |
That's important when overriding any API in subclass and absolutely
Jason's code is doing something like OSError.__new__ does now, which |
|
Sorry, I didn't realize why __new__ was being used. But what Jason's code is doing isn't any cleaner than monkey-patching. |
|
Maybe I should have focused on a more trivial example to demonstrate the place where my expectation was violated. The use of a real-world example is distracting from my intended point. Consider instead this abstract example: class SomeClass(SomeParentClass):
def __new__(cls, *args, **kwargs):
return super(SomeClass, cls).__new__(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
super(SomeClass, self).__init__(*args, **kwargs)Ignoring for a moment the incongruity of the invocation of __new__ with 'cls' due to __new__ being a staticmethod, the naive programmer expects the above SomeClass to work exactly like SomeParentClass because both overrides are implemented as a trivial pass-through. And indeed that technique will work just fine if the parent class implements both __init__ and __new__, but if the parent class (or one of its parents) does not implement either of those methods, the technique will fail, because the fall through to 'object' class. I believe this incongruity stems from the fact that __new__ and __init__ are special-cased not to be called if they aren't implemented on the class. Therefore, to write SomeClass without knowledge of the SomeParentClass implementation, one could write this instead: class SomeClass(SomeParentClass):
def __new__(cls, *args, **kwargs):
super_new = super(SomeClass, cls).__new__
if super_new is object.__new__:
return super_new(cls)
return super_new(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
super_init = super(SomeClass, self).__init__
if super_init.__objclass__ is object:
return
super_init(*args, **kwargs)Now that implementation is somewhat ugly and perhaps a bit brittle (particularly around use of __objclass__). Ignoring that for now, it does have the property that regardless of the class from which it derives, it will work, including: SomeParentClass = datetime.datetime # implements only __new__
SomeParentClass = zipfile.ZipFile # implements only __init__
class SomeParentClass: pass # implements neither __init__ nor __new__While I would prefer a language construct that didn't require this dance for special casing (or similarly require the programmer to hard-code the dance to a specific implementation of a specific parent class as Guido recommends), at the very least I would suggest that the documentation better reflect this somewhat surprising behavior. Currently, the documentation states [https://docs.python.org/2/reference/datamodel.html#object.__new__] effectively "Typical implementations of __new__ invoke the superclass’ __new__() method with appropriate arguments." It's left as an exercise to the reader to ascertain what 'appropriate arguments' are, and doesn't communicate that the introduction or omission of __new__ or __init__ to a class hierarchy affects the process by which a class is constructed/initialized. Greg Smith's blog demonstrates some even more dangerous cases. I don't understand why his concerns weren't addressed, because they seem legitimate, and I agree with his conclusion that the older behavior is more desirable, despite the concerns raised by the OP. |
|
Based on the example above, I've created a blog post to publish my recommendation for overriding these special methods in a way that's safe regardless of the parent implementation, given the status quo: http://blog.jaraco.com/2014/05/how-to-safely-override-init-or-new-in.html |
|
Hrm. I've always thought that the key point of cooperative MI was the term *cooperative*. Consider a regular (non-constructor) method. You must have a common base class that defines this method, and *that* method shouldn't be calling the super-method (because there isn't one). All cooperative classes extending this method must derive from that base class. It's the same for __init__ and __new__, except that you may treat each (keyword) argument as a separate method. But you must still have a point in the tree to "eat" that argument, and that point must not pass it up the super call chain. If in a particular framework you want unrecognized keyword arguments to the constructor to be ignored, you should define a common base class from which all your cooperative subclasses inherit. But given the prevalence of *single* inheritance, 'object' shouldn't be that common base class. |
|
Jason, I made some recommendations on this subject in my blog post a few years ago: http://rhettinger.wordpress.com/2011/05/26/super-considered-super/ ''' Each level strips-off the keyword arguments that it needs so that the final empty dict can be sent to a method that expects no arguments at all (for example, object.__init__ expects zero arguments): class Shape:
def __init__(self, shapename, **kwds):
self.shapename = shapename
super().__init__(**kwds)
class ColoredShape(Shape):
def __init__(self, color, **kwds):
self.color = color
super().__init__(**kwds)
cs = ColoredShape(color='red', shapename='circle')''' |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: