classification
Title: object.__init__ shouldn't allow args/kwds
Type: Stage:
Components: Interpreter Core Versions: Python 3.0
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: KayEss, Rhamphoryncus, benjamin.peterson, blakeross, georg.brandl, gregory.p.smith, gvanrossum, jason.coombs, jcea, jonash, rhettinger, terry.reedy
Priority: normal Keywords:

Created on 2007-03-19 03:32 by blakeross, last changed 2013-01-15 03:34 by jason.coombs. This issue is now closed.

Files
File name Uploaded Description Edit
new_init.patch gvanrossum, 2007-03-19 23:31 Proposed patch to typeobject.c.
new_init_strict.patch gvanrossum, 2007-03-20 20:54 Stricter version that doesn't quite work. :-(
Messages (29)
msg31571 - (view) Author: Blake Ross (blakeross) Date: 2007-03-19 03:32
object.__init__ currently allows any amount of args and keywords even though they're ignored. This is inconsistent with other built-ins, like list, that are stricter about what they'll accept. It's also inconsistent with object.__new__, which does throw if any are provided (if the default __init__ is to be used).

To reproduce:

object.__init__(object(), foo, bar=3)

This is a slight irritation when using cooperative super calling. I'd like each class' __init__ to cherry-pick keyword params it accepts and pass the remaining ones up the chain, but right now I can't rely on object.__init__ to throw if there are remaining keywords by that point.
msg31572 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2007-03-19 08:56
I don't really understand either why object_new() checks the arguments, not object_init():

"""
static int
object_init(PyObject *self, PyObject *args, PyObject *kwds)
{
	return 0;
}

/* If we don't have a tp_new for a new-style class, new will use this one.
   Therefore this should take no arguments/keywords.  However, this new may
   also be inherited by objects that define a tp_init but no tp_new.  These
   objects WILL pass argumets to tp_new, because it gets the same args as
   tp_init.  So only allow arguments if we aren't using the default init, in
   which case we expect init to handle argument parsing. */
static PyObject *
object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
	if (type->tp_init == object_init && (PyTuple_GET_SIZE(args) ||
	     (kwds && PyDict_Check(kwds) && PyDict_Size(kwds)))) {
		PyErr_SetString(PyExc_TypeError,
				"default __new__ takes no parameters");
		return NULL;
	}
	return type->tp_alloc(type, 0);
}
"""
msg31573 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-19 20:01
I'll try to explain why I did it this way.  I was considering the single inheritance case implementing an Immutable object, which overrides __new__ but has no need to override __init__ (since it's too late to do anything in __init__ for an Immutable object).  Since the __init__ still gets called it would be annoying to have to override it just to make the error go away if there was a check in __init__.  The other case is overriding __init__ without overriding __new__, which is the most common way of doing Mutable objects; here you wouldn't want __new__ to complain about extra args.  So the only time when you'd want complaints is if both __new__ and __init__ are the defaults, in which case it doesn't really matter whether you implement this in __init__ or in __new__, so I arbitrarily chose __new__.

I wasn't thinking of your use case at the time though (cooperative super calls to __init__, which still isn't something I engage in on a day-to-day basis).  I wonder if the right thing to do wouldn't be to implement the same check both in __init__ and in __new__.

Am I makign sense?
msg31574 - (view) Author: Blake Ross (blakeross) Date: 2007-03-19 22:45
Makes sense. I don't think we can ever be completely correct here since we're inferring intent from the presence of __init__/__new__ that's liable to be wrong in some cases, but it's likely correct often enough that it's worth doing.

If I understand correctly, we want to be more forgiving iff one of the two methods is used, so it seems like we should be complaining if both are used *or* if neither is used. After all, I could add a __new__ to my coop use case and I'd still want object to complain. If that's the case, both object_new and object_init should be complaining if ((tp->tp_new == object_new && tp->tp_init == object_init) || (tp->tp_new != object_new && tp->tp_init != object_init)).

Of course, for the paranoid, there's always the risk that __new__ will modify these class functions and change the outcome :) For instance, if a class had a __new__ and no __init__ and its __new__ changed __new__ back to object.__new__, object_init on that run would be fooled into thinking it's using the defaults for both and would complain. I think this could only be fixed in type_call, which is rather ugly...but then, this *is* a special case of the "call __init__ after __new__" behavior, and we're trying to solve it within the methods themselves. Perhaps this last point is academic enough to be ignored...I don't know why anyone would do this, although the language makes it possible.
msg31575 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-19 23:31
Attached is a patch that implements this proposal, adding copious commentary.  It doesn't seem to break anything in the test suite.

I wonder if we should even make the check more rigid: check the argument list if either the current method *is* overridden or the other one *is not* overridden.  This would make super calls check the arguments even if the other method is overridden.  What do you think?
File Added: new_init.patch
msg31576 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-19 23:35
This smells enough like a new feature that it couldn't go into 2.5.
msg31577 - (view) Author: Blake Ross (blakeross) Date: 2007-03-20 01:27
I think making the check more rigid is a good idea, since this should throw:

class a(object):
   def __init__(self, foo):
       super(a, self).__init__(foo)
   def __new__(cls, foo):
       return object.__new__(cls)
a(1)

(minor typo in the patch: "solution it" -> "solution is")
msg31578 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-20 20:54
Here's a stricter version. Unfortunately it breaks a couple of standard modules; this is a confirmation of my doubts whether the style of cooperative super calling of __init__ that you use is really the most common or "best practice".

So far I have only fixed string.py (which would otherwise prevent extensions from being built); I haven't looked into why the other tests fail: test_array, test_cpickle, test_descr, test_pickle (and maybe more?).

My conclusion: this would probably break too much code to be worth it.  So I'll have to revert to the previous version.  But anyway, here it is for your perusal.
File Added: new_init_strict.patch
msg31579 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-20 21:24
I should mention that if we can't get the strict version of this in 2.6, we should be able to get it into 3.0.
msg31580 - (view) Author: Blake Ross (blakeross) Date: 2007-03-21 22:03
Looks good. I skimmed briefly the tests you mentioned. The issue with test_array appears to be exactly the kind of bug this is intended to identify: it calls array.__init__(...), but array doesn't have its own initializer, so object's is used.

I'd guess that the others are failing due to whatever the problem with pickling is (test_descr uses pickling). I haven't looked into that yet.

I'm sure cooperative super calling of __init__ isn't all that common (it seems like the mechanism itself isn't used much yet, and may not be until it's done via keyword) but it doesn't seem like such a bad practice, especially when mixins are in the picture. There doesn't seem to be a great alternative.
msg31581 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-21 22:42
Well, but since it's been like this for a long time, I don't want to gratuitously break code. At least not in 2.6. So I'm rejecting the stricter patch for 2.6. (However, if you want to submit patches that would fix these breakages anyway, be my guest.)
msg31582 - (view) Author: Blake Ross (blakeross) Date: 2007-03-21 22:48
Holding the strict version for 3 makes sense to me. Let me know if you need anything more on my end... thanks for the fast turnaround.
msg31583 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2007-03-21 23:16
I ask myself, what should I expect from the documentation...

>>> object.__init__.__doc__
'x.__init__(...) initializes x; see x.__class__.__doc__ for signature'
>>> object.__class__
<type 'type'>
>>> type.__doc__
"type(object) -> the object's type\ntype(name, bases, dict) -> a new type"

and I still don't know ;-).
msg31584 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2007-03-22 00:44
I think the avoidance of super() is largely *because* of this kind of leniency.  Consider this snippet on python 2.3:

class Base(object):
    def __init__(self, foo=None, *args, **kwargs):
        super(Base, self).__init__(foo, *args, **kwargs)
Base(foo='bar')
Base('bar', 42)
Base('bar', 42, x=7)

All pass silently, no error checking.  Now consider a python 3.0 version, with a strict object.__init__:

class Base:
    def __init__(self, foo=None, *, y='hi', **kwargs):
        super(Base, self).__init__(**kwargs)
Base(foo='bar')  # Valid, accepted
Base('bar', 42)  # Raises exception
Base('bar', x=7)  # Raises exception

The error checking added by this bug/patch and the error checking added by PEP 3102 (keyword-only arguments) make super a lot more sane.

I think it would also help if calling a method via super() didn't allow positional arguments.  If the base class's arguments can't be given as keyword args then you probably should call it explicitly, rather than relying on super()'s MRO.

I was also going to suggest super() should automagically create empty methods your parent classes don't have, but then I realized you really should have a base class that asserts the lack of such an automagic method:

class Base(object):
    def mymethod(self, myonlyarg='hello world'):
        assert not hasattr(super(Base, self), 'mymethod')

By the time you reach this base class you will have stripped off any extra arguments that your subclasses added, leaving you with nothing to pass up (and nothing to pass to).  Having two mixins with the same method name and without a common parent class is just not sane.
msg31585 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-22 04:28
> I think it would also help if calling a method via super() didn't allow
> positional arguments.

That's absurd, *except* for __init__(), where it could make sense depending on the style of cooperation used.  But not enough to enforce this in the language; in Py3k you will be able to enforce this on a per-class basis.

> Having two mixins with the same method name and
> without a common parent class is just not sane.

Right.  This is a cornerstone of cooperative multiple inheritance that sometimes seems to be forgotten; there is a big difference between defining a method and extending a method, and only extending methods can make super calls.

The __init__ case is an exception, because there's no requirement that a subclass have a signature compatible with the superclass (if you don't get this, read up on constructors in C++ or Java).
msg31586 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2007-03-22 04:46
>> I think it would also help if calling a method via super() didn't allow
>> positional arguments.
>
> That's absurd, *except* for __init__(), where it could make sense
> depending on the style of cooperation used.  But not enough to enforce this
> in the language; in Py3k you will be able to enforce this on a per-class
> basis.

The vast majority of "positional" arguments can also be given via name.  The rare exceptions (primarily C functions) may not cooperate well anyway, so you're trading a relatively obscure limitation for better error detection.

Perhaps not that important though, since it could be taught as bad style unless absolutely needed.


>> Having two mixins with the same method name and
>> without a common parent class is just not sane.
>
> Right.  This is a cornerstone of cooperative multiple inheritance that
> sometimes seems to be forgotten; there is a big difference between defining
> a method and extending a method, and only extending methods can make super
> calls.
>
> The __init__ case is an exception, because there's no requirement that a
> subclass have a signature compatible with the superclass (if you don't get
> this, read up on constructors in C++ or Java).

I understand the desire for it to be an exception, I fail to see how it actually is one.  The namespace/signature conflicts exist just the same.

The only way I can see to handle incompatible signatures is to add a flag that says "I am the *ONLY* class allowed to subclass X" (triggering an error if violated), have super() entirely bypass it, and then call X.__init__() directly.  Even that doesn't handle X's superclasses being subclassed more than once, and it looks pretty complicated/obscure anyway.
msg31587 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-03-23 05:01
Committed revision 54539.

The committed version issues warnings rather than errors when both methods are overridden, to avoid too much breakage.

The string.py change was necessary to avoid spurious warnings (with no module/lineno!) and breakage of test_subprocess.py.  Something fishy's going on -- is string.Template() used by the warnings module or by site.py???

I'm leaving this bug open but changing the category to Py3k so remind me it needs to be merged and then changed there.
msg31588 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2007-04-02 23:12
FWIW, this change will be somewhat pervasive and will affect anything inheriting object.__init__ including immutable builtins (like tuple, float, and frozenset) as well as user-defined new-style classes that do not define their own __init__ method (perhaps using new instead).  Here are the warnings being thrown-off by the current test suite:

/py26/Lib/test/test_array.py:731: DeprecationWarning: object.__init__() takes no parameters
  array.array.__init__(self, 'c', s)

/py26/Lib/copy_reg.py:51: DeprecationWarning: object.__init__() takes no parameters
  base.__init__(obj, state)

/py26/Lib/test/test_descr.py:2308: DeprecationWarning: object.__init__() takes no parameters
  float.__init__(self, value)
msg31589 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-04-02 23:55
That's one way of looking at it. You could also say that it found two legitimate problems:

- since array doesn't define __init__() there's no point in calling it

- similarly, float doesn't define __init__()

The copy_reg warning is more subtle, and needs a work-around.  I've checked in all three fixes.
msg65418 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-04-12 20:59
Can this be closed?
msg98720 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-02-02 06:01
FYI - A discussion on why this change may have been a bad idea and breaks peoples existing code:

 http://freshfoo.com/blog/object__init__takes_no_parameters
msg102136 - (view) Author: Jonas H. (jonash) Date: 2010-04-01 22:47
What exactly is the correct solution with Python 2.6 to avoid this warning? My use case is something like

class myunicode(unicode):
  def __init__(self, *args, **kwargs):
    unicode.__init__(self, *args, **kwargs)
    self.someattribute = calculate_attribute_once()

Shall I overwrite __new__ rather than __init__? Or what :-)
msg102138 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-04-01 23:16
@dauerbaustelle
I believe your question is a separate issue and that it should have been asked on Python list. However, yes, subclasses of immutables must override __new__. For more, do ask on the list, not here.
msg156137 - (view) Author: Roundup Robot (python-dev) Date: 2012-03-17 05:06
New changeset 25b71858cb14 by Benjamin Peterson in branch 'default':
make extra arguments to object.__init__/__new__ to errors in most cases (finishes #1683368)
http://hg.python.org/cpython/rev/25b71858cb14
msg156138 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2012-03-17 05:09
Please don't add python-dev@python.org to the nosy list.
msg156139 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-17 05:11
python-dev is just the name of the robot which notes records changesets.
msg179976 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-01-14 21:08
For reference, I encountered an issue due to this change and didn't quite understand what was going on. I distilled the problem down and posted a question on stack overflow:

http://stackoverflow.com/questions/14300153/why-does-this-datetime-subclass-fail-on-python-3/14324503#14324503

The answer led me here, so now I understand. I wanted to share this use-case for posterity.

I didn't find anything in the "what's new" documents for Python 3.3 or 3.0. Was this fundamental signature change to all objects documented anywhere? Any objection if I draft a change to the docs?
msg179999 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-01-15 02:47
First, What's New " explains the new features in Python". This issue is a bugfix. AFAIK, object() has always been documented as having no parameters. The fact that passing extra args should raise a TypeError is no secret.

Second, this *is* documented. The third sentence of
http://docs.python.org/3/whatsnew/3.3.html
is " For full details, see the changelog." We really mean that ;-). The changelog is derived from Misc/NEWS in the repository. It says "Issue #1683368: object.__new__ and object.__init__ raise a TypeError if they are passed arguments and their complementary method is not overridden." That is prefixed by "Issue #1683368:", which links to this issue. This entry is easily found by searching for 'object.__init__' (or a sufficient prefix thereof).

For 3.2, the What's New sentence was "For full details, see the Misc/NEWS file" and the link went to the raw repository file.
http://hg.python.org/cpython/file/3.2/Misc/NEWS
My impression is that this issue played a role in including the prettified version, instead of just the repository link, in the on-line version of the docs. What's New for 2.7 does not even have the link.

In any case, *any* bugfix breaks code that depends on the bug. Hence the decision to make the full changelog more available and more readable.

I realize that the change to the header for What's New is hard to miss. But what are we to do? Add a new What's New in What's New doc for one release? Put the change in flashing red type?
msg180002 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2013-01-15 03:34
Aah. Indeed, that's where I should have looked. Thanks for the pointer.
History
Date User Action Args
2013-01-15 03:34:27jason.coombssetmessages: + msg180002
2013-01-15 02:47:36terry.reedysetmessages: + msg179999
2013-01-14 21:08:39jason.coombssetnosy: + jason.coombs
messages: + msg179976
2012-03-17 05:11:55benjamin.petersonsetmessages: + msg156139
2012-03-17 05:09:50gvanrossumsetnosy: - python-dev
messages: + msg156138
2012-03-17 05:06:18python-devsetnosy: + python-dev
messages: + msg156137
2010-04-01 23:16:11terry.reedysetmessages: + msg102138
2010-04-01 22:47:13jonashsetnosy: + jonash
messages: + msg102136
2010-02-02 06:01:19gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg98720
2009-04-20 10:35:10KayEsssetnosy: + KayEss
2008-10-13 12:41:32jceasetnosy: + jcea
2008-04-14 20:38:47gvanrossumsetstatus: open -> closed
2008-04-12 20:59:56benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg65418
2008-01-06 22:29:46adminsetkeywords: - py3k
versions: Python 3.0
2007-08-30 00:22:14gvanrossumsetversions: + Python 3.0
2007-03-19 03:32:33blakerosscreate