classification
Title: Add functools.partialmethod
Type: enhancement Stage: resolved
Components: Extension Modules, Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Alexander.Belopolsky, Christophe Simonis, alonho, belopolsky, eckhardt, ironfroggy, jackdied, jcea, ncoghlan, python-dev, r.david.murray, rhettinger, ssadler, vajrasky
Priority: normal Keywords: patch

Created on 2008-11-16 06:32 by ssadler, last changed 2013-11-04 13:34 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
partialbug.py ssadler, 2008-11-16 06:32 Demonstration of bug
issue4331.patch Christophe Simonis, 2010-01-09 20:59 Patch - Add descriptor to partial object + tests
functools.partial-descrget.patch anacrolix, 2012-03-24 18:23 review
4331.v1.patch alonho, 2013-10-29 00:10 review
4331.v2.patch alonho, 2013-10-29 13:11 review
issue4331_partialmethod.diff ncoghlan, 2013-10-31 13:54 Proposed implementation of partialmethod review
Messages (32)
msg75928 - (view) Author: scott sadler (ssadler) Date: 2008-11-16 06:32
Calling a function created by _functools.partial as a method raises an
exception:

"TypeError: method_new() takes exactly n non-keyword arguments (0 given)"

Where method_new is the function passed into partial() and n is the
number of arguments it expects.

This does not happen when using a python version of partial().

Strangely, in the circumstance that I originally encountered the bug,
there was one instance that I was doing this and it _DID WORK_. The
function being passed into partial() was the same as in the place where
it was failing. The only significant difference that I could see was
that the input function to partial() was being imported, rather than
being defined in the same namespace as it was used I was unable to
reproduce it in my test case (attatched).

Tested on 2.6 and 2.5.2
msg75942 - (view) Author: scott sadler (ssadler) Date: 2008-11-16 20:24
A short update, I believe that the reason that it was working in one
instance was because of some abstractions by a base class (Django model,
get_absolute_url).
msg75945 - (view) Author: Calvin Spealman (ironfroggy) Date: 2008-11-16 22:44
I don't think this is any kind of bug, it is simply a product of only 
function objects being decorated automatically as methods. Your python 
version works because it is, in fact, a function. _functools.partial 
objects are not functions, but simply callable objects.
msg75946 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-11-16 23:03
Reclassifying as a feature request.
A descriptor could be added to partial()
so that it too would have automatic
method binding just like pure python functions.
msg97470 - (view) Author: Christophe Simonis (Christophe Simonis) Date: 2010-01-09 20:59
I followed the advice of Raymond and implement a descriptor on partial.
msg98490 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-01-29 00:25
Christophe,

It looks like your patch goes out of its way to avoid creating nested partials.  This is a worthwhile goal and I think it should be done in partial_new so that partial(partial(f, x), y) returns partial(f, x, y).

If fact, I was surprised to learn that current partial implementation does not behave this way:

>>> partial(partial(f, 1), 2).func
<functools.partial object at 0x100435af8>

Does anyone know the reason for the current behavior?  It is possible that I am missing some subtlety related to keyword arguments.
msg98675 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2010-02-01 19:09
Please see issue7830 for a related patch.
msg99776 - (view) Author: Jack Diederich (jackdied) * (Python committer) Date: 2010-02-22 16:33
I'm having some trouble wrapping my head around this one.  It isn't obvious to me that
my_method(*args):
  print(args)
class A():
  meth = partial(my_method, 'argA')
ob = A()
ob.meth('argB')

should print (<A object at 0x1234>, 'argA', 'argB') and not
('argA', <A object at 0x1234>, 'argB')

The patch seems to prefer the first form but if you are using a partial shouldn't you expect 'argA' to always be the first argument to the partial-ized function?
msg99813 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 19:11
I would expect the second and would view the first as a bug.
msg99956 - (view) Author: Jack Diederich (jackdied) * (Python committer) Date: 2010-02-23 21:35
We talked about it at sprints and the semantics are ambiguous and there are alternatives.

Ambiguous:
  def show_funcs(*args): print(args)
  class A():
    run = partial(1)
  ob = A()
  ob.run(2,3)
Should this print (self, 1, 2, 3) or (1, self, 2, 3)?  And what about
  partial(ob.run, 2)(3)

Alternatives: partial is a convenience function not an optimization (it doesn't offer a speedup.  So you can write a lambda or named function that has the exact semantics you want without suffering a speed penalty.

So unless there are a lot of good use cases with obvious behavior, we should refuse the temptation to guess and leave partial as-is.
msg99957 - (view) Author: Jack Diederich (jackdied) * (Python committer) Date: 2010-02-23 21:37
correction:
  run = partial(1)
should have been
  run = partial(show_funcs, 1)
msg156712 - (view) Author: Matt Joiner (anacrolix) Date: 2012-03-24 18:23
I've attached a patch that implements the descriptor protocol for functools.partial with minimum changes.
msg181582 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2013-02-07 08:36
Just for the record, the behaviour is documented, unfortunately in the very last line of the functools documentation: "Also, partial objects defined in classes behave like static methods and do not transform into bound methods during instance attribute look-up."

Concerning how exactly they should behave during that lookup, I'd use the least surprising variant, namely that they are not treated differently from other functions: The first parameter is implicitly "self".
msg181658 - (view) Author: Matt Joiner (anacrolix) Date: 2013-02-08 04:02
What's preventing this from being committed and closed?
msg182943 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2013-02-25 14:19
There is at least one thing that is missing in the patch, it lacks the necessary tests. The partialbug.py demonstrates the issue, it could be used as a base. However, even then, there is still one thing that is problematic: The fact that partial() returns something that behaves like a static method is documented and changing that is not backward compatible.

I still think that something like this should become part of Python though. Jack Diederich argues that you can use lambda to achieve the same, but that is not always true. If you want to bind an argument to the current value of a variable instead of a constant, lambda fails. You need the closure created by a function call to bind those variables inside a local function. Having a dedicated function for that is IMHO preferable to people copying the Python-only equivalent of partial() to achieve the same effect or even inventing their own.
msg182946 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-02-25 15:02
See also issue 11470.
msg190642 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-06-05 04:48
I don't believe it is reasonable to change the behaviour of partial at this late stage of the game. It's documented as behaving like staticmethod (albeit by not implementing the descriptor protocol at all), so that's no longer something we can change. If issue 11470 is added, then we'll just implement the staticmethod-like behaviour explicitly rather than leaving it as implicit.

More importantly, the acceptance of PEP 443's functools.singledispatch makes it more desirable than ever to support partial binding for method implementations *even when the descriptor protocol is not involved*.

Accordingly, I suggest converting this proposal to a separate functools.partialmethod API that:

1. When called directly, passes the first positional argument as the first positional argument of the underlying function (providing call time binding of self, just like a normal function)
2. When retrieved from a class, returns itself
3. When retrieved from an instance, returns an appropriate bound method object (providing method lookup time binding of self, just like a normal function)

functools.partial will then continue to behave as it always has (thus posing no backwards compatibility risks), while the new partialmethod implementation should work with both class descriptor protocol based dispatch (through point 3) *and* the new functools.singledispatch mechanism (through point 1).
msg190645 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-06-05 07:42
Any associated tests may also want check that wrapping classmethod around a partialmethod generates a well behaved class method, and ditto for property.

If singledispath, classmethod, partialmethod and class and instance attribute access all work correctly, then we can be absolutely certain the results is behaving just like an ordinary function does :)
msg190648 - (view) Author: Matt Joiner (anacrolix) Date: 2013-06-05 10:00
This sounds excellent Nick.
msg201111 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-24 10:33
To clarify the current state of this:

- I'm still in favour of adding this feature for Python 3.4
- a suitable patch is still needed, as the currently attached patches modify the existing functools.partial object, rather than adding a separate "partialmethod" API
- a Python implementation would be fine

The following prototype should work as a starting point to be elaborated into a full patch with docs and tests:

class partialmethod(functools.partial):
    def __get__(self, obj, cls):
        if obj is None:
            return self
        return functools.partial(self.func,
                                 *((obj,) + self.args),
                                 **self.keywords)
    def __call__(*args, **kwds):
        self, *args = args
        call_kwds = {}
        call_kwds.update(self.keywords)
        call_kwds.update(kwds)
        return self.func(self,
                         *(self.args + args),
                         **call_kwds)

class C:
    def example(self, *args, **kwds):
        print(self, args, kwds)
    fails = functools.partial(example, 1, 2, 3, x=1)
    works = partialmethod(example, 1, 2, 3, x=1)

>>> C().fails()
1 (2, 3) {'x': 1}
>>> C().works()
<__main__.C object at 0x7f91cefeea90> (1, 2, 3) {'x': 1}
>>> C().fails(4, 5, 6)
1 (2, 3, 4, 5, 6) {'x': 1}
>>> C().works(4, 5, 6)
<__main__.C object at 0x7f91cefeea10> (1, 2, 3, 4, 5, 6) {'x': 1}
msg201279 - (view) Author: alon horev (alonho) * Date: 2013-10-25 19:27
I just want to make sure I understand the semantics concerning class methods, the following example demonstrates a usage similar to regular methods as much as possible:

class A(object):
    def add(self, x, y):
        print(self)
        return x + y
    add10 = partialmethod(add, 10)
    add10class = classmethod(partialmethod(add, 10))

assert A().add10(5) == 15 # prints <__main__.A object at 0x1097e1390>
assert A.add10class(5) == 15 # prints <class '__main__.A'>

Another option would be to return a class-bound partial from the __get__ method. It's not as consistent as the first example but perhaps nicer:

class A(object):
    def add(self, x, y):
        print(self)
        return x + y
    add10 = partialmethod(add, 10)

assert A().add10(5) == 15 # prints <__main__.A object at 0x1097e1390>
assert A.add10(5) == 15 # prints <class '__main__.A'>

Is the first option what you had in mind?
msg201310 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-26 01:57
On 26 Oct 2013 05:28, "alon horev" <report@bugs.python.org> wrote:
> Is the first option what you had in mind?

That's actually an interesting question. I was going to say yes, but then I
realised it would be better to just "do the right thing" when the
underlying object was a classmethod descriptor, rather than composing them
the other way around.

That view means we should be delegating __get__ to the underlying
descriptor and responding appropriately to the result. And for __call__ we
then can't play games at all, since what my sketch does would be wrong when
wrapping staticmethod.

We also need to make sure the descriptor does the right thing when
@abstractmethod is involved.
msg201366 - (view) Author: alon horev (alonho) * Date: 2013-10-26 14:59
Here's another attempt at a consistent api with regular methods. 
I'm contemplating whether partialmethod should support __call__. Given the fact partial is used to bind positional arguments, it will do the 'wrong' thing when calling the partialmethod directly and will shift all positional arguments (example at the last line of the snippet).

I also think staticmethod in this context is useless but agree consistency is a thing (you could just use partial instead).

If consistency hadn't held us back, the first proposal of partialmethod, working both for instances and classes, would have been most elegant.

from functools import partial

class partialmethod(object):

    def __init__(self, func, *args, **keywords):
        self.func = func
        self.args = args
        self.keywords = keywords

    def __call__(self, *args, **keywords):
        call_keywords = {}
        call_keywords.update(self.keywords)
        call_keywords.update(keywords)
        return self.func(*(self.args + args), **call_keywords)
    
    def __get__(self, obj, cls):
        return partial(self.func.__get__(obj, cls), *self.args, **self.keywords)

class A(object):
    def add(self, x, y):
        print(self)
        return x + y
    add10 = partialmethod(add, 10)
    add10class = partialmethod(classmethod(add), 10)
    add10static = partialmethod(staticmethod(add), 'self', 10)

assert A().add10(5) == 15 # prints <__main__.A object at 0x1097e1390>
assert A.add10class(5) == 15 # prints <class '__main__.A'>
assert A.add10static(5) == 15 # prints self
assert A.add10(2, 3) == 5 # prints 10 because the first positional argument is self..

Once we approve of the API I'll provide a full fledged patch.
msg201377 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-26 17:18
I like your suggestion of not providing __call__(), as I don't see a way to make it work with arbitrary underlying descriptors, and neither classmethod nor staticmethod is callable.

In terms of usage, I think this approach will be OK, as in practice I expect @classmethod, etc, will be applied to the initial method definition as appropriate, so they won't need to be inline except in test cases like these.

Additional test cases needed:

    A().add10class(5)
    A().add10static(5)
    A.add10(A(), 5)

All three should produce 15 as the result, and print the same thing as the following:

    A.add10class(5)
    A.add10static(5)
    A().add10(5)

A test method that uses a partial instance as the callable would also be helpful in assuring the edge cases are covered.

I believe it may be necessary to have a __get__ method something like:

    def _make_unbound_method(self, cls):
        def _unbound_method(*args, **kwds):
            call_keywords = self.keywords.copy()
            call_keywords.update(keywords)
            return self.func(*(self.args + args), **call_keywords)
        _unbound_method.__objclass__ = cls
        return _unbound_method


    def __get__(self, obj, cls):
        get = getattr(self.func, "__get__")
        if get is None:
            if obj is None:
                return self._make_unbound_method(cls)
            callable = self.func
        else:
            callable = get(obj, cls)
            if callable is self.func:
                return self._make_unbound_method(cls)
        return partial(callable, *self.args, **self.keywords)
msg201451 - (view) Author: alon horev (alonho) * Date: 2013-10-27 12:17
I think the following test demonstrates the API we're looking for.
1. Am I missing any functionality?
2. How does partialmethod relate to @absolutemethod?

from functools import partial

class partialmethod(object):

    def __init__(self, func, *args, **keywords):
        # func could be a descriptor like classmethod which isn't callable,
        # so we can't inherit from partial (it verifies func is callable)
        if isinstance(func, partialmethod):
            # flattening is mandatory in order to place cls/self before all other arguments
            # it's also more efficient since only one function will be called
            self.func = func.func
            self.args = func.args + args
            self.keywords = {}
            self.keywords.update(func.keywords)
            self.keywords.update(keywords)
        else:
            self.func = func
            self.args = args
            self.keywords = keywords
    
    def _make_unbound_method(self):
        def _method(*args, **keywords):
            keywords.update(self.keywords)
            cls_or_self, *rest = args
            call_args = (cls_or_self,) + self.args + tuple(rest)
            return self.func(*call_args, **keywords)
        return _method

    def __get__(self, obj, cls):
        get = getattr(self.func, "__get__", None)
        if get is None:
            if obj is None:
                return self._make_unbound_method()
            return partial(self._make_unbound_method(), obj) # returns a bound method
        else:
            callable = get(obj, cls)
            if callable is self.func:
                return self._make_unbound_method()
        return partial(callable, *self.args, **self.keywords)

class A(object):
    def add(self, x, y):
        return self, x + y

    add10 = partialmethod(add, 10)
    add10class = partialmethod(classmethod(add), 10)
    add10static = partialmethod(staticmethod(add), 'self', 10)

    return15 = partialmethod(add10, 5) # nested partialmethod
    
    add2partial = partial(add, x=2)
    return12 = partialmethod(add2partial, y=10) # partialmethod over partial
    
def test():
    cls = A
    instance = cls()

    assert instance.add10class(5) == (cls, 15)
    assert cls.add10class(5) == (cls, 15)

    assert instance.add10static(5) == ('self', 15)
    assert cls.add10static(5) == ('self', 15)
    
    assert instance.add10(5) == (instance, 15)
    assert cls.add10(instance, 5) == (instance, 15)
    
    assert instance.return15() == (instance, 15)
    assert cls.return15(instance) == (instance, 15)

    assert instance.return12() == (instance, 12)
    assert cls.return12(instance) == (instance, 12)

test()
msg201453 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-27 12:32
On 27 Oct 2013 22:17, "alon horev" <report@bugs.python.org> wrote:
>
>
> alon horev added the comment:
>
> I think the following test demonstrates the API we're looking for.
> 1. Am I missing any functionality?

The keyword arg handling is backwards for unbound methods (the call time
kwds should override the preset ones).

Otherwise, looks good to me.

> 2. How does partialmethod relate to @absolutemethod?

Do you mean @abstractmethod?

We need a __isabstractmethod__ property implementation that delegates the
question to the underlying descriptor.

See http://docs.python.org/dev/library/abc#abc.abstractmethod for details.
msg201592 - (view) Author: alon horev (alonho) * Date: 2013-10-29 00:10
Adding a patch with tests and documentation. Please feel free to comment on anything: my English, coding/testing style. 

I'm pretty sure the documentation can be better but it turns out I speak better Python than English.

Two decisions I've made and unsure of:
1. I didn't use @wraps or copied attributes from the wrapped function  (__doc__, __dict__) to the partialmethod object. This is intentionally so partial and partialmethod would have similar semantics.
2. I've implemented a __repr__ although in all cases __get__ returns a partial object or bound method. I consider it nice for debugging when looking at an object's __dict__.
msg201633 - (view) Author: alon horev (alonho) * Date: 2013-10-29 13:11
I've changed the test according to the code review. Thanks
msg201816 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-10-31 13:54
Updated patch based on Alon's last patch.

The major functional change is to ensure __self__ is set appropriately on any bound methods returned by the descriptor.

I also updated the docs and docstring, and added a What's New entry (as well as rewording the existing entry for functools.singledispatch)

There were a few other cosmetic changes, with the most noticeable being moving the partialmethod implementation and tests adjacent to the existing ones for functools.partial.

Assuming nobody pokes any significant holes in this idea and implementation in the meantime, I'll commit this before beta 1.
msg201998 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-03 06:43
New changeset 46d3c5539981 by Nick Coghlan in branch 'default':
Issue #4331: Added functools.partialmethod
http://hg.python.org/cpython/rev/46d3c5539981
msg202084 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-11-04 04:27
Should we add partialmethod to __all__ for consistency?
msg202135 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-11-04 13:34
Indeed, added to __all__ in http://hg.python.org/cpython/rev/ac1685661b07
History
Date User Action Args
2013-11-04 13:34:43ncoghlansetmessages: + msg202135
2013-11-04 04:27:13vajraskysetnosy: + vajrasky
messages: + msg202084
2013-11-03 06:43:39ncoghlansetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2013-11-03 06:43:01python-devsetnosy: + python-dev
messages: + msg201998
2013-10-31 13:54:40ncoghlansetfiles: + issue4331_partialmethod.diff

messages: + msg201816
2013-10-31 12:06:08anacrolixsetnosy: - anacrolix
2013-10-31 12:01:27ncoghlansetstage: needs patch -> commit review
2013-10-29 13:11:29alonhosetfiles: + 4331.v2.patch

messages: + msg201633
2013-10-29 10:55:15ncoghlansetassignee: ncoghlan
2013-10-29 00:10:27alonhosetfiles: + 4331.v1.patch

messages: + msg201592
2013-10-27 12:32:22ncoghlansetmessages: + msg201453
2013-10-27 12:17:03alonhosetmessages: + msg201451
2013-10-26 17:18:38ncoghlansetmessages: + msg201377
2013-10-26 14:59:35alonhosetmessages: + msg201366
2013-10-26 01:57:43ncoghlansetmessages: + msg201310
2013-10-25 19:27:55alonhosetnosy: + alonho
messages: + msg201279
2013-10-24 10:33:01ncoghlansetmessages: + msg201111
stage: needs patch
2013-06-05 10:00:50anacrolixsetmessages: + msg190648
2013-06-05 07:42:14ncoghlansetmessages: + msg190645
2013-06-05 04:48:41ncoghlansettitle: Can't use _functools.partial() created function as method -> Add functools.partialmethod
nosy: + ncoghlan

messages: + msg190642

versions: - Python 2.7, Python 3.3
components: + Library (Lib)
2013-02-25 15:02:11r.david.murraysetmessages: + msg182946
2013-02-25 14:19:14eckhardtsetmessages: + msg182943
2013-02-08 09:20:10Ramchandra Aptesetversions: + Python 3.3, Python 3.4, - Python 3.0
2013-02-08 04:02:49anacrolixsetmessages: + msg181658
2013-02-07 08:36:05eckhardtsetnosy: + eckhardt
messages: + msg181582
2012-03-24 18:23:40anacrolixsetfiles: + functools.partial-descrget.patch

messages: + msg156712
2012-03-24 16:14:44anacrolixsetnosy: + anacrolix
2011-12-11 01:28:07jceasetnosy: + jcea
2010-02-23 21:37:15jackdiedsetmessages: + msg99957
2010-02-23 21:35:44jackdiedsetmessages: + msg99956
2010-02-22 19:11:42r.david.murraysetnosy: + r.david.murray
messages: + msg99813
2010-02-22 16:33:51jackdiedsetmessages: + msg99776
2010-02-18 21:09:22jackdiedsetnosy: + jackdied
2010-02-01 19:09:23Alexander.Belopolskysetmessages: + msg98675
2010-01-29 00:25:25Alexander.Belopolskysetnosy: + Alexander.Belopolsky
messages: + msg98490
2010-01-09 20:59:31Christophe Simonissetfiles: + issue4331.patch
keywords: + patch
messages: + msg97470
2009-03-05 20:23:48Christophe Simonissetnosy: + Christophe Simonis
2008-12-03 05:03:56belopolskysetnosy: + belopolsky
2008-11-16 23:03:10rhettingersettype: behavior -> enhancement
messages: + msg75946
nosy: + rhettinger
versions: + Python 3.0, Python 2.7, - Python 2.6, Python 2.5
2008-11-16 22:44:28ironfroggysetnosy: + ironfroggy
messages: + msg75945
2008-11-16 20:24:31ssadlersetmessages: + msg75942
2008-11-16 06:32:58ssadlercreate