classification
Title: Add a "starcaller" function
Type: Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mariatta, josh.r, ncoghlan, rhettinger, steven.daprano
Priority: normal Keywords:

Created on 2016-10-06 21:18 by josh.r, last changed 2016-10-07 00:07 by steven.daprano.

Messages (3)
msg278212 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2016-10-06 21:18
Not sure if this is the right venue to propose this, but I'd like to propose adding a starcaller method to the standard library, either functools or operator (not sure if the proposal is more like partial or more like methodcaller).

Basically, right now, when you want to take an existing function and call it repeatedly with preconstructed tuples using functional tools (e.g. sorted or groupby's key argument, multiprocessing.Pool's various map-like methods), you need to either have a special starmethod (e.g. Pool.map vs. Pool.starmap), or if Python doesn't provide one, you need use Python-defined functions (and for stuff like Pool, you can't even use lambda due to pickling issues, so you need to define the utility somewhere else). This means that you can't reuse builtins for stuff like the old "find runs of consecutive numbers" example from the itertools docs ( https://docs.python.org/2.6/library/itertools.html#examples ).

In terms of behavior:

    starfunc = starcaller(func)

would be behave roughly the same as the following (less efficient) Python 2 code:

    starfunc = functools.partial(apply, func)

(and perhaps starcaller(func, kwonly=True) would behave like functools.partial(apply, func, ()) to extend it to dicts).

This would make it possible to write the old consecutive numbers example:

    for k, g in groupby(enumerate(data), lambda (i,x):i-x):

(which is no longer legal since Py3 banned sequence unpacking in arguments like that) as:

    for k, g in groupby(enumerate(data), starcaller(operator.sub)):

It would also mean that instead of constantly needing to write new "star methods", you can just reuse existing functions with new batteries. For example, multiprocessing.Pool has a map and starmap function, but imap and imap_unordered have no starimap or starimap_unordered equivalents, which means using them at all requires you to def a function at trivial wrapper at global scope (possibly a long way away from the point of use) just to use an existing function ( def whydidineedthis(args): return existingfunction(*args) ).

With starcaller, there is no need to add starimap and friends, because the same result is easily achieved with:

    with multiprocessing.Pool() as pool:
        for res in pool.imap(starcaller(existingfunction), ...):

Is this a reasonable proposal? I could get an implementation ready fairly quickly, I'd just need to feedback on the name, which module it should be put in (functools or operator seem equally plausible) and whether the idea of using a keyword argument to the constructor makes more sense (kwonly=True) vs. expecting users to do functools.partial(starcaller(existingfunc), ()), vs. a separate class like doublestarcaller.
msg278220 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2016-10-06 22:26
Hi Josh,

I think python ideas mailing list might have been a better venue for this.
https://mail.python.org/mailman/listinfo/python-ideas
msg278222 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-10-07 00:07
This was discussed on Python-Ideas back in July:

https://mail.python.org/pipermail/python-ideas/2016-July/041153.html

I don't recall any opposition, although Nick suggested that possibly a better idea was to resurrect the `apply` built-in into functools:

https://mail.python.org/pipermail/python-ideas/2016-July/041159.html
History
Date User Action Args
2016-10-07 00:07:31steven.dapranosetnosy: + steven.daprano, ncoghlan
messages: + msg278222
2016-10-06 22:32:53ned.deilysetnosy: + rhettinger
2016-10-06 22:26:24Mariattasetnosy: + Mariatta
messages: + msg278220
2016-10-06 21:18:28josh.rcreate