This patch takes the existing "simplegeneric" decorator, currently an
internal implementation detail of the pkgutil module, and exposes it as
a feature of the functools module.

Documentation and tests have been added, and the pkgutil code has been
updated to use the functools implementation.

Open issue: The syntax for registering an overload is rather manual:

    def xxx_impl(xxx):
        pass
    generic_fn.register(XXX, xxx_impl)

It might be better to make the registration function a decorator:

    @generic_fn.register(XXX)
    def xxx_impl(xxx):
        pass

However, this would involve changing the existing (working) code, and I
didn't want to do that before there was agreement that the general idea
(of exposing the functionality) was sound.

PJE seems to have borrowed the time machine :-). Based on the code the
register function is already a decorator:

    def register(typ, func=None):
        if func is None:
            return lambda f: register(typ, f)
        registry[typ] = func
        return func

The returned lambda is a one argument decorator. so your syntax:

    @generic_fn.register(XXX)
    def xxx_impl(xxx):
        pass

Already works. A test to validate this behavior should probably be added.

I don't mean to bikeshed, but could we call this function
functools.generic instead of functools.simplegeneric? The only reason I
can think of for keeping it simplegeneric would be to avoid a future
name clash with the Generic Function PEP and if/when that PEP get's
implemented, I would think that the functionality would live in
builtins, not functools.

Well spotted! I missed that when I checked. I will add tests and
documentation.

I agree that generic is better. I only left it as it was because the
original intent was simply to move the existing code - but that's not a
particularly good reason for keeping a clumsy name. There shouldn't be a
clash, as any more general mechanism can either be in its own module or
the existing function can be extended in a compatible manner. I'll make
this change too.

Thanks for the feedback!

Here's an updated patch.

The reason I like the simplegeneric name is that that is exactly what
this feature is: a *simple* generic implementation that is deliberately
limited to dispatching on the first argument (because that is easily
explained to users that are already familiar with OOP and especially the
existing Python magic method dispatch mechanism.

So the name isn't just about avoiding name clashes, it's also about
setting appropriate expectations as to what is supported. Yes, the name
is a little clumsy but one thing I do *not* want to see happen is a
swathe of feature requests asking that this become an all-singing
all-dancing generic function mechanism like RuleDispatch.

Don't forget that actually *writing* generic functions (i.e. using the
@functools.simplegeneric decorator itself) should be far less common
than using the .register() method of existing generic functions.

Fair comment. As Ryan said, it's a bit of a bikeshed issue. I prefer
"generic", on the basis that I'd prefer to keep the simple name for the
simple use - something as complex as the RuleDispatch version could use
the name "dispatch" (if they want to keep it the name simple) or
"multimethod" (to emphasize that it dispatches on more than just one
argument).

If the consensus is for keeping it as "simplegeneric", I'll go with
that, but I'll wait for other views first.

BTW, another option is simply to clarify the limitations in the
documentation. I can add something there if that would be enough for you.

The patch looks fine to me. Tests pass.

I have no opinion about the name. Both "simplegeneric" and "generic" are
OK to me.

I wonder if being able to use register() directly instead of as a
decorator should be dropped.

Also IMHO the Python 2.3 backwards compatibility (__name__ isn't
setable) can be dropped.

Agreed about the compatibility. It's there from pkgutil, where to be
honest, it's even less necessary, as simplegeneric was for internal use
only, there. I'm certainly not aware of any backward compatibility
requirements for functools.

Assuming nobody speaks up to the contrary, I'll rip out the
compatibility bits in the next version of the patch.

I'm unsure about the non-decorator version of register. I can imagine
use cases for it - consider pprint, for example, where you might want to
register str as the overload for your particular type. But it's not a
big deal either way.

I think that registering existing functions is an important use case, so
I vote for keeping the non-decorator version of register.

Another thing that we may want to document is that [simple]generic
doesn't dispatch based on registered abstract base classes.

>>> class A:
...     pass
...
>>> class C:
...     __metaclass__ = abc.ABCMeta
...
>>> C.register(A)
>>> @generic
... def pprint(obj):
...     print str(obj)
...
>>> @pprint.register(C)
... def pprint_C(obj):
...     print "Charlie", obj
...
>>> pprint(C())
Charlie <__main__.C object at 0xb7c5336c>
>>> pprint(A())
<__main__.A instance at 0xb7c5336c>

Failure to respect isinstance() should be fixed, not documented :)

As far as registering existing functions goes, I also expect registering
lambdas and functools.partial will be popular approaches, so keeping
direct registration is a good idea. There isn't any ambiguity between
the one-argument and two-argument forms.

Agreed (in principle). However, in practice the subtleties of override
order must be documented (and a method of implementation must be
established!!!) Consider:

>>> class A:
...     pass
...
>>> class C:
...     __metaclass__ = abc.ABCMeta
...
>>> class D:
...     __metaclass__ = abc.ABCMeta
...
>>> C.register(A)
>>> D.register(A)
>>> @generic
... def pprint(obj):
...     print "Base", str(obj)
...
>>> @pprint.register(C)
... def pprint_C(obj):
...     print "Charlie", obj
...
>>> @pprint.register(D)
... def pprint_D(obj):
...     print "Delta", obj
...
>>> pprint(A())

What should be printed? A() is a C and a D, but which takes precedence?
There is no concept of a MRO for ABCs, so how would the "correct" answer
be defined? "Neither" may not be perfect, but at least it's clearly
defined. Relying on order of registration for overloads of the generic
function seems to me to be unacceptable, before anyone suggests it, as
it introduces a dependency on what order code is imported.

So while the theory makes sense, the practice is not so clear.
Respecting ABCs seems to me to contradict the "simple" aspect of
simplegeneric, so a documented limitation is appropriate. 

(But given the above, I'm more inclined now to leave the name as
"simplegeneric", precisely to make this point :-))

Hmm, there is such a thing as being *too* simple... a generic function
implementation that doesn't even respect ABCs seems pretty pointless to
me (e.g. I'd like to be able to register a default Sequence
implementation for pprint and have all declared Sequences use it
automatically if there isn't a more specific override).

I'll wait until I have a chance to actually play with the code a bit
before I comment further though.

Very good point. Registering for the standard ABCs seems like an
important use case. Unfortunately, it seems to me that ABCs simply don't
provide that capability - is there a way, for a given class, of listing
all the ABCs it's registered under? Even if the order is arbitrary,
that's OK.

Without that, I fail to see how *any* generic function implementation
("simple" or not) could support ABCs. (Excluding obviously broken
approaches such as registration-order dependent overload resolution).

The problem is that ABCs are all about isinstance testing, where generic
functions are all about *avoiding* isinstance testing. (As a compromise,
you could have a base generic function that did isinstance testing for
the sequence ABC).

Even more inconveniently, the existence of unregister() on ABCs makes it
difficult for the generic to cache the results of the isinstance()
checks (you don't want to be going through the chain of registered ABCs
every time calling isinstance(), since that would be painfully slow).

That said, it is already the case that if you only *register* with an
ABC, you don't get any of the methods - you have to implement them
yourself. It's only when you actually *inherit* from the ABC that the
methods are provided "for free". I guess the case isn't really any
different here - if you changed your example so that A inherited from C
and D rather than merely registering with them, then C & D would appear
in the MRO and the generic would recognise them.

So perhaps just documenting the limitation is the right answer after all.

Here's an updated patch. I've reverted to the name "simplegeneric" and
documented the limitation around ABCs (I've tried to give an explanation
why it's there, as well as a hint on now to work around the limitation -
let me know if I'm overdoing it, or the text needs rewording).

I've also fixed the wrapper to use update_wrapper to copy the
attributes. That way, there's no duplication.

Unassigning - the lack of support for ABC registration still bothers me,
but a) I don't have a good answer for it, and b) I'm going to be busy
for a while working on some proposed changes to the with statement.

The problem with generic functions supporting ABCs is it's a bug with
the way ABCs work and not a problem with the generic function
implementation. The register() method of an ABC only fakes out
isinstance checks, it doesn't actually make the abstract base class a
base class of the class. It doesn't make any sense for a class to say it
is an instance of an ABC, but not have that ABC in it's MRO. It's not a
base class if it's not in the MRO!

The documentation for lack of ABC support should read something like:

+ Note that generic functions do not work with classes which have
+ been declared as an abstract base class using the
+ abc.ABCMeta.register() method because this method doesn't make
+ that abstract base class a base class of the class - it only fakes
+ out instance checks.

Perhaps a bug should be opened for the abc.ABCMeta.register() method.

However, I'd say that just because virtual abstract base classes are
wonky doesn't mean that a solid generic function implementation
shouldn't be added to standard library.

I raised issue 5405. Armin Ronacher commented over there that it's not
even possible in principle to enumerate the ABCs a class implements
because ABCs can do semantic checks (e.g., checking for the existence of
a special method).

So documenting the limitation is all we can manage, I guess.

Given the point Armin raised, I once again agree that documenting the
limitation is a reasonable approach. Longer-term, being able to subcribe
to ABCs (and exposing the registration list if it isn't already visible)
is likely to be the ultimate solution.

Please don't introduce this without a PEP.

Changes as Guido has stated that he wants a PEP.

I don't propose to raise a PEP myself. The issue with ABCs seems to me to be a fundamental design issue, and I think it's better to leave raising any PEP, and managing the subsequent discussion, to someone with a deeper understanding of, and interest in, generic functions.

Not sure if the lack of a champion means that this issue should be closed. I'm happy if that's the consensus (but I'm also OK with it being left open indefinitely, until someone cares enough to pick it up).

Generic functions are a lesser-known paradigm than OO, and nowhere do common Python documents (including the official docs) try to teach them. That means the first public appearance of generic functions in the stdlib should really be well thought out if we don't want to encourage poor practices. I agree with Guido that a PEP is required to flesh out all the details.

An elaborate PEP for generic functions already exists, PEP 3124 [
http://www.python.org/dev/peps/pep-3124/]. Also note the reasons for
deferment. I'd be interested in creating a "more limited" generic function
implementation based on this PEP, minus func_code rewriting and the other
fancier items. Sadly I won't have any bandwidth to work on it until January
of next year.

I'd vote for keeping this issue open because of that.

On Thu, Jul 22, 2010 at 5:45 AM, Antoine Pitrou <report@bugs.python.org>wrote:

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Generic functions are a lesser-known paradigm than OO, and nowhere do
> common Python documents (including the official docs) try to teach them.
> That means the first public appearance of generic functions in the stdlib
> should really be well thought out if we don't want to encourage poor
> practices. I agree with Guido that a PEP is required to flesh out all the
> details.
>
> ----------
> nosy: +pitrou
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue5135>
> _______________________________________
>

A couple more relevant links.

I brought this issue up in the context of a JSON serialisation discussion on python-ideas: http://mail.python.org/pipermail/python-ideas/2010-July/007854.html

Andrey Popp mentioned his pure Python generic functions library in that thread: http://pypi.python.org/pypi/generic

> The register() method of an ABC only fakes out isinstance checks, it
> doesn't actually make the abstract base class a base class of the class.
> It doesn't make any sense for a class to say it is an instance of an
> ABC, but not have that ABC in [its] MRO.

I disagree.  If someone writes a class and registers them with an ABC, it is their duty to make sure that the class actually complies.  Virtual subclasses are provided for use by consenting adults IMO.

Just as an FYI, it *is* possible to do generic functions that work with Python's ABCs (PEAK-Rules supports it for Python 2.6), but it requires caching, and a way of handling ambiguities.  In PEAK-Rules' case, unregistering is simply ignored, and ambiguity causes an error at call time.  But simplegeneric can avoid ambiguities, since it's strictly single-dispatch.  Basically, you just have two dictionaries instead of one.

The first dictionary is the same registry that's used now, but the second is a cache of "virtual MROs" you'll use in place of a class' real MRO.  The virtual MRO is built by walking the registry for classes that the class is a subclass of, but which are *not* found in the class's MRO, e.g.:

    for rule_cls in registry:
        if issubclass(cls, rule_cls) and rule_cls not in real_mro:
            # insert rule_cls into virtual_mro for cls

You then insert those classes (abcs) in the virtual MRO at the point just *after* the last class in the MRO that says it's a subclass of the abc in question.

IOW, you implement it such that an abc declaration appears in the MRO just after the class that was registered for it.  (This has to be recursive, btw, and the MRO cache has to be cleared when a new method is registered with the generic function.)

This approach, while not trivial, is still "simple", in that it has a consistent, unambiguous resolution order.  Its main downside is that it holds references to the types of objects it has been called with.  (But that could be worked around with a weak key dictionary, I suppose.)  It also doesn't reset the cache on unregistration of an abc subclass, and it will be a bit slower on the first call with a previously-unseen type.

The downside of a PEP, to me, is that it will be tempting to go the full overloading route -- which isn't necessarily a bad thing, but it won't be a *simple* thing, and it'll be harder to get agreement on what it should do and how -- especially with respect to resolution order.

Still, if someone wants to do a PEP on *simple* generics -- especially one that can replace pkgutil.simplegeneric, and could be used to refactor things like copy.copy, pprint.pprint, et al to use a standardized registration mechanism, I'm all in favor -- with or without abc registration support.

Btw, the current patch on this issue includes code that is there to support classic classes, and metaclasses written in C.  Neither should be necessary in 3.x.  Also, a 3.x version could easily take advantage of type signatures, so that:

@foo.register
def foo_bar(baz: bar):
    ...

could be used instead of @foo.register(bar, foo_bar).

But all that would be PEP territory, I suppose.

Thanks for the detailed explanation.  Note that type annotations are disallowed in the stdlib, as per PEP 8.

Guido said "Please don't introduce this without a PEP." and that has not happened and if it did, the result would probably look quite different from the patches. So this is 'unripe' and no current action here is possible.

For the record, this has been implemented as PEP 443.