inspect.getfullargspec (and potentially getargspec?) could *use* function.__signature__ if set. This allows functions that lie about their signature with the new introspection tool (Signature) to still work with older code.

Please consider the attached patch (getargsspec_01.patch).

It modifies 'getargspec' and 'getfullargspec' to use the 'inspect.signature' API. The entire test suite passes just fine.

This also will address issue #16490.

I can also update the docs, if it's necessary.

Can somebody review the patch? I'd be cool if this lands in 3.4.

While I plan to change Idle code to use signature directly, instead of getfullargspec, I agree that changing inspect also, and in 3.4, is a good idea. It may, however, affect tests other than Idle's, if there are any other stdlib consumers. I will try to look at the patch after changing Idle.

I upgraded the description to a "should". Argument Clinic and other changes in Python 3.4 greatly improve introspection support for various aspects of the runtime and standard library (for example, issue 20223 will handle the new functools.partialmethod support). By also making those enhancements available via getfullargpsec, we should significantly increase the amount of existing code which benefits automatically for those enhancements, as it will no longer require explicit migration to the new PEP 362 APIs at the application/library level.

Reviewing the patch now.

If we modify inspect.getfullargspec, shouldn't we modify inspect.getargspec too?  "deprecated" doesn't mean "unsupported", it means "not recommended for new code, please stop using it".

Larry, 
getargspec uses getfullargspec, so it's covered.

getargspec() is just a thin wrapper around getfullargspec(), so we get that for free.

Similarly, getcallargs() is also implicitly updated to full PEP 362 support by changing the implementation of getfullargspec().

The other two standard library consumers appear to be IDLE (as Terry noted above) and xmlrpc.server.ServerHTMLDoc.docroutine (the self-documenting XML-RPC server API, which sadly appears to be lacking test cases).

Reviewing those cases (especially the XML-RPC one) have convinced me we need a backwards compatibility fallback in the updated getfullargspec implementation. Specifically, if the new API throws ValueError (because it can't find a way to build a coherent signature object), we should fall back to the old approach. Otherwise we run the risk of introducing unexpected exceptions into introspection code.

That is, the new call to signature in getfullargspec should look something like:

    try:
        sig = signature(func)
    except ValueError:
        if not hasattr(func, "__code__"):
            raise # The legacy fallback doesn't support this input type
        # The callable signature appears to be incoherent, so we fall
        # back to the legacy implementation to preserve compatibility
        args, varargs, kwonlyargs, varkw = _getfullargs(func.__code__)
        return FullArgSpec(args, varargs, varkw, func.__defaults__,
                kwonlyargs, func.__kwdefaults__, func.__annotations__)

I suspect this may actually be a better solution for IDLE rather than updating it to call inspect.signature directly (since IDLE probably *wants* the more graceful fallback to the old behaviour). 

The way I would document all this in What's New is to update the current argument clinic section to also explain that we've taken advantage of PEP 362 to improve introspection in multiple areas, including for code that is using the introspection APIs that predate PEP 362.

The relevant code in CallTips.py is
  argspec = ""
  if hasattr(ob, '__call__'):
...
    if isinstance(fob, (types.FunctionType, types.MethodType)):
      argspec = inspect.formatargspec(*inspect.getfullargspec(fob))

So I want to broaden the second condition (or remove it), but if it returns something .formatargspec cannot digest, the outer call will have to be conditioned. My impression is that str(signature object) returns pretty much what .formatargspec does, but I have not experimented yet.

My #20122 patch moves currently commented out tests in CallTips.py that use the above to test_calltips.py. I will commit as soon as I test on 3.4. Until then, getfullargspec is not being used in the Idle test suite.

I just want to mention, while we're all thinking about this stuff: I plan to enhance the Signature object to reflect "optional groups".  Right now Signature objects cannot express parameters that are optional but don't have a published default.  (e.g. a string parameter for a builtin having a default of NULL).  A lot of these are being converted using optional groups, and I want to make it at least possible for user code to introspect those functions and understand that those are optional parameters.

My original plan was to add a "group" member, containing an arbitrary identifier of the "group" this parameter belongs to.  I'm not sure that's sufficient; I may also need a "parent_group" parameter, or something like that, to represent the group that this group is nested in.  I'm still thinking about it.  But I'm hoping to get to this in the next two or three days.

Sounds reasonable - we'll need that to convert range() and slice()
anyway (I totally failed at finding time to look at the builtins this
weekend, but I'd still like to handle those before the 3rd beta).

An alternative would be to actually add "ParameterGroup" as a new
class that contains parameters (assuming tuple isn't sufficient), so
the structure of the parameters more directly matches the underlying
signature.

Larry,  

I saw your message on the tracker regarding adding support for parameters groups to the signature object. Would you mind if I join the discussion with my ideas of how this feature might be implemented? 

Yury

On Sunday, January 19, 2014 at 5:44 AM, Larry Hastings wrote:

> 
> Larry Hastings added the comment:
> 
> I just want to mention, while we're all thinking about this stuff: I plan to enhance the Signature object to reflect "optional groups". Right now Signature objects cannot express parameters that are optional but don't have a published default. (e.g. a string parameter for a builtin having a default of NULL). A lot of these are being converted using optional groups, and I want to make it at least possible for user code to introspect those functions and understand that those are optional parameters.
> 
> My original plan was to add a "group" member, containing an arbitrary identifier of the "group" this parameter belongs to. I'm not sure that's sufficient; I may also need a "parent_group" parameter, or something like that, to represent the group that this group is nested in. I'm still thinking about it. But I'm hoping to get to this in the next two or three days.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org (mailto:report@bugs.python.org)>
> <http://bugs.python.org/issue17481>
> _______________________________________

Yuri, I am sure your ideas for enhancing signature objects would be welcome. Either a new issue or a thread on pydev or python-ideas lists would be best.

When responding by email, please snip the quotation and footer, except possibly a line of the quoted message.

Terry,

Thanks. 

> When responding by email, please snip the quotation and footer, except possibly a line of the quoted message.

My email client played an evil (and a bit embarrassing) trick with me, showing Larry's name without an actual email address, which was "report@bugs.python.org". Won't happen again.

> Otherwise we run the risk of introducing unexpected exceptions into introspection code.

That's a good catch. I'll make a new patch, keeping the old implementation of getfullargsspec intact, and falling back to it if no signature can be found.

Yury: fire away, either here or in a new issue as is best.  (Sorry, brain mildly fried, not sure what would be best issue-tracker hygiene.)

>> Otherwise we run the risk of introducing unexpected exceptions into introspection code.

> That's a good catch. I'll make a new patch, keeping the old implementation of getfullargsspec intact, and falling back to it if no signature can be found.

Nick, while I was working on the second patch (writing a new unittest for it specifically), I realized, that it's not that easy to make the old version of "getfullargsspec" to spit out any exception that it doesn't currently do with the proposed 'getargsspec_01.patch'.

See, the old "getfullargsspec" does the following:

1. Check if the passed object is a function, with 'inspect.isfunction'. If not - throw a TypeError.  That behaviour is duplicated in the patch, so we are safe here.

2. Call on the object's __code__ '_getfullargs', which validates that the passed code object is a valid code object, and simply returns its attributes rearranged a bit.

Now, to have any exception in (2), we need: either a broken __code__ object, or something that is an instance of "types.FunctionType" (hence, defined in python) but doesn't have the "__code__" attribute. And that's kind of hard to achieve.

> Yury: fire away, either here or in a new issue as is best.  (Sorry, brain mildly fried, not sure what would be best issue-tracker hygiene.)

Larry, I ended up with a relatively big core dump of my thoughts, so I decided to post it on python-dev. Let's discuss it there, and we can create an issue here once we decide something.

Ah, I had indeed missed the fact that getfullargspec() was still calling isfunction(). In that case, is the patch currently actually buying us anything much beyond handling __signature__ attributes? Most of the new types that inspect.signature() supports will still fail that preliminary check, so code will need to use the new API explicitly in order to benefit from it.

By contrast, if we remove the check, then there's a wider range of exceptions that may be thrown, but also a much wider variety of inputs supported.

Instead of keeping the check, we could just unconditionally convert exceptions from the signature call to a TypeError in order to maintain compatibility with the old external behaviour.

> Instead of keeping the check, we could just unconditionally convert exceptions from the signature call to a TypeError in order to maintain compatibility with the old external behaviour.

Agreed. See the new patch (getargsspec_02.patch)

Unfortunately, we have to keep the old 'ismethod' check in place for backwards compatibility purposes.

Larry,

The attached patch contains one failing unit-test: 'inspect.signature' returns 'None' for '_testapi.docstring_no_signature'. It should instead raise a ValueError.

Larry,

I created a separate issue for that: http://bugs.python.org/issue20313

Larry, Nick,

So what's the resolution on this one? Do I have a green light?

My solution for pydoc was to call isroutine() instead of isfunction(), and yes handle it throwing an exception.

(I just checked, and I wasn't catching TypeError, only ValueError.  I'll fix that in my next patch for #20189.)

Another case of "don't land it until Larry has dealt with the builtins".

The patch itself looks fine, though :)

The patch from #20189 has landed.

There's a major difference between getfullargspec/getargspec and inspect.signature: getfullargspec shows you the "self" parameter for bound methods, and inspect.signature does not.

>>> class C:
...    def foo(self, a):  pass
... 
>>> c = C()
>>>
>>> import inspect
>>> str(inspect.signature(c.foo))
'(a)'
>>> inspect.getfullargspec(c.foo)
FullArgSpec(args=['self', 'a'], varargs=None, varkw=None, defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={})
>>> inspect.getargspec(c.foo)
ArgSpec(args=['self', 'a'], varargs=None, keywords=None, defaults=None)

This is why help() (currently) shows bound parameters for methods implemented in Python, but doesn't show them for methods implemented in C.  pydoc uses inspect.getfullargspec for pure Python functions and inspect.signature for builtins.

> There's a major difference between getfullargspec/getargspec and inspect.signature: getfullargspec shows you the "self" parameter for bound methods, and inspect.signature does not.

Larry, yes, that's correct. The attached patch simulates this behaviour, with:

    if ismethod(func):
        func = func.__func__

I'm attaching 'getargsspec_03.patch', as the previous one (02) was a bit crippled.

Larry and Nick,

Please review the final patch (getargsspec_04.patch). I'd like to commit it tomorrow.

This one is the last one (hopefully), and it supports builtin methods properly -- i.e. works around 'self' parameter correctly. To do that, I check if the '__text_signature__' starts with "($", etc.

New changeset 6d1e8162e855 by Yury Selivanov in branch 'default':
inspect.getfullargspec: Use inspect.signature API behind the scenes #17481
http://hg.python.org/cpython/rev/6d1e8162e855

Committed.
Thanks for the reviews!

New changeset 0fa2750c7241 by Yury Selivanov in branch 'default':
inspect.test.getfullargspec: Add a unittest to ensure correct annotations
http://hg.python.org/cpython/rev/0fa2750c7241