Title: Argument Clinic: Fix signature of optional positional-only arguments
Type: Stage:
Components: Argument Clinic Versions: Python 3.8, Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: larry, remi.lapeyre, rhettinger, serhiy.storchaka, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2017-01-17 15:17 by vstinner, last changed 2019-03-15 19:50 by remi.lapeyre.

File name Uploaded Description Edit
ac_optional_positional.patch vstinner, 2017-01-17 15:17
getattr_ac.patch vstinner, 2017-01-17 15:25
Messages (8)
msg285650 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-17 15:17
When a function has only positional arguments and at least one argument is optional, the expected signature is:

  func(mandatory_arg1, mandatory_arg2[, optional_arg3[, optinal_arg4]])

For example, the signature of format() is inconsistent with its documentation.

$ python3 -c 'help(format)'|cat
Help on built-in function format in module builtins:

format(value, format_spec='', /)
    Return value.__format__(format_spec)
    format_spec defaults to the empty string

.. function:: format(value[, format_spec])

Attached patch is a first attempt to fix the issue. The problem is that my heuristic to check if an argument is "optional" doesn't seem to work as expected in all cases. I chose to check if the C default is NULL.

The problem is that some functions defines a C default to NULL whereas the Python default is set to a different value and is correct.

Example with _io.FileIO.truncate:

    /*[clinic input]
        size as posobj: object = NULL

whereas the documentation says that the default is None:

   .. method:: truncate(size=None)

It's easy to fix the default, but in this case my change doesn't fix the signature anymore since the C default is still NULL:

    /*[clinic input]
        size as posobj: object(c_default="NULL") = None

We need a different heuristic than C default is NULL, or we should fix functions where the heuristic fails.
msg285651 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-17 15:18
List of functions modified when Argument Clinic is run to regenerate .c.h files:

msg285656 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-17 15:21
See also issue #20291: "Argument Clinic should understand *args and **kwargs parameters".
msg285657 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-17 15:25
This issue is blocking me to convert more functions to Argument Clinic. See for example attached getattr_ac.patch which converts getattr() to AC. Without ac_optional_positional.patch, AC generates the signature:

   "getattr($module, object, name, default=None, /)\n"

whereas the following signature is expected:

   "getattr($module, object, name[, default])\n"
msg285659 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-17 15:34
The problem is that

  func(mandatory_arg1, mandatory_arg2[, optional_arg3[, optinal_arg4]])

is not compatible with the inspect module.

In many case a meaningful default value was added if this is possible. For example the Python default shown in the signature can be set to '', 'utf-8' or 'strict' while the C default value is NULL for performance. If the parameter is upper index in the sequence it can be set to sys.maxsize (Py_SSIZE_T_MAX in C).

This is not always possible. For example there is not default value for dict.pop().
msg285661 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-17 15:52
Please don't change this part of Argument Clinic without Larry. There were several attempts to solve this problem, I don't like current status, but this is Larry's decision.
msg338014 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2019-03-15 18:52
Argument Clinic can currently only generate signatures for functions whose signatures can be represented in Python.  This is because the signatures are parsed by the inspect module, which in turn uses the ast module to generate a parse tree; it then examines the parse tree and uses that to generate the Signature object.

(By the time you see a signature of a function using inspect, the signature has actually made a round-trip through the ast module, been turned into a Signature object, then str() has been run on it to re-generate the text representation from scratch.  The fact that it's usually identical to the original text signature buried in the function's docstring is a happy accident.)

Changing Argument Clinic to generate signatures with square brackets in them to signify optional parameters is insufficient.  You'd also have to upgrade inspect to support this new syntax--otherwise there'd be no point.  And *that* would be a lot of code.
msg338024 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-03-15 19:50
Thanks for the explanation.

> And *that* would be a lot of code.

Would an effort to make inspect support this be appreciated or do you think this is not important?
Date User Action Args
2019-03-15 19:50:22remi.lapeyresetnosy: + remi.lapeyre

messages: + msg338024
versions: + Python 3.8
2019-03-15 18:52:19larrysetmessages: + msg338014
2019-03-15 17:24:08SilentGhostlinkissue36306 superseder
2017-01-17 15:52:04serhiy.storchakasetmessages: + msg285661
2017-01-17 15:34:50serhiy.storchakasetnosy: + yselivanov
messages: + msg285659
2017-01-17 15:25:17vstinnersetfiles: + getattr_ac.patch

messages: + msg285657
2017-01-17 15:21:44vstinnersetmessages: + msg285656
2017-01-17 15:18:15vstinnersetmessages: + msg285651
2017-01-17 15:17:14vstinnercreate