Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate vectorcall code to parse arguments using Argument Clinic #87613

Open
vstinner opened this issue Mar 9, 2021 · 7 comments
Open

Generate vectorcall code to parse arguments using Argument Clinic #87613

vstinner opened this issue Mar 9, 2021 · 7 comments
Assignees
Labels
3.13 new features, bugs and security fixes topic-argument-clinic topic-C-API type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

vstinner commented Mar 9, 2021

BPO 43447
Nosy @vstinner, @serhiy-storchaka, @corona10, @erlend-aasland, @kumaraditya303

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/corona10'
closed_at = None
created_at = <Date 2021-03-09.11:34:23.944>
labels = ['expert-C-API', '3.10']
title = 'Generate vectorcall code to parse arguments using Argument Clinic'
updated_at = <Date 2022-03-06.09:38:05.594>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-03-06.09:38:05.594>
actor = 'kumaraditya'
assignee = 'corona10'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-03-09.11:34:23.944>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43447
keywords = []
message_count = 2.0
messages = ['388354', '389214']
nosy_count = 5.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'corona10', 'erlendaasland', 'kumaraditya']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43447'
versions = ['Python 3.10']

@vstinner
Copy link
Member Author

vstinner commented Mar 9, 2021

To optimize the creation of objects, a lot of "tp_new" methods are defined twice: once in the legacy way (tp_new slot), once with the new VECTORCALL calling convention (tp_vectorcall slot). My concern is that the VECTORCALL implementation copy/paste most of the code just to parse arguments, whereas the specific code is just a few lines.

Example with the float type constructor:
--------------

static PyObject *
float_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
    PyObject *return_value = NULL;
    PyObject *x = NULL;

    if ((type == &PyFloat_Type) &&
        !_PyArg_NoKeywords("float", kwargs)) {
        goto exit;
    }
    if (!_PyArg_CheckPositional("float", PyTuple_GET_SIZE(args), 0, 1)) {
        goto exit;
    }
    if (PyTuple_GET_SIZE(args) < 1) {
        goto skip_optional;
    }
    x = PyTuple_GET_ITEM(args, 0);
skip_optional:
    return_value = float_new_impl(type, x);

exit:
    return return_value;
}

/*[clinic input]
@classmethod
float.\_\_new__ as float_new
    x: object(c_default="NULL") = 0
    /

Convert a string or number to a floating point number, if possible.
[clinic start generated code]*/

static PyObject *
float_new_impl(PyTypeObject *type, PyObject *x)
/*[clinic end generated code: output=ccf1e8dc460ba6ba input=f43661b7de03e9d8]*/
{
    if (type != &PyFloat_Type) {
        if (x == NULL) {
            x = _PyLong_GetZero();
        }
        return float_subtype_new(type, x); /* Wimp out */
    }

    if (x == NULL) {
        return PyFloat_FromDouble(0.0);
    }
    /* If it's a string, but not a string subclass, use
       PyFloat_FromString. */
    if (PyUnicode_CheckExact(x))
        return PyFloat_FromString(x);
    return PyNumber_Float(x);
}

static PyObject *
float_vectorcall(PyObject *type, PyObject * const*args,
                 size_t nargsf, PyObject *kwnames)
{
    if (!_PyArg_NoKwnames("float", kwnames)) {
        return NULL;
    }

    Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
    if (!_PyArg_CheckPositional("float", nargs, 0, 1)) {
        return NULL;
    }

    PyObject *x = nargs >= 1 ? args[0] : NULL;
    return float_new_impl((PyTypeObject *)type, x);
}

Here the float_new() function (tp_new slot) is implemented with Argument Clinic: float_new() C code is generated from the [clinic input] DSL: good!

My concern is that float_vectorcall() code is hand written, it's boring to write and boring to maintain.

Would it be possible to add a new [clinic input] DSL for vectorcall? I expect something like that:
--------------

static PyObject *
float_vectorcall(PyObject *type, PyObject * const*args,
                 size_t nargsf, PyObject *kwnames)
{
    if (!_PyArg_NoKwnames("float", kwnames)) {
        return NULL;
    }

    Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
    if (!_PyArg_CheckPositional("float", nargs, 0, 1)) {
        return NULL;
    }

    PyObject *x = nargs >= 1 ? args[0] : NULL;
    return float_vectorcall_impl(type, x);
}

static PyObject *
float_vectorcall_impl(PyObject *type, PyObject *x)
{
    return float_new_impl((PyTypeObject *)type, x);
}

where float_vectorcall() C code would be generated, and float_vectorcall_impl() would be the only part written manually. float_vectorcall_impl() gets a clean API and its body is way simpler to write and to maintain!

@vstinner vstinner added 3.10 only security fixes topic-C-API labels Mar 9, 2021
@corona10
Copy link
Member

I agree with we should update AC to generate vectorcall.
I am going to investigate what we can :)

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@vstinner
Copy link
Member Author

vstinner commented Nov 3, 2022

I don't have the bandwidth to work on this issue, so I just close it.

@erlend-aasland
Copy link
Contributor

I agree with we should update AC to generate vectorcall.
I am going to investigate what we can :)

Dong-hee, did you ever get to this? :) If so, I'd be interested in hearing what you found out.

@erlend-aasland erlend-aasland added the 3.13 new features, bugs and security fixes label Aug 17, 2023
@corona10
Copy link
Member

corona10 commented Aug 17, 2023

Dong-hee, did you ever get to this? :) If so, I'd be interested in hearing what you found out.

Sorry, I didn't make progress on this issue. One thing I considered at that moment was supporting positional arguments only cases first. Implementing keyword arguments will be quite complex and not many use cases exist we should cover. so it is not urgent.

If you want to take a look at this issue, it will be fine.

@corona10
Copy link
Member

enumrate is good example when you want to handle keyword arguments. as you can read it's too complex.

enumerate_vectorcall(PyObject *type, PyObject *const *args,
size_t nargsf, PyObject *kwnames)
{
PyTypeObject *tp = _PyType_CAST(type);
Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
Py_ssize_t nkwargs = 0;
if (kwnames != NULL) {
nkwargs = PyTuple_GET_SIZE(kwnames);
}
// Manually implement enumerate(iterable, start=...)
if (nargs + nkwargs == 2) {
if (nkwargs == 1) {
if (!check_keyword(kwnames, 0, "start")) {
return NULL;
}
} else if (nkwargs == 2) {
PyObject *kw0 = PyTuple_GET_ITEM(kwnames, 0);
if (_PyUnicode_EqualToASCIIString(kw0, "start")) {
if (!check_keyword(kwnames, 1, "iterable")) {
return NULL;
}
return enum_new_impl(tp, args[1], args[0]);
}
if (!check_keyword(kwnames, 0, "iterable") ||
!check_keyword(kwnames, 1, "start")) {
return NULL;
}
}
return enum_new_impl(tp, args[0], args[1]);
}
if (nargs + nkwargs == 1) {
if (nkwargs == 1 && !check_keyword(kwnames, 0, "iterable")) {
return NULL;
}
return enum_new_impl(tp, args[0], NULL);
}
if (nargs == 0) {
PyErr_SetString(PyExc_TypeError,
"enumerate() missing required argument 'iterable'");
return NULL;
}
PyErr_Format(PyExc_TypeError,
"enumerate() takes at most 2 arguments (%d given)", nargs + nkwargs);
return NULL;
}

@erlend-aasland
Copy link
Contributor

Sorry, I didn't make progress on this issue. One thing I considered at that moment was supporting positional arguments only cases first. Implementing keyword arguments will be quite complex and not many use cases exist we should cover. so it is not urgent.

If you want to take a look at this issue, it will be fine.

No need to be sorry :) I don't have the bandwidth to look at it right now, but I hope to be able to pick it up this fall. Let's keep the issue open. If you find time for this, that is great, but there is no urgency :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.13 new features, bugs and security fixes topic-argument-clinic topic-C-API type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

4 participants