Author vstinner
Recipients scoder, serhiy.storchaka, vstinner
Date 2016-08-22.13:00:25
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1471870825.41.0.667040150166.issue27809@psf.upfronthosting.co.za>
In-reply-to
Content
Serhiy Storchaka: "The problem is that passing keyword arguments as a dict is not the most efficient way due to an overhead of creating a dict. For now keyword arguments are pushed on the stack as interlaced array of keyword names and values. It may be more efficient to push values and names as continuous arrays (issue27213)."

You describe one specific kind of function call: from Python to Python. Sure, in this case, we *can* avoid the creation of a dictionary. Such optimization can be implemented in call_function() in Python/ceval.c.

The problem is that in other cases, it's harder to avoid the creation of a dictionary:

* C functions defined with METH_KEYWORDS require a dict
* tp_new, tp_init and tp_call slots require a dict

Many C functions "pass through" keyword arguments. Example with the builtin sorted() function:

    static PyObject *
    builtin_sorted(PyObject *self, PyObject *args, PyObject *kwds)
    {
        ...
        v = PyObject_Call(callable, newargs, kwds);
        ...
    }

Example of a tp_new slot, the type constructor, type_new():

    static PyObject *
    type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)
    {
        ...
        if (winner != metatype) {
            if (winner->tp_new != type_new) /* Pass it to the winner */
                return winner->tp_new(winner, args, kwds);
            metatype = winner;
        }
        ...
        if (init_subclass(type, kwds) < 0)
            goto error;
        ...
    }

    static int
    init_subclass(PyTypeObject *type, PyObject *kwds)
    {
        ...
        tmp = PyObject_Call(func, tuple, kwds);
        ...
    }

Well, the question can be: where do these keyword argument come from? It's like that in 99% of cases, these C functions are called from Python functions which use the Python stack for positional and keyword arguments.

The last part of my full fastcall project is to modify tp_new, tp_init and tp_call slots to support a fastcall mode which uses the fastcall calling convention (add new flags to tp_flags: Py_TPFLAGS_FASTNEW, Py_TPFLAGS_FASTINIT, Py_TPFLAGS_FASTCALL).

I'm not sure that this part of my plan is feasible since it is a major backward incompatible change: it breaks all C code calling directly tp_new, tp_init and tp_call, and such code is common (even inside the stdlib). I plan to open a discussion to see if it's worth it. It would require to modify all code calling directly tp_new, tp_init and tp_call to use a new wrapper function, function which would only be available in the C API of Python 3.6...

--

I understand your idea of avoid dictionaries for keyword parameters, but maybe we need an intermediate step and maybe even two functions:

* one function taking a dict: _PyObject_FastCallDict()
* one function taking an array of (key, value) pairs: _PyObject_FastCall()

_PyObject_FastCallDict() would pass directly the Python dict to C functions defined with METH_KEYWORDS, to tp_new, tp_init and tp_call slots.

In a perfect world, _PyObject_FastCallDict() wouldn't be needed. In practice, dictionaries are used *everywhere* in the current C code base, so we need a function for the transition.

_PyObject_FastCallDict() is private, so we can remove it whenever we want.

--

I'm patching more and more code, and I hate having to add an extra ", NULL" when calling the current _PyObject_FastCall().

Maybe we should have a short name, _PyObject_FastCall(), for the common case: no keyword parameter, and use longer name for keywords: _PyObject_FastCallKeywords() (as PyArg_ParseTuple and PyArg_ParseTupleKeywords).

--

To summarize, I propose 3 functions:

* _PyObject_FastCall(PyObject **args, int nargs)
* _PyObject_FastCallKeywords(PyObject **args, int nargs, PyObject **kwargs, int nkwargs): nkwargs is the number of (key, value) pairs
* _PyObject_FastCallDict(PyObject **args, int nargs, PyObject *kwargs): kwargs is a Python dictionary

METH_FASTCALL would use _PyObject_FastCallKeywords format: "PyObject **kwargs, int nkwargs" for keyword parameters.

What do you think?
History
Date User Action Args
2016-08-22 13:00:25vstinnersetrecipients: + vstinner, scoder, serhiy.storchaka
2016-08-22 13:00:25vstinnersetmessageid: <1471870825.41.0.667040150166.issue27809@psf.upfronthosting.co.za>
2016-08-22 13:00:25vstinnerlinkissue27809 messages
2016-08-22 13:00:25vstinnercreate