This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Redundant store can be removed from _PyFunction_FastCallDict
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: vstinner Nosy List: Eric Lippert, serhiy.storchaka, tim.peters, vstinner
Priority: normal Keywords: patch

Created on 2018-08-30 17:27 by Eric Lippert, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9009 merged Eric Lippert, 2018-08-30 18:02
Messages (5)
msg324395 - (view) Author: Eric Lippert (Eric Lippert) * Date: 2018-08-30 17:27
In _PyFunction_FastCallDict we have local nk assigned to be the size of a dictionary, and then local i is assigned to twice the size of the same dictionary, and then nk is assigned to half of i, which it already is:

   nk = (kwargs != NULL) ? PyDict_GET_SIZE(kwargs) : 0;
   if (nk != 0) {
     ...
     pos = i = 0;
     while (PyDict_Next(kwargs, &pos, &k[i], &k[i+1])) {
       ...
       i += 2;
     }
     nk = i / 2;

I am attempting to understand the performance characteristics of this hot path, and I spent far too long trying to figure out why nk was being assigned a value it already has. :)

I propose that the redundant store be replaced with an assertion that i/2 is equal to nk.

I will submit a pull request presently.
msg324407 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-31 08:26
Technically it's possible that the dictionary mutates itself during iterating and so that it's size change, so "nk = i / 2;" make sure that we pass the proper length to the function call.

Maybe I'm wrong, but we would need an unit test for that.

While I reworked a lot of code in call.c (and I moved code to call.c ;-)), this specific statement already existed before my work. I moved it from somewhere else. Extract of Objects/funcobject.c:

static PyObject *
function_call(PyObject *func, PyObject *arg, PyObject *kw)
{
    ...
    if (kw != NULL && PyDict_Check(kw)) {
        ...
        while (PyDict_Next(kw, &pos, &k[i], &k[i+1])) {
            Py_INCREF(k[i]);
            Py_INCREF(k[i+1]);
            i += 2;
        }
        nk = i/2;
    }
    ...
}

It seems like the code is quite old :-)

commit 6d6c1a35e08b95a83dbe47dbd9e6474daff00354
Author: Tim Peters <tim.peters@gmail.com>
Date:   Thu Aug 2 04:15:00 2001 +0000

    Merge of descr-branch back into trunk.
msg324413 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-08-31 09:42
I think it is technically not possible. Neither PyDict_Next() nor Py_INCREF() mutate the dict, call the user code or release GIL. If it could be possible, we would have a potential writing out of a buffer here.
msg324420 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-08-31 14:19
> I think it is technically not possible. Neither PyDict_Next() nor Py_INCREF() mutate the dict, call the user code or release GIL. If it could be possible, we would have a potential writing out of a buffer here.

When I read PyDict_Next(), I'm thinking at the Python level which can execute arbitrary code. But it seems like you are right: the exact C implementation of PyDict_Next() doesn't seem to modify the dictionary.

PR 9009 seems safe.
msg324439 - (view) Author: Eric Lippert (Eric Lippert) * Date: 2018-08-31 17:21
If it were possible that the interpreter iterating over a dictionary could cause the dictionary to change size then I suspect that this would be a rich source of bugs to mine. :-) 

It would be great if we could make it a documented, enforced invariant that the interpreter reading a dictionary never produces a side effect visible to the interpreter.

Re: "this specific statement already existed before my work."

Thanks for the historical note; the first thing I thought when I read this line was "that's got to be a copy-paste left over from an earlier version of the algorithm."
History
Date User Action Args
2022-04-11 14:59:05adminsetgithub: 78732
2018-10-22 15:55:36lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-08-31 17:21:23Eric Lippertsetmessages: + msg324439
2018-08-31 17:02:16rhettingersetnosy: + tim.peters
2018-08-31 14:19:52vstinnersetmessages: + msg324420
2018-08-31 09:42:11serhiy.storchakasetmessages: + msg324413
2018-08-31 08:26:09vstinnersetmessages: + msg324407
2018-08-30 23:58:31rhettingersetassignee: vstinner

nosy: + serhiy.storchaka, vstinner
2018-08-30 18:02:06Eric Lippertsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8479
2018-08-30 17:27:21Eric Lippertcreate