classification
Title: PyEval_EvalCodeEx() can no longer be called with code which has (CO_NEWLOCALS | CO_OPTIMIZED) flags
Type: Stage: resolved
Components: C API Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2021-03-18 11:11 by vstinner, last changed 2021-03-18 13:52 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24918 merged vstinner, 2021-03-18 11:43
Messages (4)
msg389009 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 11:11
Cython generates a __Pyx_PyFunction_FastCallDict() function which calls PyEval_EvalCodeEx(). With Python 3.9, it worked well. With Python 3.10 in debug mode, it fails with an assertion error:

python3.10: Python/ceval.c:5148: PyEval_EvalCodeEx: Assertion `(((PyCodeObject *)_co)->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == 0' failed.

With Python 3.10 in release mode, it does crash.

Context of the failed assertion:

* Assertion added recently to CPython 3.10 by python/cpython@0332e56
* The code object flags = (CO_NEWLOCALS | CO_OPTIMIZED | CO_NOFREE)
* Code co_argcount = 2
* Code co_kwonlyargcount = 0
* Cython __Pyx_PyFunction_FastCallDict() called with: nargs=1 and kwargs=NULL

See the Cython issue to a reproducer: https://github.com/cython/cython/issues/4025#issuecomment-801829541

In Python 3.9, _PyFunction_Vectorcall() has the following fast-path:

    if (co->co_kwonlyargcount == 0 && nkwargs == 0 &&
        (co->co_flags & ~PyCF_MASK) == (CO_OPTIMIZED | CO_NEWLOCALS | CO_NOFREE))
    {
        if (argdefs == NULL && co->co_argcount == nargs) {
            return function_code_fastcall(tstate, co, stack, nargs, globals);
        }
        else if (nargs == 0 && argdefs != NULL
                 && co->co_argcount == PyTuple_GET_SIZE(argdefs)) {
            /* function called with no arguments, but all parameters have
               a default value: use default values as arguments .*/
            stack = _PyTuple_ITEMS(argdefs);
            return function_code_fastcall(tstate, co,
                                          stack, PyTuple_GET_SIZE(argdefs),
                                          globals);
        }
    }

When the bug occurs, __Pyx_PyFunction_FastCallDict() doesn't take the fast-path because nargs < co_argcount (1 < 2).

In Python 3.10, _PyFunction_Vectorcall() is very different:

    if (((PyCodeObject *)f->fc_code)->co_flags & CO_OPTIMIZED) {
        return _PyEval_Vector(tstate, f, NULL, stack, nargs, kwnames);
    }
    else {
        return _PyEval_Vector(tstate, f, f->fc_globals, stack, nargs, kwnames);
    }


PyEval_EvalCodeEx() must not crash if the code object has (CO_NEWLOCALS | CO_OPTIMIZED | CO_NOFREE) flags.
msg389011 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 11:17
If I the comment the following assertion in PyEval_EvalCodeEx():

assert ((((PyCodeObject *)_co)->co_flags & (CO_NEWLOCALS | CO_OPTIMIZED)) == 0);

The garbage collector fails while visiting builtins of the function with func_traverse(), at line:

   Py_VISIT(f->func_builtins);

I guess that the problem is that f->func_builtins doesn't hold a strong reference to builtins.


Error:

Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7fffea69d650
object refcount : 2265
object type     : 0x870e80
object type name: dict
object repr     : {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built-in'), '__build_class__': <built-in function __build_class__>, '__import__': <built-in function __import__>, 'abs': <built-in function abs>, 'all': <built-in function all>, 'any': <built-in function any>, 'ascii': <built-in function ascii>, ...}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007ffff7c20740 (most recent call first):
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 339 in __init__
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 1953 in __init__
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 2066 in __init__
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Symtab.py", line 1549 in declare_c_class
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Nodes.py", line 4782 in analyse_declarations
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Nodes.py", line 431 in analyse_declarations
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/ModuleNode.py", line 124 in analyse_declarations
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1608 in visit_ModuleNode
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1598 in __call__
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Pipeline.py", line 335 in run
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Pipeline.py", line 355 in run_pipeline
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 515 in run_pipeline
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Compiler/Main.py", line 727 in compile_single
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1208 in cythonize_one
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1102 in cythonize
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Cythonize.py", line 97 in cython_compile
  File "/home/vstinner/dev/numpy/env/lib/python3.10/site-packages/Cython/Build/Cythonize.py", line 223 in main
  File "/home/vstinner/dev/numpy/env/bin/cythonize", line 33 in <module>

Extension modules: Cython.Plex.Actions, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.Tempita._tempita, Cython.Compiler.Visitor, Cython.Compiler.FlowControl (total: 6)

Program received signal SIGABRT, Aborted.
msg389013 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 11:39
In commit 46496f9d12582bf11f4911ad0f23315d6f277907, I modified _PyEval_BuiltinsFromGlobals() to return a borrowed reference rather than a strong reference. It seems like I forgot to remove a Py_DECREF() in PyEval_EvalCodeEx()!

I'm working on a fix.
msg389015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 13:51
New changeset fc980e0be19776ee05dfc5380eb5d6a8092935cb by Victor Stinner in branch 'master':
bpo-43541: Fix PyEval_EvalCodeEx() regression (GH-24918)
https://github.com/python/cpython/commit/fc980e0be19776ee05dfc5380eb5d6a8092935cb
History
Date User Action Args
2021-03-18 13:52:01vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-03-18 13:51:42vstinnersetmessages: + msg389015
2021-03-18 11:43:43vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request23681
2021-03-18 11:39:56vstinnersetmessages: + msg389013
2021-03-18 11:17:33vstinnersetmessages: + msg389011
2021-03-18 11:11:13vstinnercreate