gcc 8 has added a new warning heuristic to detect invalid function casts and a stock python build seems to hit that warning quite often. The most common is the cast of a METH_NOARGS function (that uses just one argument) to a PyCFunction. The fix is pretty simple but needs to be applied widely. I'm slowly knocking them off in my spare time; WIP here, which has a few other types of warnings mixed in that I'll sift out during submission and also create separate bug reports for:

https://github.com/siddhesh/cpython/tree/func-cast

I'll clean up and post PR(s) once I am done but I figured I should file this report first since it is a pretty big change in terms of number of files touched and wanted to be sure that I'm making changes the way the community prefers.

Argument Clinic generates the following declaration for the second parameter of METH_NOARGS functions:

PyObject *Py_UNUSED(ignored)

It would be nice to follow the same style.

If the first parameter is of type PyObject* too, the explicit cast to PyCFunction can be removed.

Skip the curses module. It will be converted to Argument Clinic.

We are getting hit by that quite often on Fedora, with the transition to gcc 8 and it creates unnecessary noise at our build logs. Thanks for working on that.

When you sent your PR I can test it within our build system and verify if it works.

I don't have GCC 8 so I cannot verify this bug, but *function pointer casts* are fine - any function pointer can be cast to any other function pointer - it is only that they must *not* be called unless cast back again to be compatible with the function definition. Any fix to the contrary might well *cause* undefined behaviour!

Could you provide a sample of the *actual warnings* so that they could be studied?

I don't have GCC 8 so I cannot verify this bug, but *function pointer casts* are fine - any function pointer can be cast to any other function pointer - it is only that they must *not* be called unless cast back again to be compatible with the function definition. Any fix to the contrary might well *cause* undefined behaviour!

Please see the attached PR; METH_NOARGS callbacks are inconsistent in their signature and many have just one argument while they're called with two arguments, the second being NULL. The patch fixes these to consistently take and call with two arguments.

Could you provide a sample of the *actual warnings* so that they could be studied?

Here's one of a few hundred.

Objects/bytesobject.c:3085:25: warning: cast between incompatible function types from ‘PyObject * ()(striterobject *)’ {aka ‘struct _object * ()(struct <anonymous> *)’} to ‘PyObject * ()(PyObject *, PyObject *)’ {aka ‘struct _object * ()(struct _object *, struct _object *)’} [-Wcast-function-type]
{"__reduce__", (PyCFunction)striter_reduce, METH_NOARGS,
^
This is a new warning in gcc8, so you'll likely not find much reference, but here's a gcc bug report that might give you more context:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84531

Yea, I looked into ceval.c and the function is called incorrectly, so there is undefined behaviour there - it has been wrong all along, in 3.5 all the way down to 2-something

        if (flags & (METH_NOARGS | METH_O)) {
            PyCFunction meth = PyCFunction_GET_FUNCTION(func);
            PyObject *self = PyCFunction_GET_SELF(func);
            if (flags & METH_NOARGS && na == 0) {
                C_TRACE(x, (*meth)(self,NULL));

                x = _Py_CheckFunctionResult(func, x, NULL);
            }

The warning in GCC shouldn't probably have been enabled at all in -Wall -Wextra because the cast is explicit. However, it is somewhat true.

However, the correct way to fix would be to have the METH_NOARGS case cast the function to the right prototype. There exists lots of existing code that *is* going to break too.

Perhaps PyCFunction should declare no prototype, i.e. empty parentheses, for backwards compatibility:

    typedef PyObject *(*PyCFunction)();

and deprecate it; start using a new typedef for it - and then add proper casts in every place that call a function.

The warning in GCC shouldn't probably have been enabled at all in -Wall -Wextra because the cast is explicit. However, it is somewhat true.

The explicit cast is precisely what enables the more nuanced function cast warning where it checks the function for type compatibility. That is, it will check the types of the return and arguments and then warn in case of mismatch.

However, the correct way to fix would be to have the METH_NOARGS case cast the function to the right prototype. There exists lots of existing code that *is* going to break too.

AFAICT, there is no right prototype for the METH_NOARGS function; there used to be a PyCFunctionWithNoArguments but that seems to have fallen out of favour some time back. The prototype that seems to be commonly in use (and in clinic as well, which is what I based my patches on) is the PyCFunction, which seems like a safe way to do things.

Perhaps PyCFunction should declare no prototype, i.e. empty parentheses, for backwards compatibility:

typedef PyObject \*(*PyCFunction)();

and deprecate it; start using a new typedef for it - and then add proper casts in every place that call a function.

I have a patch in the works that makes it PyObject *(*)(PyObject *, PyObject *, ...)

which allows for two compulsory arguments (fits in with most cases, provided the METH_NOARGS patch is accepted) and then the rest depending on the type of the cast function. The rest of the PyMethodDef functions are much simpler fixes this way. It also seems like a more intuitive description of the callbacks.

Then there are getter and setter and other function pointers that need similar fixes to METH_NOARGS.

I forgot to clarify that the function cast warning allows for variable argument casts as a wildcard, which is my basis for the PyObject *(*)(PyObject *, PyObject *, ...) fix proposal.

Siddhesh, it looks like your fixes make the C function signatures match the signature expected in the PyMethodDef structure. If so, I suggest to remove the (PyCFunction) casts from those structure definitions as well. For instance, now that we have

PyObject *Noddy_name(Noddy *self, PyObject *Py_UNUSED(ignored))

I suggest changing

PyMethodDef Noddy_methods[] = {
{"name", (PyCFunction)Noddy_name, METH_NOARGS, ...

to

PyMethodDef Noddy_methods[] = {
{"name", Noddy_name, METH_NOARGS, ...

I suspect the casts were only added to hide compiler warnings related to this bug.

If you are proposing to add an ellipsis (...) to the definition of PyCFunction, that seems misguided. I understand this is incompatible under standard C. Are you relying on a GCC extension perhaps? Python is used with other compilers too.

Sorry, I realize there is a problem remaining with the pointer types for "Noddy_name" (Noddy vs PyObject pointers), so you can't remove the cast there. But my suggestion should still apply to other places, for instance the "error_out" method in Doc/howto/cporting.rst.

Fair enough, I'll reduce my scope of changes for this patchset, especially since I'm unable to find enough time to work on the remaining changes I had thought of in the coming weeks.

I'll post an updated patch soonish.

New changeset 55edd0c by Serhiy Storchaka (Siddhesh Poyarekar) in branch 'master':
bpo-33012: Fix invalid function cast warnings with gcc 8 for METH_NOARGS. (GH-6030)
55edd0c

The commit 55edd0c introduced a new warning:

gcc -pthread -c -Wno-unused-result -Wsign-compare -g -Og -Wall -Wstrict-prototypes -O0 -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -fPIC -DPy_BUILD_CORE -o Objects/longobject.o Objects/longobject.c
Objects/longobject.c:5359:5: warning: initialisation depuis un type pointeur incompatible [-Wincompatible-pointer-types]
long_long, /nb_int/
^~~~~~~~~
Objects/longobject.c:5359:5: note: (près de l'initialisation de « long_as_number.nb_int »)
Objects/longobject.c:5376:5: warning: initialisation depuis un type pointeur incompatible [-Wincompatible-pointer-types]
long_long, /* nb_index */
^~~~~~~~~
Objects/longobject.c:5376:5: note: (près de l'initialisation de « long_as_number.nb_index »)

It seems like a conversion to (unaryfunc) is needed when passing long_long as nb_int in long_as_number.

Yeah, there are multiple such uses that need wrappers to actually fix for gcc8, which will be released this week. I think I'll have more time for some more patches in this vein this weekend.

New changeset 6405fee by Serhiy Storchaka in branch 'master':
bpo-33012: Fix invalid function casts for long_long. (GH-6652)
6405fee

The following PRs fix warnings in casts to PyCFunction for method conventions different from METH_NOARGS, METH_O and
METH_VARARGS. PR 6748 does this for Argument Clinic generated code (all files except Tools/clinic/clinic.py are generated). PR 6749 does it for hand-written code. It also fixes few missed or new cases for METH_NOARGS.

Is it possible/feasible to fix that for the 3.7 and 3.6 branches as well?

See also bpo-33454 about a real bug found thanks to these warnings.

I think it safer to silence this kind of warnings in 3.7 and 3.6. PR 6757 does this.

New changeset ef91dde by Serhiy Storchaka in branch '3.7':
bpo-33012: Add -Wno-cast-function-type for gcc 8. (GH-6757)
ef91dde

New changeset 20b797d by Miss Islington (bot) in branch '3.6':
bpo-33012: Add -Wno-cast-function-type for gcc 8. (GH-6757)
20b797d

I propose to enhance the changes made by PR 6748 and PR 6749 so that gcc 8 triggers a warning when the function type of a PyMethodDef element does not match its flags by using a macro (see below) that casts the function first to the function type corresponding to those flags and then to (void *) as done in those PRs to silence the warning when cast to PyCFunction. This would allow detecting bugs such as the one found in bpo-33454.

With the following patch (it is just an example) gcc 8 would emit a warning if bisect_right() does not match the type (in the sense defined by -Wcast-function-type) of PyCFunctionWithKeywords:

diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c
index 831e10aa60..85fb0c188e 100644
--- a/Modules/_bisectmodule.c
+++ b/Modules/_bisectmodule.c
@@ -216,15 +216,14 @@ If x is already in a, insert it to the left of the leftmost x.\n\
 Optional args lo (default 0) and hi (default len(a)) bound the\n\
 slice of a to be searched.\n");
 
+#define SET_METH_VARARGS_KEYWORDS(func) \
+    (PyCFunction)(void *)(PyCFunctionWithKeywords)(func), METH_VARARGS|METH_KEYWORDS
+
 static PyMethodDef bisect_methods[] = {
-    {"bisect_right", (PyCFunction)bisect_right,
-        METH_VARARGS|METH_KEYWORDS, bisect_right_doc},
-    {"insort_right", (PyCFunction)insort_right,
-        METH_VARARGS|METH_KEYWORDS, insort_right_doc},
-    {"bisect_left", (PyCFunction)bisect_left,
-        METH_VARARGS|METH_KEYWORDS, bisect_left_doc},
-    {"insort_left", (PyCFunction)insort_left,
-        METH_VARARGS|METH_KEYWORDS, insort_left_doc},
+    {"bisect_right", SET_METH_VARARGS_KEYWORDS(bisect_right), bisect_right_doc},
+    {"insort_right", SET_METH_VARARGS_KEYWORDS(insort_right), insort_right_doc},
+    {"bisect_left", SET_METH_VARARGS_KEYWORDS(bisect_left), bisect_left_doc},
+    {"insort_left", SET_METH_VARARGS_KEYWORDS(insort_left), insort_left_doc},
     {NULL, NULL} /* sentinel */
 };

Great idea!

But the problem is that additional flags can be used, e.g. METH_CLASS.

But the problem is that additional flags can be used, e.g. METH_CLASS.

Right, https://docs.python.org/3/c-api/structures.html says: "The individual flags indicate either a calling convention or a binding convention". This may be overcome by introducing another macro with 2 arguments, the second argument being used to set the binding convention flag:

#define SET_METH_VARARGS_KEYWORDS_FLAG(func, flag) \
    (PyCFunction)(void *)(PyCFunctionWithKeywords)func, METH_VARARGS|METH_KEYWORDS|flag
#define SET_METH_VARARGS_KEYWORDS(func) SET_METH_VARARGS_KEYWORDS_FLAG(func, 0x0000)

The refactoring would be quite large though.

Well, there's only one problem with casting to void *: while converting the function pointer to another *is* standard-compliant, and GCC is being just hypersensitive here, casting a function pointer to void * isn't, though it is a common extension (http://port70.net/~nsz/c/c11/n1570.html#J.5.7).

Pedantically the correct way is to cast to a function pointer with no prototype (empty parentheses) and from that to the target type. See for example. See for example https://godbolt.org/g/FdPdUj

Pedantically the correct way is to cast to a function pointer with no prototype (empty parentheses) and from that to the target type. See for example. See for example https://godbolt.org/g/FdPdUj

This is one way that the gcc diagnostics explicitly allow in addition to variable arguments like so: https://godbolt.org/g/Dtb4fv

There are still quite a few similar warnings on git-master. Do they indicate the same problem or should I open new issues?

For example:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I. -I./Include -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c
In file included from Objects/bytearrayobject.c:96:
Objects/clinic/bytearrayobject.c.h:677:23: warning: cast between incompatible function types from ‘PyObject * ()(PyByteArrayObject *, PyObject * const, Py_ssize_t)’ {aka ‘struct _object * ()(struct <anonymous> *, struct _object * const, long int)’} to ‘PyObject * ()(PyObject *, PyObject *)’ {aka ‘struct _object * ()(struct _object *, struct _object *)’} [-Wcast-function-type]
{"__reduce_ex__", (PyCFunction)bytearray_reduce_ex, METH_FASTCALL, bytearray_reduce_ex__doc__},
^
Objects/bytearrayobject.c:2136:5: note: in expansion of macro ‘BYTEARRAY_REDUCE_EX_METHODDEF’
BYTEARRAY_REDUCE_EX_METHODDEF
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I'm using GCC 8.1.0 on Arch Linux.

Pedantically the correct way is to cast to a function pointer with no prototype (empty parentheses)

Thanks for raising this point Antti. This is hinted at by the gcc 8 documentation on -Wcast-function-type: "The function type void (*) (void) is special and matches everything, which can be used to suppress this warning" [1]. One cannot use empty parentheses though as this raises -Wstrict-prototypes on Python builds with gcc and -Wcast-function-type with gcc 8.

So (void *) may be replaced with (void (*) (void)) in my previous code snippets and also in PR 6748 and PR 6749 (I just checked my code).

[1] https://gcc.gnu.org/onlinedocs/gcc-8.1.0/gcc/Warning-Options.html#Warning-Options

There are still quite a few similar warnings on git-master. Do they indicate the same problem or should I open new issues?

This is the same issue, the warning line ends with [-Wcast-function-type].

The problem with invalid function signatures was initially reported in old bpo-1648268.

If I click the link for PR 6748, I see a page with "We went looking everywhere, but couldn’t find those commits." Maybe the PR needs to be refreshed or closed and maybe reopened. PR 6749 looks normal.

I marked bpo-34992 as a duplicate of this issue:

I use Fedora 28 and gcc (GCC) 8.1.1 20180712 (Red Hat 8.1.1-5)

and I get a lot of warnings, with the standard config of Python (./configure)

Objects/call.c: In function '_PyMethodDef_RawFastCallDict':
Objects/call.c:515:24: warning: cast between incompatible function types from 'PyCFunction' {aka 'struct _object * ()(struct _object *, struct _object *)'} to 'PyObject * ()(PyObject *, PyObject *, PyObject *)' {aka 'struct _object * ()(struct _object *, struct _object *, struct _object *)'} [-Wcast-function-type]
result = ((PyCFunctionWithKeywords)meth) (self, argstuple, kwargs);
^
Objects/call.c:530:20: warning: cast between incompatible function types from 'PyCFunction' {aka 'struct _object * ()(struct _object *, struct _object *)'} to 'PyObject * ()(PyObject *, PyObject * const*, Py_ssize_t)' {aka 'struct _object * ()(struct _object *, struct _object * const, long int)'} [-Wcast-function-type]
result = ((_PyCFunctionFast)meth) (self, args, nargs);
^
Objects/call.c:538:49: warning: cast between incompatible function types from 'PyCFunction' {aka 'struct _object * ()(struct _object *, struct _object *)'} to 'PyObject * ()(PyObject *, PyObject * const, Py_ssize_t, PyObject *)' {aka 'struct _object * ()(struct _object *, struct _object * const, long int, struct _object *)'} [-Wcast-function-type]
_PyCFunctionFastWithKeywords fastmeth = (_PyCFunctionFastWithKeywords)meth;

....

+- 827 warnings

Try this - https://github.com/siddhesh/cpython/tree/func-cast

New changeset 4a934d4 by Serhiy Storchaka in branch 'master':
bpo-33012: Fix invalid function cast warnings with gcc 8 in Argument Clinic. (GH-6748)
4a934d4

New changeset 8152402 by Serhiy Storchaka in branch 'master':
bpo-33012: Fix signatures of METH_NOARGS funstions. (GH-10736)
8152402

New changeset 62be742 by Serhiy Storchaka in branch 'master':
bpo-33012: Fix invalid function cast warnings with gcc 8. (GH-6749)
62be742

New changeset 62be742 by Serhiy Storchaka in branch 'master':
bpo-33012: Fix invalid function cast warnings with gcc 8. (GH-6749)

I still get something 100 warnings with GCC 8.2.1 on Fedora 29. I wrote PR 10744 to add a macro to cast a function pointer.

I chose to add a macro, so it will be easier to spot where we cast function pointers and change the implementation (cast) if needed at a single place.

New changeset ad8ac54 by Serhiy Storchaka in branch '3.7':
bpo-33012: Fix signatures of METH_NOARGS functions. (GH-10736) (GH-10748)
ad8ac54

New changeset d5c8bd8 by Miss Islington (bot) in branch '3.6':
bpo-33012: Fix signatures of METH_NOARGS functions. (GH-10736) (GH-10748)
d5c8bd8

New changeset 1c60715 by Serhiy Storchaka in branch 'master':
bpo-33012: Fix more invalid function cast warnings with gcc 8. (GH-10751)
1c60715

Only one function cast warning is left, and it is a separate issue: bpo-33015.

New changeset 1659c08 by Miss Islington (bot) in branch '3.7':
bpo-33012: Fix more invalid function cast warnings with gcc 8. (GH-10751)
1659c08

New changeset 77000bb by Victor Stinner in branch '3.6':
bpo-33012: Fix more invalid function cast warnings with gcc 8. (GH-10751) (GH-10796)
77000bb

I marked bpo-36197 as a duplicate of this issue:

"""
gcc -pthread -c -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -DPy_BUILD_CORE -o Objects/memoryobject.o Objects/memoryobject.c
Objects/memoryobject.c:3112:21: warning: cast between incompatible function types from 'PyObject * ()(PyMemoryViewObject *, PyObject *, PyObject *)' {aka 'struct _object * ()(struct <anonymous> *, struct _object *, struct _object *)'} to 'PyObject * ()(PyObject *, PyObject *)' {aka 'struct _object * ()(struct _object *, struct _object *)'} [-Wcast-function-type]
{"tobytes", (PyCFunction)memory_tobytes, METH_VARARGS|METH_KEYWORDS, memory_tobytes_doc},

I am preparing a small PR for this issue.
"""

=> PR 12179

New changeset 359a2f3 by Victor Stinner (Stéphane Wirtel) in branch 'master':
bpo-33012: Fix compilation warnings in memoryobject.c and _collectionsmodule.c (GH-12179)
359a2f3

Looking at:

359a2f3

This is not fixing the underlying issue but hiding it. The right fix would be to use a union for ml_meth providing members for the 3 different function. So the developer could assign them correctly and the compiler would warn if he would do something wrong. Casting to (void *) is just hiding the problem not fixing it!

Yes, and this particular case was fixed in adfffc7.

And how do you deal with METH_VARARGS|METH_KEYWORDS functions which have 3 arguments?

PyObject* myfunc(PyObject *py_obj, PyObject *args, PyObject *kwargs)

FYI I modified SystemError("bad call flags") error message to include the method name in bpo-39884.