initproc is declared to return an int, but what returned values mean is not documented. Noddy_init in http://docs.python.org/3/extending/newtypes.html?highlight=initproc#adding-data-and-methods-to-the-basic-example can be seen to return 0 on success and -1 on error, but that's about it.

Also, when I wrote a function which return 1 on error, on every second invocation the exception would be ignored:
static int Reader_init(Reader *self, PyObject *args, PyObject *keywds)
{
    ...
    if (flags && path) {
            PyErr_SetString(PyExc_ValueError, "cannot use both flags and path");
            return 1;
    }
    ...
}

>>> obj(123, '/tmp')
>>> obj(123, '/tmp')
...
ValueError
>>> obj(123, '/tmp')
>>> obj(123, '/tmp')
...
ValueError

I'm not sure how to interpret this since I couldn't find the documentation for the expected value.