Ignoring the question of whether owning the reference is the right thing
or not, the patch looks fine, although I don't see a reason for the
decrements to not be Py_DECREF since the function calls just won't even
happen if the object that the buffer is being created for is NULL.

As for changing whether Py_buffer holds a reference, that's tricky.
Removing the increment from PyBuffer_FillInfo() segfaults the
interpreter, so some code already assumes it does increment. There is
also the question of what the common use-case is; are buffers more like
items from a container, or their own object? The former says to
increment, the latter says to decrement. And since Py_buffer objects are
not true Python objects you can't just Py_DECREF() them.

My gut says to let buffers own a reference and to fix the "s#" leak.