classification
Title: PyBuffer_FillInfo() from 3.3
Type: Stage: needs patch
Components: Documentation Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: skrah Nosy List: arigo, docs@python, python-dev, skrah
Priority: normal Keywords:

Created on 2014-06-16 14:03 by arigo, last changed 2014-06-29 22:18 by skrah. This issue is now closed.

Files
File name Uploaded Description Edit
xymodule.c arigo, 2014-06-16 14:03 C extension module
Messages (6)
msg220721 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2014-06-16 14:03
Following the documentation at https://docs.python.org/3/c-api/buffer.html, if we write a custom object in C with a getbufferproc that simply returns PyBuffer_FillInfo(...), then it works fine up to Python 3.2 but segfaults in Python 3.3 depending on what we do with it.  The segfault is caused by memoryobject.c:last_dim_is_contiguous(), which reads from the "strides" array without checking that it is not NULL.

Attached the simplest example of C extension module I could make.  When executing this:

>>> import xy
>>> m=memoryview(bytearray(b"abcdef"))
>>> m[:5] = xy.gm

...it segfaults in Python 3.3.  (I'm told it works fine in 3.2 and segfaults in 3.4 as well, but didn't confirm it.)
msg220727 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-06-16 14:29
The flags from mb_getbuf() have to be passed to PyBuffer_FillInfo():

   return PyBuffer_FillInfo(view, (PyObject *)self,
                            internal, 5,
                            /*readonly=*/0, flags);


The flags contain the actual request that the consumer
sends to the buffer provider, so they cannot be hardcoded.

In this example, memoryview sends the PyBUF_FULL request to
mb_getbuf(). If the request is successful, it can assume
non-NULL strides.


Should the documentation be improved?
msg220730 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2014-06-16 15:20
Ah! Thanks.  I systematically missed the "flags" arguments passed in the getbufferproc function.

(I thought I had to tell PyBuffer_FillInfo() what kind of format we actually provide, but it seems that the interface is the other way around.  I don't see how I would implement a getbufferproc that forces a specific format, but I don't need that, so it's fine.)
msg220741 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-06-16 17:49
Yes, PyBuffer_FillInfo() is a bit confusing:  The canonical usage *looks* as
if "flags" were somehow used in the Py_buffer struct to mark a buffer as having
certain capabilities:

Modules/_io/bufferedio.c:
-------------------------

    if (PyBuffer_FillInfo(&buf, NULL, start, len, 0, PyBUF_CONTIG) == -1)
        return -1;
    memobj = PyMemoryView_FromBuffer(&buf);

What really goes on since Python 3.3 is that memobj is a memoryview
with (shape,strides,format), since missing Py_buffer values are
reconstructed.

So all memoryviews -- when acting as an exporter as in this example --
have PyBUF_FULL capabilities.  To make that clear, we should actually
change the PyBUF_CONTIG in the above example to PyBUF_FULL.  Then
the mental model is:

    PyBuffer_FillInfo(..., PyBUF_FULL)

                 ^
                 |  "Request" full buffer to export.

               memobj

                 ^
                 |  Request anything from PyBUF_SIMPLE to PyBUF_FULL. Note
                 |  that e.g. with PyBUF_SIMPLE the consumer's shape and
                 |  strides will be NULL!

              Consumer

And that model is consistent with the usage of PyBuffer_FillInfo() inside
a getbufferproc, where "flags" of course can be anything.
msg220742 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-06-16 18:01
I think it's worth adding a note to the docs about passing the flags unmodified inside a getbufferproc.
msg221908 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-06-29 22:16
New changeset 49cdb04bfcc6 by Stefan Krah in branch '3.4':
Issue #21778:  Clarify use of flags if PyBuffer_FillInfo() is used inside a
http://hg.python.org/cpython/rev/49cdb04bfcc6
History
Date User Action Args
2014-06-29 22:18:22skrahsetstatus: open -> closed
resolution: fixed
2014-06-29 22:16:59python-devsetnosy: + python-dev
messages: + msg221908
2014-06-25 12:04:03skrahsetassignee: docs@python -> skrah
2014-06-16 18:01:17skrahsetassignee: docs@python
components: + Documentation
versions: + Python 3.4, Python 3.5
nosy: + docs@python

messages: + msg220742
resolution: not a bug -> (no value)
stage: needs patch
2014-06-16 17:49:41skrahsetmessages: + msg220741
2014-06-16 15:20:51arigosetresolution: not a bug
2014-06-16 15:20:24arigosetmessages: + msg220730
2014-06-16 14:29:48skrahsetnosy: + skrah
messages: + msg220727
2014-06-16 14:03:43arigosetfiles: + xymodule.c
2014-06-16 14:03:24arigocreate