classification
Title: Allow objects implemented in pure Python to export PEP 3118 buffers
Type: Stage:
Components: Interpreter Core Versions: Python 3.5
process
Status: open Resolution:
Dependencies: 10181 Superseder:
Assigned To: Nosy List: Arfrever, Jacob.Holm, alex, eric.snow, kermode, mark.dickinson, ncoghlan, pitrou, rhettinger, sbt, scoder, vadmium
Priority: normal Keywords:

Created on 2012-01-16 10:29 by ncoghlan, last changed 2014-10-14 16:38 by skrah.

Files
File name Uploaded Description Edit
llindstrom-newbuffer-0dd8ba1c2c2c.tar.gz kermode, 2013-08-16 22:53 Extension module containing BufferMixin and Py_buffer. Source.
Messages (11)
msg151347 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-01-16 10:29
Currently, there's no straightforward way to create new classes in Python that delegate the PEP 3118 buffer APIs to another object that supports them (e.g. bytes, bytearray, memoryview, array.array, a NumPy array).

I see a few possible ways to go about this:

1. Add a single new special method, __buffer__. This creates an entry in the underlying getbuffer slot, but leaves releasebuffer empty.

The getbuffer wrapper then calls __buffer__ and delegates the getbuffer call to the returned object.

Any created Py_buffer instances refer directly to the underlying PEP 3118 supporting object, not the original object that defines __buffer__

The downside of this approach is that you can't implement a completely new PEP 3118 supporting object in pure Python (since you don't get access to the flags)

2. As above, but any Py_buffer structs returned are updated to refer to the original object, not the underlying one. A releasebuffer wrapper is also created, that calls back into __buffer__ to retrieve the object again. This approach has problems if __buffer__ can change to refer to different objects over time.

3. Define separate __getbuffer__ and __releasebuffer__ special methods. All C types implementing the PEP 3118 slots get these special methods automatically (as with any other slot). The flag variables are passed into __getbuffer__, and it is expected to return a memoryview object defining the view. The underlying getbuffer slot wrapper stores the current object in the Py_buffer "obj" slot, and the returned memoryview in "internal". The rest of the Py_buffer fields are populated based on the memoryview contents.

The releasebuffer wrapper then reverses this process, by passing the memoryview instance from the internal slot into the __releasebuffer__ call.
msg151351 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-01-16 11:47
Reviewing Stefan's work on #10181 suggests to me that option 3 is the only feasible approach.

(Allowing memoryview subclasses will permit types to pass their own state between __getbuffer__ and __releasebuffer__)
msg154683 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-03-01 09:33
I'm trying to understand what you want to be able to write. Do you
perhaps have a short example? Also, to get the bigger picture: Is
this related to your strview proposal?

http://mail.python.org/pipermail/python-ideas/2011-December/012993.html
msg154684 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-01 09:48
Consider a Python wrapper around a bytes object, or mmap or similar that
wants to pass PEP 3118 buffer requests through to the underlying object.
Currently, there's no way to write such a delegation - the delegating class
has to be written in C.
msg154692 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-01 13:19
To answer your other question, no, strview isn't related - that's strictly a PEP 3118 *consumer*, which is well supported from the Python side now that memoryview is fixed.

The trick will be to allow a Python implemented object to be a PEP 3118 exporter *without* having to inherit from a C implemented type that does the slot mapping. Since PEP 3118 didn't describe a Python level API for the protocol, it may actually require a new PEP.

One example for what you could do with it: use the new memoryview.cast() to provide multidimensional views on an exporter that only supports 1D exports natively.
msg154855 - (view) Author: Stefan Behnel (scoder) * Date: 2012-03-03 21:24
FWIW, Cython lets user code implement the buffer interface for extension types using the special methods "__getbuffer__()" and "__releasebuffer__()", so providing the same methods (although with a different signature) also for normal Python types would IMHO fit nicely.
msg156440 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2012-03-20 17:42
FWIW pypy has an __buffer__ method (used exclusively internally, AFAIK), which has semantics similar to your first proposal.
msg156479 - (view) Author: Stefan Behnel (scoder) * Date: 2012-03-21 06:59
Ok, just for the record: a single __buffer__() special method with delegation-only semantics would also work for Cython. Taking this path would provide a cleaner separation of the (then delegation-only) Python level protocol and the (all purpose) C level protocol. I think I'd prefer that.

I assume __buffer__() takes no arguments, right?
msg156485 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-21 10:16
The reason I don't particularly like the "delegation only" API is that the combination of the new memoryview implementation and bytes/mmap/etc to get a flat region of memory to play with means you could do some quite interesting things entirely at the Python level.

If you couldn't even use memoryview.cast() to change the shape of the exported memory, it makes the Python level API clearly inferior in power compared to what you can do in C.
msg163058 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-06-17 11:43
I suggest a PEP for 3.4 as the best way forward for this.
msg195436 - (view) Author: Lenard Lindstrom (kermode) Date: 2013-08-16 22:53
A fourth way to add __getbuffer__ and __releasebuffer__ special methods to a Python class is through a new base class/mixin. The Py_buffer struct pointer passed to __getbuffer__ and __releasebuffer__ is wrapped with another special object type, which exposes the C struct fields as attributes. The base class is a direct subclass of object. Both the base and wrapper classes are implemented in an extension module. 

The use case is unit testing. A memoryview object may be adequate for simple C-contiguous arrays, but fails with F-contiguous or any non-contiguous array. It cannot export any arbitrary view, especially a deliberately invalid view, useful for testing error handlers.

My implementation is at https://bitbucket.org/llindstrom/newbuffer . It is used in Pygame 1.9.2 unit tests: https://bitbucket.org/pygame/pygame . The newbuffer module exports two classes, BufferMixin and Py_buffer. The BufferMixin class adds bf_getbuffer and bf_releasebuffer slot functions to base class PyObject, nothing more. The Py_buffer class is low level, exposing pointer fields as integer addresses. It is designed for use with ctypes and is modelled on ctypes.Structure.

Some limitations. In a class declaration, BufferMixin must be first in the inheritance list for the subclass to inherit the PEP 3118 interface. A buffer interface cannot be added to a builtin.

I suggest this is a practical alternative to the three previously proposed solutions to this issue. This option takes its precedence from the ctypes module, which adds dangerous memory level access to Python, but optionally. It does not require modification of the base code, only an addition to the standard library. Finally, this approach has been demonstrated in a real-world application.
History
Date User Action Args
2014-10-14 16:38:56skrahsetnosy: - skrah
2013-12-04 10:58:28vadmiumsetnosy: + vadmium
2013-12-01 05:02:26ncoghlansetversions: + Python 3.5, - Python 3.4
2013-08-16 22:53:36kermodesetfiles: + llindstrom-newbuffer-0dd8ba1c2c2c.tar.gz
nosy: + kermode
messages: + msg195436

2013-06-16 00:09:20Jacob.Holmsetnosy: + Jacob.Holm
2012-07-07 22:28:58Arfreversetnosy: + Arfrever
2012-06-17 11:43:28ncoghlansetmessages: + msg163058
versions: + Python 3.4, - Python 3.3
2012-05-27 16:26:26sbtsetnosy: + sbt
2012-03-21 10:16:02ncoghlansetmessages: + msg156485
2012-03-21 08:00:58mark.dickinsonsetnosy: + mark.dickinson
2012-03-21 06:59:12scodersetmessages: + msg156479
2012-03-20 17:42:40alexsetnosy: + alex
messages: + msg156440
2012-03-03 21:24:25scodersetnosy: + scoder
messages: + msg154855
2012-03-02 03:33:36eric.snowsetnosy: + eric.snow
2012-03-01 13:19:37ncoghlansetmessages: + msg154692
2012-03-01 09:48:56ncoghlansetmessages: + msg154684
2012-03-01 09:33:13skrahsetmessages: + msg154683
2012-01-23 10:40:57rhettingersetnosy: + rhettinger
2012-01-16 11:47:46ncoghlansetmessages: + msg151351
2012-01-16 10:30:14ncoghlansettitle: Add a Python level special method to retrieve a PEP 3118 object -> Allow objects implemented in pure Python to export PEP 3118 buffers
2012-01-16 10:29:30ncoghlansetdependencies: + Problems with Py_buffer management in memoryobject.c (and elsewhere?)
title: Add a __buffer__ special method to retrieve a PEP 3118 object -> Add a Python level special method to retrieve a PEP 3118 object
2012-01-16 10:29:10ncoghlancreate