New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow objects implemented in pure Python to export PEP 3118 buffers #58006
Comments
Currently, there's no straightforward way to create new classes in Python that delegate the PEP-3118 buffer APIs to another object that supports them (e.g. bytes, bytearray, memoryview, array.array, a NumPy array). I see a few possible ways to go about this:
The getbuffer wrapper then calls __buffer__ and delegates the getbuffer call to the returned object. Any created Py_buffer instances refer directly to the underlying PEP-3118 supporting object, not the original object that defines __buffer__ The downside of this approach is that you can't implement a completely new PEP-3118 supporting object in pure Python (since you don't get access to the flags)
The releasebuffer wrapper then reverses this process, by passing the memoryview instance from the internal slot into the __releasebuffer__ call. |
Reviewing Stefan's work on bpo-10181 suggests to me that option 3 is the only feasible approach. (Allowing memoryview subclasses will permit types to pass their own state between __getbuffer__ and __releasebuffer__) |
I'm trying to understand what you want to be able to write. Do you http://mail.python.org/pipermail/python-ideas/2011-December/012993.html |
Consider a Python wrapper around a bytes object, or mmap or similar that |
To answer your other question, no, strview isn't related - that's strictly a PEP-3118 *consumer*, which is well supported from the Python side now that memoryview is fixed. The trick will be to allow a Python implemented object to be a PEP-3118 exporter *without* having to inherit from a C implemented type that does the slot mapping. Since PEP-3118 didn't describe a Python level API for the protocol, it may actually require a new PEP. One example for what you could do with it: use the new memoryview.cast() to provide multidimensional views on an exporter that only supports 1D exports natively. |
FWIW, Cython lets user code implement the buffer interface for extension types using the special methods "__getbuffer__()" and "__releasebuffer__()", so providing the same methods (although with a different signature) also for normal Python types would IMHO fit nicely. |
FWIW pypy has an __buffer__ method (used exclusively internally, AFAIK), which has semantics similar to your first proposal. |
Ok, just for the record: a single __buffer__() special method with delegation-only semantics would also work for Cython. Taking this path would provide a cleaner separation of the (then delegation-only) Python level protocol and the (all purpose) C level protocol. I think I'd prefer that. I assume __buffer__() takes no arguments, right? |
The reason I don't particularly like the "delegation only" API is that the combination of the new memoryview implementation and bytes/mmap/etc to get a flat region of memory to play with means you could do some quite interesting things entirely at the Python level. If you couldn't even use memoryview.cast() to change the shape of the exported memory, it makes the Python level API clearly inferior in power compared to what you can do in C. |
I suggest a PEP for 3.4 as the best way forward for this. |
A fourth way to add __getbuffer__ and __releasebuffer__ special methods to a Python class is through a new base class/mixin. The Py_buffer struct pointer passed to __getbuffer__ and __releasebuffer__ is wrapped with another special object type, which exposes the C struct fields as attributes. The base class is a direct subclass of object. Both the base and wrapper classes are implemented in an extension module. The use case is unit testing. A memoryview object may be adequate for simple C-contiguous arrays, but fails with F-contiguous or any non-contiguous array. It cannot export any arbitrary view, especially a deliberately invalid view, useful for testing error handlers. My implementation is at https://bitbucket.org/llindstrom/newbuffer . It is used in Pygame 1.9.2 unit tests: https://bitbucket.org/pygame/pygame . The newbuffer module exports two classes, BufferMixin and Py_buffer. The BufferMixin class adds bf_getbuffer and bf_releasebuffer slot functions to base class PyObject, nothing more. The Py_buffer class is low level, exposing pointer fields as integer addresses. It is designed for use with ctypes and is modelled on ctypes.Structure. Some limitations. In a class declaration, BufferMixin must be first in the inheritance list for the subclass to inherit the PEP-3118 interface. A buffer interface cannot be added to a builtin. I suggest this is a practical alternative to the three previously proposed solutions to this issue. This option takes its precedence from the ctypes module, which adds dangerous memory level access to Python, but optionally. It does not require modification of the base code, only an addition to the standard library. Finally, this approach has been demonstrated in a real-world application. |
@JelleZijlstra can this be closed thanks to the Buffer PEP? |
Yes! Fixed by PEP-688. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: