diff -r 43e0ceb5dde9 Doc/c-api/buffer.rst --- a/Doc/c-api/buffer.rst Fri Feb 08 07:56:45 2013 -0800 +++ b/Doc/c-api/buffer.rst Wed Feb 13 19:45:50 2013 -0800 @@ -7,7 +7,7 @@ .. sectionauthor:: Greg Stein .. sectionauthor:: Benjamin Peterson - +.. sectionauthor:: Demian Brecht .. index:: object: buffer @@ -19,17 +19,6 @@ interface to access the object data directly, without needing to copy it first. -Two examples of objects that support the buffer interface are strings and -arrays. The string object exposes the character contents in the buffer -interface's byte-oriented form. An array can also expose its contents, but it -should be noted that array elements may be multi-byte values. - -An example user of the buffer interface is the file object's :meth:`write` -method. Any object that can export a series of bytes through the buffer -interface can be written to a file. There are a number of format codes to -:cfunc:`PyArg_ParseTuple` that operate against an object's buffer interface, -returning data from the target object. - Starting from version 1.6, Python has been providing Python-level buffer objects and a C-level buffer API so that any built-in or used-defined type can expose its characteristics. Both, however, have been deprecated because of @@ -37,104 +26,70 @@ of a new C-level buffer API and a new Python-level object named :class:`memoryview`. -The new buffer API has been backported to Python 2.6, and the -:class:`memoryview` object has been backported to Python 2.7. It is strongly -advised to use them rather than the old APIs, unless you are blocked from -doing so for compatibility reasons. +Buffer and memoryview objects are useful as a way to expose the data from +another object's +buffer interface to the Python programmer. They can also be used as a +zero-copy slicing mechanism. Using their ability to reference a block of +memory, it is possible to expose any data to the Python programmer quite +easily. The memory could be a large, constant array in a C extension, it could +be a raw block of memory for manipulation before passing to an operating +system library, or it could be used to pass around structured data in its +native, in-memory format. +The buffer objects look very similar to string objects at the +Python programming level: they support slicing, indexing, concatenation, and +some other standard string operations. However, their data can come from one +of two sources: from a block of memory, or from another object which exports +the buffer interface. -The new-style Py_buffer struct -============================== +The new C-level buffer API has been backported to Python 2.6 and the +Python-level :class:`memoryview` object has been backported to Python 2.7. +To be clear, in Python 2.6 and 2.7, there are two C-level interfaces, both +exposed by the :ctype:`PyBufferProcs` interface: the new buffer API and the +old buffer API. The :class:`memoryview` object is the product of the new API +while :class:`buffer` is the product of the old API (the former of which is +only available in 2.7+). +It is strongly recommended to use the new C API and the Python-level +:class:`memoryview` object rather than the old APIs, unless you are blocked +from doing so for one of the following examples of compatability issues: -.. ctype:: Py_buffer +===================== =============== ================== +Object :meth:`buffer` :meth:`memoryview` +--------------------- --------------- ------------------ +:class:`mmap.mmap` y n +:class:`array.array` y n +:class:`bytearray` y y +:class:`str` y y +:class:`unicode` y n +===================== =============== ================== - .. cmember:: void *buf +(y/n denotes whether or not the given type is supported by the Python-level +:meth:`buffer` or :meth:`memoryview` methods.) - A pointer to the start of the memory for the object. +memoryview objects +================== - .. cmember:: Py_ssize_t len - :noindex: +More information on the memoryview interface is provided in the section +:ref:`memoryview-structs`, under the description for :ctype:`Py_buffer`. - The total length of the memory in bytes. +The "memoryview object" is defined in :file:`memoryobject.h` +and is included in :file:`Python.h`. +A memoryview object is an extended buffer Python-level object that replaces the +buffer object in Python 3.0. - .. cmember:: int readonly +In addition to the generic buffer support (slicing, indexing, etc), +:class:`memoryview` objects also support ``tolist()`` and ``tobytes()`` which +returns list and str objects respectively. - An indicator of whether the buffer is read only. +.. _`memoryview API`: - .. cmember:: const char *format - :noindex: +New buffer related functions +============================ - A *NULL* terminated string in :mod:`struct` module style syntax giving - the contents of the elements available through the buffer. If this is - *NULL*, ``"B"`` (unsigned bytes) is assumed. +.. cfunction:: PyObject* PyMemoryView_FromObject(PyObject *obj) - .. cmember:: int ndim - - The number of dimensions the memory represents as a multi-dimensional - array. If it is 0, :cdata:`strides` and :cdata:`suboffsets` must be - *NULL*. - - .. cmember:: Py_ssize_t *shape - - An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the - shape of the memory as a multi-dimensional array. Note that - ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to - :cdata:`len`. - - .. cmember:: Py_ssize_t *strides - - An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the - number of bytes to skip to get to a new element in each dimension. - - .. cmember:: Py_ssize_t *suboffsets - - An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim`. If these - suboffset numbers are greater than or equal to 0, then the value stored - along the indicated dimension is a pointer and the suboffset value - dictates how many bytes to add to the pointer after de-referencing. A - suboffset value that it negative indicates that no de-referencing should - occur (striding in a contiguous memory block). - - Here is a function that returns a pointer to the element in an N-D array - pointed to by an N-dimesional index when there are both non-NULL strides - and suboffsets:: - - void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, - Py_ssize_t *suboffsets, Py_ssize_t *indices) { - char *pointer = (char*)buf; - int i; - for (i = 0; i < ndim; i++) { - pointer += strides[i] * indices[i]; - if (suboffsets[i] >=0 ) { - pointer = *((char**)pointer) + suboffsets[i]; - } - } - return (void*)pointer; - } - - - .. cmember:: Py_ssize_t itemsize - - This is a storage for the itemsize (in bytes) of each element of the - shared memory. It is technically un-necessary as it can be obtained - using :cfunc:`PyBuffer_SizeFromFormat`, however an exporter may know - this information without parsing the format string and it is necessary - to know the itemsize for proper interpretation of striding. Therefore, - storing it is more convenient and faster. - - .. cmember:: void *internal - - This is for use internally by the exporting object. For example, this - might be re-cast as an integer by the exporter and used to store flags - about whether or not the shape, strides, and suboffsets arrays must be - freed when the buffer is released. The consumer should never alter this - value. - - -Buffer related functions -======================== - + Return a memoryview object from an object that defines the buffer interface. .. cfunction:: int PyObject_CheckBuffer(PyObject *obj) @@ -300,22 +255,8 @@ only share a contiguous chunk of memory of "unsigned bytes" of the given length. Return 0 on success and -1 (with raising an error) on error. - -MemoryView objects -================== - -A memoryview object is an extended buffer object that could replace the buffer -object (but doesn't have to as that could be kept as a simple 1-d memoryview -object). It, unlike :ctype:`Py_buffer`, is a Python object (exposed as -:class:`memoryview` in :mod:`builtins`), so it can be used with Python code. - -.. cfunction:: PyObject* PyMemoryView_FromObject(PyObject *obj) - - Return a memoryview object from an object that defines the buffer interface. - - -Old-style buffer objects -======================== +buffer objects +============== .. index:: single: PyBufferProcs @@ -323,21 +264,10 @@ :ref:`buffer-structs`, under the description for :ctype:`PyBufferProcs`. A "buffer object" is defined in the :file:`bufferobject.h` header (included by -:file:`Python.h`). These objects look very similar to string objects at the -Python programming level: they support slicing, indexing, concatenation, and -some other standard string operations. However, their data can come from one -of two sources: from a block of memory, or from another object which exports -the buffer interface. +:file:`Python.h`). -Buffer objects are useful as a way to expose the data from another object's -buffer interface to the Python programmer. They can also be used as a -zero-copy slicing mechanism. Using their ability to reference a block of -memory, it is possible to expose any data to the Python programmer quite -easily. The memory could be a large, constant array in a C extension, it could -be a raw block of memory for manipulation before passing to an operating -system library, or it could be used to pass around structured data in its -native, in-memory format. - +Old buffer related functions +============================ .. ctype:: PyBufferObject diff -r 43e0ceb5dde9 Doc/c-api/typeobj.rst --- a/Doc/c-api/typeobj.rst Fri Feb 08 07:56:45 2013 -0800 +++ b/Doc/c-api/typeobj.rst Wed Feb 13 19:45:50 2013 -0800 @@ -1444,3 +1444,93 @@ Return the size of the segment *segment* that *ptrptr* is set to. ``*ptrptr`` is set to the memory buffer. Returns ``-1`` on error. + +.. _memoryview-structs: + +MemoryView Object Structure +=========================== + +.. ctype:: Py_buffer + + .. cmember:: void *buf + + A pointer to the start of the memory for the object. + + .. cmember:: Py_ssize_t len + :noindex: + + The total length of the memory in bytes. + + .. cmember:: int readonly + + An indicator of whether the buffer is read only. + + .. cmember:: const char *format + :noindex: + + A *NULL* terminated string in :mod:`struct` module style syntax giving + the contents of the elements available through the buffer. If this is + *NULL*, ``"B"`` (unsigned bytes) is assumed. + + .. cmember:: int ndim + + The number of dimensions the memory represents as a multi-dimensional + array. If it is 0, :cdata:`strides` and :cdata:`suboffsets` must be + *NULL*. + + .. cmember:: Py_ssize_t *shape + + An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the + shape of the memory as a multi-dimensional array. Note that + ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to + :cdata:`len`. + + .. cmember:: Py_ssize_t *strides + + An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim` giving the + number of bytes to skip to get to a new element in each dimension. + + .. cmember:: Py_ssize_t *suboffsets + + An array of :ctype:`Py_ssize_t`\s the length of :cdata:`ndim`. If these + suboffset numbers are greater than or equal to 0, then the value stored + along the indicated dimension is a pointer and the suboffset value + dictates how many bytes to add to the pointer after de-referencing. A + suboffset value that it negative indicates that no de-referencing should + occur (striding in a contiguous memory block). + + Here is a function that returns a pointer to the element in an N-D array + pointed to by an N-dimesional index when there are both non-NULL strides + and suboffsets:: + + void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, + Py_ssize_t *suboffsets, Py_ssize_t *indices) { + char *pointer = (char*)buf; + int i; + for (i = 0; i < ndim; i++) { + pointer += strides[i] * indices[i]; + if (suboffsets[i] >=0 ) { + pointer = *((char**)pointer) + suboffsets[i]; + } + } + return (void*)pointer; + } + + + .. cmember:: Py_ssize_t itemsize + + This is a storage for the itemsize (in bytes) of each element of the + shared memory. It is technically un-necessary as it can be obtained + using :cfunc:`PyBuffer_SizeFromFormat`, however an exporter may know + this information without parsing the format string and it is necessary + to know the itemsize for proper interpretation of striding. Therefore, + storing it is more convenient and faster. + + .. cmember:: void *internal + + This is for use internally by the exporting object. For example, this + might be re-cast as an integer by the exporter and used to store flags + about whether or not the shape, strides, and suboffsets arrays must be + freed when the buffer is released. The consumer should never alter this + value. +