diff -r 69b354a8c90f -r 78fb1181787a Doc/c-api/buffer.rst --- a/Doc/c-api/buffer.rst Tue Aug 09 18:09:21 2011 +0200 +++ b/Doc/c-api/buffer.rst Tue Aug 09 20:28:59 2011 +0200 @@ -7,6 +7,7 @@ .. sectionauthor:: Greg Stein .. sectionauthor:: Benjamin Peterson +.. sectionauthor:: Stefan Krah .. index:: @@ -20,7 +21,7 @@ While each of these types have their own semantics, they share the common characteristic of being backed by a possibly large memory buffer. It is -then desireable, in some situations, to access that buffer directly and +then desirable, in some situations, to access that buffer directly and without intermediate copying. Python provides such a facility at the C level in the form of the *buffer @@ -60,8 +61,10 @@ resource leaks. -The buffer structure -==================== +.. _buffer-structure: + +Buffer structure +================ Buffer structures (or simply "buffers") are useful as a way to expose the binary data from another object to the Python programmer. They can also be @@ -81,93 +84,306 @@ .. c:type:: Py_buffer - .. c:member:: void *buf + .. c:member:: void \*obj - A pointer to the start of the memory for the object. + A new reference to the exporting object or *NULL*. The reference is owned + by the consumer and automatically decremented and set to *NULL* by + :c:func:`PyBuffer_Release`. + + For temporary buffers that are wrapped by :c:func:`PyMemoryView_FromBuffer` + this field must be *NULL*. + + .. c:member:: void \*buf + + A pointer to the start of the logical structure described by the buffer + fields. This can be any location within the underlying physical memory + block of the exporter. For example, with negative :c:member:`~Py_buffer.strides` + the value may point to the end of the memory block. .. c:member:: Py_ssize_t len - :noindex: - The total length of the memory in bytes. + ``product(shape) * itemsize``. This is NOT the length of the underlying + memory block, but the length that the logical structure would have if it + were copied to a contiguous representation. + + Accessing ``((char *)buf)[0] up to ((char *)buf)[len-1]`` is only valid + if the buffer has been obtained by a request that guarantees contiguity. In + most cases such a request will be :c:macro:`PyBUF_SIMPLE` or :c:macro:`PyBUF_WRITABLE`. .. c:member:: int readonly - An indicator of whether the buffer is read only. + An indicator of whether the buffer is read-only. This field is controlled + by the :c:macro:`PyBUF_WRITABLE` flag. - .. c:member:: const char *format - :noindex: + .. c:member:: Py_ssize_t itemsize - A *NULL* terminated string in :mod:`struct` module style syntax giving - the contents of the elements available through the buffer. If this is - *NULL*, ``"B"`` (unsigned bytes) is assumed. + Item size in bytes of a single element. Same as the value of :func:`struct.calcsize` + called on non-NULL :c:member:`~Py_buffer.format` values. + + .. c:member:: const char \*format + + A *NUL* terminated string in :mod:`struct` module style syntax describing + the contents of a single item. If this is *NULL*, ``"B"`` (unsigned bytes) + is assumed. + + This field is controlled by the :c:macro:`PyBUF_FORMAT` flag. .. c:member:: int ndim - The number of dimensions the memory represents as a multi-dimensional - array. If it is 0, :c:data:`strides` and :c:data:`suboffsets` must be - *NULL*. + The number of dimensions the memory represents as an n-dimensional array. + If it is 0, :c:member:`~Py_Buffer.buf` points to a single item representing + a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides` + and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*. - .. c:member:: Py_ssize_t *shape + .. c:member:: Py_ssize_t \*shape - An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the - shape of the memory as a multi-dimensional array. Note that - ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to - :c:data:`len`. + An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` + indicating the shape of the memory as an n-dimensional array. Note that + ``shape[0] * ... * shape[ndim-1] * itemsize`` MUST be equal to + :c:member:`~Py_buffer.len`. - .. c:member:: Py_ssize_t *strides + Shape values are restricted to ``shape[n] >= 0``. The case + ``shape[n] = 0`` requires special attention. See `complex arrays`_ + for further information. - An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the - number of bytes to skip to get to a new element in each dimension. + The shape array is read-only for the consumer. - .. c:member:: Py_ssize_t *suboffsets + .. c:member:: Py_ssize_t \*strides - An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim`. If these - suboffset numbers are greater than or equal to 0, then the value stored - along the indicated dimension is a pointer and the suboffset value - dictates how many bytes to add to the pointer after de-referencing. A - suboffset value that it negative indicates that no de-referencing should - occur (striding in a contiguous memory block). + An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` + giving the number of bytes to skip to get to a new element in each + dimension. - Here is a function that returns a pointer to the element in an N-D array - pointed to by an N-dimensional index when there are both non-NULL strides - and suboffsets:: + Stride values can be any integer. For regular arrays, strides are + usually positive, but a consumer MUST be able to handle the case + ``strides[n] <= 0``. See `complex arrays`_ for further information. - void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, - Py_ssize_t *suboffsets, Py_ssize_t *indices) { - char *pointer = (char*)buf; - int i; - for (i = 0; i < ndim; i++) { - pointer += strides[i] * indices[i]; - if (suboffsets[i] >=0 ) { - pointer = *((char**)pointer) + suboffsets[i]; - } - } - return (void*)pointer; - } + The strides array is read-only for the consumer. + .. c:member:: Py_ssize_t \*suboffsets - .. c:member:: Py_ssize_t itemsize + An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`. + If ``suboffsets[n] >= 0``, the values stored along the nth dimension are + pointers and the suboffset value dictates how many bytes to add to each + pointer after de-referencing. A suboffset value that is negative + indicates that no de-referencing should occur (striding in a contiguous + memory block). - This is a storage for the itemsize (in bytes) of each element of the - shared memory. It is technically un-necessary as it can be obtained - using :c:func:`PyBuffer_SizeFromFormat`, however an exporter may know - this information without parsing the format string and it is necessary - to know the itemsize for proper interpretation of striding. Therefore, - storing it is more convenient and faster. + This type of array representation is used by the Python Imaging Library + (PIL). See `complex arrays`_ for further information how to access elements + of such an array. - .. c:member:: void *internal + The suboffsets array is read-only for the consumer. + + .. c:member:: void \*internal This is for use internally by the exporting object. For example, this might be re-cast as an integer by the exporter and used to store flags about whether or not the shape, strides, and suboffsets arrays must be - freed when the buffer is released. The consumer should never alter this + freed when the buffer is released. The consumer MUST NOT alter this value. +.. _buffer-request-types: + +Buffer request types +==================== + +Buffers are usually obtained by sending a buffer request to an exporting +object via :c:func:`PyObject_GetBuffer`. Since the complexity of the logical +structure of the memory can vary drastically, the consumer uses the *flags* +argument to specify the exact buffer type it can handle. + +All :c:data:`Py_buffer` fields are unambiguously defined by the request +type. + +request-independent fields +~~~~~~~~~~~~~~~~~~~~~~~~~~ +The following fields are not influenced by *flags* and must always be filled in +with the correct values: :c:member:`~Py_buffer.obj`, :c:member:`~Py_buffer.buf`, +:c:member:`~Py_buffer.len`, :c:member:`~Py_buffer.itemsize`, :c:member:`~Py_buffer.ndim`. + + +readonly, format +~~~~~~~~~~~~~~~~ + + .. c:macro:: PyBUF_WRITABLE + + Controls the :c:member:`~Py_buffer.readonly` field. If set, the exporter + MUST return a writable buffer or failure. Otherwise, the exporter MAY + return a readable or writable buffer, but the choice MUST be consistent + for all consumers. + + .. c:macro:: PyBUF_FORMAT + + Controls the :c:member:`~Py_buffer.format` field. If set, this field MUST + be filled in correctly. Otherwise, this field MUST be *NULL*. + + +These two flags can be \|'d to any of the flags in the next section. +Since :c:macro:`PyBUF_SIMPLE` is defined as 0, :c:macro:`PyBUF_WRITABLE` +can be used as a stand-alone flag to request a simple writable buffer. + + +shape, strides, suboffsets +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The flags that control the logical structure of the memory are listed +in decreasing order of complexity. Note that each flag contains all bits +of the flags below it. + + ++-----------------------------+-------+---------+------------+ +| Request | shape | strides | suboffsets | ++=============================+=======+=========+============+ +| .. c:macro:: PyBUF_INDIRECT | yes | yes | if needed | ++-----------------------------+-------+---------+------------+ +| .. c:macro:: PyBUF_STRIDES | yes | yes | NULL | ++-----------------------------+-------+---------+------------+ +| .. c:macro:: PyBUF_ND | yes | NULL | NULL | ++-----------------------------+-------+---------+------------+ +| .. c:macro:: PyBUF_SIMPLE | NULL | NULL | NULL | ++-----------------------------+-------+---------+------------+ + + +contiguity requests +~~~~~~~~~~~~~~~~~~~ + +C or Fortran contiguity can be explicitly requested, with and without stride +information. Without stride information, the buffer must be C-contiguous. + ++-----------------------------------+-------+---------+------------+--------+ +| Request | shape | strides | suboffsets | contig | ++===================================+=======+=========+============+========+ +| .. c:macro:: PyBUF_C_CONTIGUOUS | yes | yes | NULL | C | ++-----------------------------------+-------+---------+------------+--------+ +| .. c:macro:: PyBUF_F_CONTIGUOUS | yes | yes | NULL | F | ++-----------------------------------+-------+---------+------------+--------+ +| .. c:macro:: PyBUF_ANY_CONTIGUOUS | yes | yes | NULL | C or F | ++-----------------------------------+-------+---------+------------+--------+ +| .. c:macro:: PyBUF_ND | yes | NULL | NULL | C | ++-----------------------------------+-------+---------+------------+--------+ + + +compound requests +~~~~~~~~~~~~~~~~~ + +All possible requests are fully defined by some combination of the flags in +the previous section. For convenience, the buffer protocol provides frequently +used combinations as single flags. + +In the following table *U* stands for undefined contiguity. The consumer would +have to call :c:func:`PyBuffer_IsContiguous` to determine contiguity. + + + ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| Request | shape | strides | suboffsets | contig | readonly | format | ++===============================+=======+=========+============+========+==========+========+ +| .. c:macro:: PyBUF_FULL | yes | yes | if needed | U | 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_FULL_RO | yes | yes | if needed | U | 1 or 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_RECORDS | yes | yes | NULL | U | 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_RECORDS_RO | yes | yes | NULL | U | 1 or 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_STRIDED | yes | yes | NULL | U | 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_STRIDED_RO | yes | yes | NULL | U | 1 or 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_CONTIG | yes | NULL | NULL | C | 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_CONTIG_RO | yes | NULL | NULL | C | 1 or 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ + + +Complex arrays +============== + +NumPy-style: shape and strides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The logical structure of NumPy-style arrays is defined by :c:member:`~Py_buffer.itemsize`, +:c:member:`~Py_buffer.ndim`, :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides`. + +If ``ndim = 0``, the memory location pointed to by :c:member:`~Py_buffer.buf` is +interpreted as a scalar of size :c:member:`~Py_buffer.itemsize`. In that case, +both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are *NULL*. + +If :c:member:`~Py_buffer.strides` is *NULL*, the array is interpreted as +as standard n-dimensional C-array. Otherwise, the consumer must access an +n-dimensional array as follows: + + ``ptr = (char *)buf + indices[0] * strides[0] + ... + indices[n-1] * strides[n-1]`` + ``item = *((typeof(item) *)ptr);`` + + +As noted above, :c:member:`~Py_buffer.buf` can point to any location within +the actual memory block. An exporter can check the validity of a buffer with +this function: + +.. code-block:: python + + def verify_structure(memlen, itemsize, ndim, shape, strides, offset): + """Verify that the parameters represent a valid array within + the bounds of the allocated memory: + char *mem: start of the physical memory block + memlen: length of the physical memory block + offset: (char *)buf - mem + """ + if offset % itemsize: + return False + if offset < 0 or offset+itemsize > memlen: + return False + if any(v % itemsize for v in strides): + return False + + if ndim <= 0: + return ndim == 0 and not shape and not strides + if 0 in shape: + return True + + imin = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] <= 0) + imax = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] > 0) + + return 0 <= offset+imin and offset+imax+itemsize <= memlen + + +PIL-style: shape, strides and suboffsets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In addition to the regular items, PIL-style arrays can contain pointers +that must be followed in order to get to the next element in a dimension. +For example, the regular three-dimensional C-array ``char v[2][2][3]`` can +also be viewed as an array of 2 pointers to 2 two-dimensional arrays: +``char (*v[2])[2][3]``. In suboffsets representation, those two pointers +can be embedded at the start of :c:member:`~Py_buffer.buf`, pointing +to two ``char x[2][3]`` arrays that can be located anywhere in memory. + + +Here is a function that returns a pointer to the element in an N-D array +pointed to by an N-dimensional index when there are both non-NULL strides +and suboffsets:: + + void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, + Py_ssize_t *suboffsets, Py_ssize_t *indices) { + char *pointer = (char*)buf; + int i; + for (i = 0; i < ndim; i++) { + pointer += strides[i] * indices[i]; + if (suboffsets[i] >=0 ) { + pointer = *((char**)pointer) + suboffsets[i]; + } + } + return (void*)pointer; + } + Buffer-related functions ======================== - .. c:function:: int PyObject_CheckBuffer(PyObject *obj) Return 1 if *obj* supports the buffer interface otherwise 0. When 1 is @@ -175,133 +391,36 @@ succeed. -.. c:function:: int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) +.. c:function:: int PyObject_GetBuffer(PyObject *exporter, Py_buffer *view, int flags) - Export a view over some internal data from the target object *obj*. - *obj* must not be NULL, and *view* must point to an existing - :c:type:`Py_buffer` structure allocated by the caller (most uses of - this function will simply declare a local variable of type - :c:type:`Py_buffer`). The *flags* argument is a bit field indicating - what kind of buffer is requested. The buffer interface allows - for complicated memory layout possibilities; however, some callers - won't want to handle all the complexity and instead request a simple - view of the target object (using :c:macro:`PyBUF_SIMPLE` for a read-only - view and :c:macro:`PyBUF_WRITABLE` for a read-write view). + Send a request to *exporter* to fill in *view* as specified by *flags*. + If the exporter cannot provide a buffer of the exact type, it MUST raise + :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to *NULL* and + return -1. - Some exporters may not be able to share memory in every possible way and - may need to raise errors to signal to some consumers that something is - just not possible. These errors should be a :exc:`BufferError` unless - there is another error that is actually causing the problem. The - exporter can use flags information to simplify how much of the - :c:data:`Py_buffer` structure is filled in with non-default values and/or - raise an error if the object can't support a simpler view of its memory. + On success, fill in *view*, set :c:member:`view->obj` to a new reference + to *exporter* and return 0. - On success, 0 is returned and the *view* structure is filled with useful - values. On error, -1 is returned and an exception is raised; the *view* - is left in an undefined state. - - The following are the possible values to the *flags* arguments. - - .. c:macro:: PyBUF_SIMPLE - - This is the default flag. The returned buffer exposes a read-only - memory area. The format of data is assumed to be raw unsigned bytes, - without any particular structure. This is a "stand-alone" flag - constant. It never needs to be '|'d to the others. The exporter will - raise an error if it cannot provide such a contiguous buffer of bytes. - - .. c:macro:: PyBUF_WRITABLE - - Like :c:macro:`PyBUF_SIMPLE`, but the returned buffer is writable. If - the exporter doesn't support writable buffers, an error is raised. - - .. c:macro:: PyBUF_STRIDES - - This implies :c:macro:`PyBUF_ND`. The returned buffer must provide - strides information (i.e. the strides cannot be NULL). This would be - used when the consumer can handle strided, discontiguous arrays. - Handling strides automatically assumes you can handle shape. The - exporter can raise an error if a strided representation of the data is - not possible (i.e. without the suboffsets). - - .. c:macro:: PyBUF_ND - - The returned buffer must provide shape information. The memory will be - assumed C-style contiguous (last dimension varies the fastest). The - exporter may raise an error if it cannot provide this kind of - contiguous buffer. If this is not given then shape will be *NULL*. - - .. c:macro:: PyBUF_C_CONTIGUOUS - PyBUF_F_CONTIGUOUS - PyBUF_ANY_CONTIGUOUS - - These flags indicate that the contiguity returned buffer must be - respectively, C-contiguous (last dimension varies the fastest), Fortran - contiguous (first dimension varies the fastest) or either one. All of - these flags imply :c:macro:`PyBUF_STRIDES` and guarantee that the - strides buffer info structure will be filled in correctly. - - .. c:macro:: PyBUF_INDIRECT - - This flag indicates the returned buffer must have suboffsets - information (which can be NULL if no suboffsets are needed). This can - be used when the consumer can handle indirect array referencing implied - by these suboffsets. This implies :c:macro:`PyBUF_STRIDES`. - - .. c:macro:: PyBUF_FORMAT - - The returned buffer must have true format information if this flag is - provided. This would be used when the consumer is going to be checking - for what 'kind' of data is actually stored. An exporter should always - be able to provide this information if requested. If format is not - explicitly requested then the format must be returned as *NULL* (which - means ``'B'``, or unsigned bytes). - - .. c:macro:: PyBUF_STRIDED - - This is equivalent to ``(PyBUF_STRIDES | PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_STRIDED_RO - - This is equivalent to ``(PyBUF_STRIDES)``. - - .. c:macro:: PyBUF_RECORDS - - This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT | - PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_RECORDS_RO - - This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT)``. - - .. c:macro:: PyBUF_FULL - - This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT | - PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_FULL_RO - - This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT)``. - - .. c:macro:: PyBUF_CONTIG - - This is equivalent to ``(PyBUF_ND | PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_CONTIG_RO - - This is equivalent to ``(PyBUF_ND)``. + Calls to :c:func:`PyObject_GetBuffer` and :c:func:`PyBuffer_Release` must + always be paired similar to to :c:func:`malloc` and :c:func:`free`. Thus, + after the consumer is done with the buffer, :c:func:`PyBuffer_Release` must + be called exactly once. .. c:function:: void PyBuffer_Release(Py_buffer *view) - Release the buffer *view*. This should be called when the buffer is no - longer being used as it may free memory from it. + Release the buffer *view* and decrement :c:member:`view->obj`. This function + MUST be called when the buffer is no longer being used, otherwise reference + leaks may occur. + + It is an error to call this function on a buffer that was not obtained via + :c:func:`PyObject_GetBuffer`. .. c:function:: Py_ssize_t PyBuffer_SizeFromFormat(const char *) - Return the implied :c:data:`~Py_buffer.itemsize` from the struct-stype - :c:data:`~Py_buffer.format`. + Return the implied :c:data:`~Py_buffer.itemsize` from :c:data:`~Py_buffer.format`. + This function is not yet implemented. .. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) @@ -318,9 +437,23 @@ given shape with the given number of bytes per element. -.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags) +.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *exporter, void *buf, Py_ssize_t len, int readonly, int flags) - Fill in a buffer-info structure, *view*, correctly for an exporter that can - only share a contiguous chunk of memory of "unsigned bytes" of the given - length. Return 0 on success and -1 (with raising an error) on error. + Handle buffer requests for an exporter that wants to expose *buf* of size *len* + with writability set according to *readonly*. *buf* is interpreted a a sequence + of unsigned bytes. + The *flags* argument indicates the request type. This function always fills in + *view* as specified by flags, unless *buf* has been designated as read-only + and :c:macro:`PyBUF_WRITABLE` is set in *flags*. + + On success, set :c:member:`view->obj` to a new reference to *exporter* and + return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set + :c:member:`view->obj` to *NULL* and return -1; + + If this function is used as part of a :ref:`getbufferproc `, + *exporter* MUST be set to the exporting object. Otherwise, *exporter* MUST + be NULL. + + + diff -r 69b354a8c90f -r 78fb1181787a Doc/c-api/memoryview.rst --- a/Doc/c-api/memoryview.rst Tue Aug 09 18:09:21 2011 +0200 +++ b/Doc/c-api/memoryview.rst Tue Aug 09 20:28:59 2011 +0200 @@ -17,16 +17,24 @@ Create a memoryview object from an object that provides the buffer interface. If *obj* supports writable buffer exports, the memoryview object will be - readable and writable, other it will be read-only. + read/write, otherwise it may be either read-only or read/write at the + discretion of the exporter. + +.. c:function:: PyObject *PyMemoryView_FromBytes(char *bytes, Py_ssize_t size, int flags) + + Create a memoryview object wrapping the given string *bytes*. *flags* can + be one of :c:macro:`PyBUF_READ` or :c:macro:`PyBUF_WRITE`. .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view) Create a memoryview object wrapping the given buffer structure *view*. - The memoryview object then owns the buffer represented by *view*, which - means you shouldn't try to call :c:func:`PyBuffer_Release` yourself: it - will be done on deallocation of the memoryview object. + *view* MUST have full buffer information. If :c:func:`PyBuffer_FillInfo` + is used to create the view, either :c:macro:`PyBUF_FULL` or + :c:macro:`PyBUF_FULL_RO` must be used for the *flags* parameter. + For simple byte buffers, :c:func:`PyMemoryView_FromBytes` is the + preferred function. .. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) diff -r 69b354a8c90f -r 78fb1181787a Doc/c-api/typeobj.rst --- a/Doc/c-api/typeobj.rst Tue Aug 09 18:09:21 2011 +0200 +++ b/Doc/c-api/typeobj.rst Tue Aug 09 20:28:59 2011 +0200 @@ -1193,46 +1193,69 @@ .. sectionauthor:: Greg J. Stein .. sectionauthor:: Benjamin Peterson - - -The :ref:`buffer interface ` exports a model where an object can expose its internal -data. - -If an object does not export the buffer interface, then its :attr:`tp_as_buffer` -member in the :c:type:`PyTypeObject` structure should be *NULL*. Otherwise, the -:attr:`tp_as_buffer` will point to a :c:type:`PyBufferProcs` structure. - +.. sectionauthor:: Stefan Krah .. c:type:: PyBufferProcs - Structure used to hold the function pointers which define an implementation of - the buffer protocol. + This structure holds pointers to the functions required by the + :ref:`Buffer protocol `. The protocol defines how + an exporter object can expose its internal data to consumer objects. - .. c:member:: getbufferproc bf_getbuffer +.. c:member:: getbufferproc PyBufferProcs.bf_getbuffer - This should fill a :c:type:`Py_buffer` with the necessary data for - exporting the type. The signature of :data:`getbufferproc` is ``int - (PyObject *obj, Py_buffer *view, int flags)``. *obj* is the object to - export, *view* is the :c:type:`Py_buffer` struct to fill, and *flags* gives - the conditions the caller wants the memory under. (See - :c:func:`PyObject_GetBuffer` for all flags.) :c:member:`bf_getbuffer` is - responsible for filling *view* with the appropriate information. - (:c:func:`PyBuffer_FillView` can be used in simple cases.) See - :c:type:`Py_buffer`\s docs for what needs to be filled in. + The signature of this function is:: + int (PyObject *exporter, Py_buffer *view, int flags); - .. c:member:: releasebufferproc bf_releasebuffer + Handle a request to *exporter* to fill in *view* as specified by *flags*. + A standard implementation of this function will take these steps: - This should release the resources of the buffer. The signature of - :c:data:`releasebufferproc` is ``void (PyObject *obj, Py_buffer *view)``. - If the :c:data:`bf_releasebuffer` function is not provided (i.e. it is - *NULL*), then it does not ever need to be called. + - Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`, + set :c:data:`view->obj` to *NULL* and return -1. - The exporter of the buffer interface must make sure that any memory - pointed to in the :c:type:`Py_buffer` structure remains valid until - releasebuffer is called. Exporters will need to define a - :c:data:`bf_releasebuffer` function if they can re-allocate their memory, - strides, shape, suboffsets, or format variables which they might share - through the struct bufferinfo. + - Fill in the requested fields. - See :c:func:`PyBuffer_Release`. + - Increment an internal counter for the number of exports. + + - Set :c:data:`view->obj` to *exporter* and increment :c:data:`view->obj`. + + - Return 0. + + The individual fields of *view* are described in section + :ref:`Buffer structure `, the rules how an exporter + must react to specific requests are in section + :ref:`Buffer request types `. + + All memory pointed to in the :c:type:`Py_buffer` structure belongs to + the exporter and must remain valid until there are no consumers left. + :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`, + :c:member:`~Py_buffer.suboffsets` and :c:member:`~Py_buffer.internal` + are read-only for the consumer. + + :c:func:`PyBuffer_FillInfo` provides an easy way of exposing a simple + bytes buffer while dealing correctly with all request types. + + :c:func:`PyObject_GetBuffer` is the interface for the consumer that + wraps this function. + +.. c:member:: releasebufferproc PyBufferProcs.bf_releasebuffer + + The signature of this function is:: + + void (PyObject *exporter, Py_buffer *view); + + Handle a request to release the resources of the buffer. If no resources + need to be released, this field may be *NULL*. A standard implementation + of this function will take these steps: + + - Decrement an internal counter for the number of exports. + + - If the counter is 0, free all memory associated with *view*. + + + This function MUST NOT decrement :c:data:`view->obj`, since that is + done automatically in :c:func:`PyBuffer_Release`. + + + :c:func:`PyBuffer_Release` is the interface for the consumer that + wraps this function. diff -r 69b354a8c90f -r 78fb1181787a Include/memoryobject.h --- a/Include/memoryobject.h Tue Aug 09 18:09:21 2011 +0200 +++ b/Include/memoryobject.h Tue Aug 09 20:28:59 2011 +0200 @@ -6,8 +6,10 @@ extern "C" { #endif +PyAPI_DATA(PyTypeObject) PyManagedBuffer_Type; PyAPI_DATA(PyTypeObject) PyMemoryView_Type; +#define PyManagedBuffer_Check(op) (Py_TYPE(op) == &PyManagedBuffer_Type) #define PyMemoryView_Check(op) (Py_TYPE(op) == &PyMemoryView_Type) #ifndef Py_LIMITED_API @@ -57,8 +59,8 @@ #ifndef Py_LIMITED_API PyAPI_FUNC(PyObject *) PyMemoryView_FromBuffer(Py_buffer *info); - /* create new if bufptr is NULL - will be a new bytesobject in base */ +PyAPI_FUNC(PyObject *) PyMemoryView_FromBytes(char *bytes, Py_ssize_t size, + int flags); #endif @@ -68,7 +70,24 @@ #ifndef Py_LIMITED_API typedef struct { PyObject_HEAD - Py_buffer view; + int released; + Py_ssize_t exports; /* number of direct memoryview exports */ + Py_buffer master; /* snapshot buffer obtained from the original base object */ +} PyManagedBufferObject; + +/* To provide a sliced view, shape, strides and suboffsets may need to be + altered. If ndim <= Py_MEMORYVIEW_MAXSTATIC, the view uses the static + arrays. Otherwise, dynamic memory will be allocated. */ +#define Py_MEMORYVIEW_MAXSTATIC 3 +typedef struct { + PyObject_HEAD + PyManagedBufferObject *mbuf; + Py_ssize_t shape[Py_MEMORYVIEW_MAXSTATIC]; + Py_ssize_t strides[Py_MEMORYVIEW_MAXSTATIC]; + Py_ssize_t suboffsets[Py_MEMORYVIEW_MAXSTATIC]; + int released; + Py_ssize_t exports; /* number of buffer re-exports */ + Py_buffer view; /* private copy of the ancestor's view */ } PyMemoryViewObject; #endif diff -r 69b354a8c90f -r 78fb1181787a Include/object.h --- a/Include/object.h Tue Aug 09 18:09:21 2011 +0200 +++ b/Include/object.h Tue Aug 09 20:28:59 2011 +0200 @@ -157,8 +157,6 @@ Py_ssize_t *shape; Py_ssize_t *strides; Py_ssize_t *suboffsets; - Py_ssize_t smalltable[2]; /* static store for shape and strides of - mono-dimensional buffers. */ void *internal; } Py_buffer; diff -r 69b354a8c90f -r 78fb1181787a Lib/test/test_buffer.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/Lib/test/test_buffer.py Tue Aug 09 20:28:59 2011 +0200 @@ -0,0 +1,1405 @@ +# +# The ndarray object from _testbuffer.c is a complete implementation of +# a PEP-3118 buffer provider. It is independent from NumPy's ndarray +# and the tests don't require NumPy. +# +# If NumPy is present, some tests check both ndarray implementations +# against each other. +# + +import unittest +from test import support +from itertools import permutations, product +from random import randrange, sample +from sysconfig import get_config_var +from platform import architecture +import time, random +import warnings + +try: + from _testbuffer import * +except ImportError: + ndarray = None + +try: + import struct +except ImportError: + struct = None + +try: + with warnings.catch_warnings(): + from numpy import ndarray as numpy_array +except ImportError: + numpy_array = None + + +SHORT_TEST = True + + +# ====================================================================== +# Random lists by format specifier +# ====================================================================== + +FMTCHRS = "cbB?hHiIlLqQfdP" +TYPERANGE = {} +def get_type_range(fmt): + """Return range of a type specified by a format character.""" + if fmt == 'c': + lh = (0, 256) + elif fmt == '?': + lh = (0, 2) + elif fmt == 'f': + lh = (-(1<<63), 1<<63) + elif fmt == 'd': + lh = (-(1<<1023), 1<<1023) + else: + for exp in (128, 127, 64, 63, 32, 31, 16, 15, 8, 7): + try: + struct.pack(fmt, (1< memlen: + return False + if any(v % itemsize for v in strides): + return False + + if ndim <= 0: + return ndim == 0 and not shape and not strides + if 0 in shape: + return True + + imin = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] <= 0) + imax = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] > 0) + + return 0 <= offset+imin and offset+imax+itemsize <= memlen + +def indices(shape): + """Generate all possible tuples of indices.""" + iterables = [range(v) for v in shape] + return product(*iterables) + +def memory_index(indices, t): + """Location of an item in the underlying memory.""" + memlen, itemsize, ndim, shape, strides, offset = t + p = offset + for i in range(ndim): + p += strides[i]*indices[i] + return p + +def is_overlapping(t): + """The structure 't' is overlapping if at least one memory location + is visited twice while iterating through all possible tuples of + indices.""" + memlen, itemsize, ndim, shape, strides, offset = t + visited = 1<= 95 and valid: + minshape = 0 + elif n >= 90: + minshape = 1 + shape = [0] * ndim + + for i in range(ndim): + shape[i] = randrange(minshape, maxshape+1) + else: + ndim = len(shape) + + maxstride = 5 + n = randrange(100) + zero_stride = True if n >= 95 and n & 1 else False + + strides = [0] * ndim + strides[ndim-1] = itemsize * randrange(-maxstride, maxstride+1) + if not zero_stride and strides[ndim-1] == 0: + strides[ndim-1] = itemsize + + for i in range(ndim-2, -1, -1): + maxstride *= shape[i+1] if shape[i+1] else 1 + if zero_stride: + strides[i] = itemsize * randrange(-maxstride, maxstride+1) + else: + strides[i] = ((1,-1)[randrange(2)] * + itemsize * randrange(1, maxstride+1)) + + imin = imax = 0 + if not 0 in shape: + imin = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] <= 0) + imax = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] > 0) + + nitems = imax - imin + if valid: + offset = -imin * itemsize + memlen = offset + (imax+1) * itemsize + else: + memlen = (-imin + imax) * itemsize + offset = -imin-itemsize if randrange(2) == 0 else memlen + return memlen, itemsize, ndim, shape, strides, offset + +def randslice_from_slicelen(slicelen, listlen): + """Create a random slice of len slicelen that fits into listlen.""" + maxstart = listlen - slicelen + start = randrange(maxstart+1) + maxstep = (listlen - start) // slicelen if slicelen else 1 + step = randrange(1, maxstep+1) + stop = start + slicelen * step + s = slice(start, stop, step) + _, _, _, control = slice_indices(s, listlen) + if control != slicelen: + raise RuntimeError + return s + +def randslice_from_shape(ndim, shape): + """Create two sets of slices for an array x with shape 'shape' + such that shapeof(x[lslices]) == shapeof(x[rslices]).""" + lslices = [0] * ndim + rslices = [0] * ndim + for n in range(ndim): + l = shape[n] + slicelen = randrange(1, l+1) if l > 0 else 0 + lslices[n] = randslice_from_slicelen(slicelen, l) + rslices[n] = randslice_from_slicelen(slicelen, l) + return tuple(lslices), tuple(rslices) + +def rand_aligned_slices(maxdim=5, maxshape=16): + """Create (lshape, rshape, tuple(lslices), tuple(rslices)) such that + shapeof(x[lslices]) == shapeof(y[rslices]), where x is an array + with shape 'lshape' and y is an array with shape 'rshape'.""" + ndim = randrange(1, maxdim+1) + minshape = 2 + n = randrange(100) + if n >= 95: + minshape = 0 + elif n >= 90: + minshape = 1 + all_random = True if randrange(100) >= 80 else False + lshape = [0]*ndim; rshape = [0]*ndim + lslices = [0]*ndim; rslices = [0]*ndim + + for n in range(ndim): + small = randrange(minshape, maxshape+1) + big = randrange(minshape, maxshape+1) + if big < small: + big, small = small, big + + # Create a slice that fits the smaller value. + if all_random: + start = randrange(-small, small+1) + stop = randrange(-small, small+1) + step = (1,-1)[randrange(2)] * randrange(1, small+2) + s_small = slice(start, stop, step) + _, _, _, slicelen = slice_indices(s_small, small) + else: + slicelen = randrange(1, small+1) if small > 0 else 0 + s_small = randslice_from_slicelen(slicelen, small) + + # Create a slice of the same length for the bigger value. + s_big = randslice_from_slicelen(slicelen, big) + if randrange(2) == 0: + rshape[n], lshape[n] = big, small + rslices[n], lslices[n] = s_big, s_small + else: + rshape[n], lshape[n] = small, big + rslices[n], lslices[n] = s_small, s_big + + return lshape, rshape, tuple(lslices), tuple(rslices) + +def randitems_from_structure(fmt, t): + """Return a list of random items for structure 't' with format 'fmt'.""" + memlen, itemsize, _, _, _, _ = t + return randitems(fmt, memlen//itemsize, numpy_workaround=True) + +def ndarray_from_structure(items, fmt, t, flags=0): + """Return ndarray from the tuple returned by rand_structure()""" + memlen, itemsize, ndim, shape, strides, offset = t + return ndarray(items, shape=shape, strides=strides, fmt=fmt, offset=offset, + flags=ND_WRITABLE|flags) + +def numpy_array_from_structure(items, fmt, t): + """Return numpy_array from the tuple returned by rand_structure()""" + memlen, itemsize, ndim, shape, strides, offset = t + buf = bytearray(memlen) + for j, v in enumerate(items): + struct.pack_into(fmt, buf, j*itemsize, v) + return numpy_array(buffer=buf, shape=shape, strides=strides, + dtype=fmt, offset=offset) + + +# ====================================================================== +# Actual tests +# ====================================================================== + +def genslices(n): + """Generate all possible slices for a single dimension.""" + return product(range(-n, n+1), range(-n, n+1), range(-n, n+1)) + +def genslices_ndim(ndim, shape): + """Generate all possible slice tuples for 'shape'.""" + iterables = [genslices(shape[n]) for n in range(ndim)] + return product(*iterables) + +def rslice(n, allow_empty=False): + """Generate random slice for a single dimension of length n. + If zero=True, the slices may be empty, otherwise they will + be non-empty.""" + minlen = 0 if allow_empty or n == 0 else 1 + slicelen = randrange(minlen, n+1) + return randslice_from_slicelen(slicelen, n) + +def rslices(n, allow_empty=False): + """Generate random slices for a single dimenension.""" + for i in range(5): + yield rslice(n, allow_empty) + +def rslices_ndim(ndim, shape, iterations=25): + """Generate random slice tuples for 'shape'.""" + # non-empty slices + for v in range(iterations): + yield tuple(rslice(shape[n]) for n in range(ndim)) + # possibly empty slices + for v in range(iterations): + yield tuple(rslice(shape[n], allow_empty=True) for n in range(ndim)) + # invalid slices + yield tuple(slice(0,1,0) for _ in range(ndim)) + +def rpermutation(iterable, r=None): + pool = tuple(iterable) + r = len(pool) if r is None else r + yield tuple(random.sample(pool, r)) + + +ITERATIONS=100 +MAXDIM=5 +MAXSHAPE=10 + +if SHORT_TEST: + ITERATIONS=10 + MAXDIM=3 + MAXSHAPE=4 + genslices = rslices + genslices_ndim = rslices_ndim + permutations = rpermutation + + +@unittest.skipUnless(struct, 'struct module required for this test.') +@unittest.skipUnless(ndarray, 'ndarray object required for this test') +class TestBufferProtocol(unittest.TestCase): + + def setUp(self): + self.sizeof_void_p = get_config_var('SIZEOF_VOID_P') + if not self.sizeof_void_p: + self.sizeof_void_p = 8 if architecture()[0] == '64bit' else 4 + + def verify(self, nd, flags={-1}, + obj=-1, itemsize={1}, fmt=-1, readonly={1}, + ndim={1}, shape=-1, strides=-1, + lst=-1): + # Verify ndarray 'nd' against expected values. Default values are + # deliberately initialized to invalid types. + if shape: + expected_len = prod(shape)*itemsize + else: + expected_len = itemsize if (nd.flags&ND_SCALAR) else len(lst) + + suboffsets = () + if flags & ND_PIL: + self.assertGreater(ndim, 0) + strides = [self.sizeof_void_p] + list(strides[1:]) + suboffsets = [0] + [-1 for v in range(ndim-1)] + + self.assertIs(nd.obj, obj) + self.assertEqual(nd.len, expected_len ) + self.assertEqual(nd.itemsize, itemsize) + self.assertEqual(nd.fmt, fmt) + self.assertEqual(nd.readonly, readonly) + self.assertEqual(nd.ndim, ndim) + self.assertEqual(nd.shape, tuple(shape)) + self.assertEqual(nd.strides, tuple(strides)) + self.assertEqual(nd.suboffsets, tuple(suboffsets)) + + self.assertEqual(nd.tolist(), lst) + + def test_ndarray_exceptions(self): + nd = ndarray([9], [1]) + ndm = ndarray([9], [1], flags=ND_VAREXPORT) + + # Initialization of a new ndarray or mutation of an existing array. + for c in (ndarray, nd.push, ndm.push): + # Invalid types. + self.assertRaises(TypeError, c, {1,2,3}) + self.assertRaises(TypeError, c, [1,2,'3']) + self.assertRaises(TypeError, c, [1,2,(3,4)]) + self.assertRaises(TypeError, c, [1,2,3], shape={3}) + self.assertRaises(TypeError, c, [1,2,3], shape=[3], strides={1}) + self.assertRaises(TypeError, c, [1,2,3], shape=[3], offset=[]) + self.assertRaises(TypeError, c, [1], shape=[1], fmt={}) + self.assertRaises(TypeError, c, [1], shape=[1], flags={}) + self.assertRaises(TypeError, c, [1], shape=[1], getbuf={}) + + # ND_FORTRAN flag is only valid without strides. + self.assertRaises(TypeError, c, [1], shape=[1], strides=[1], + flags=ND_FORTRAN) + + # ND_PIL flag is only valid with ndim > 0. + self.assertRaises(TypeError, c, [1], shape=[], flags=ND_PIL) + + # Invalid items. + self.assertRaises(ValueError, c, [], shape=[1]) + self.assertRaises(ValueError, c, ['XXX'], shape=[1], fmt="L") + # Invalid combination of items and format. + self.assertRaises(struct.error, c, [1000], shape=[1], fmt="B") + self.assertRaises(ValueError, c, [1,(2,3)], shape=[2], fmt="B") + self.assertRaises(ValueError, c, [1,2,3], shape=[3], fmt="QL") + + # Invalid shape. + self.assertRaises(ValueError, c, [1], shape=[-1]) + self.assertRaises(ValueError, c, [1,2,3], shape=['3']) + self.assertRaises(OverflowError, c, [1], shape=[2**128]) + # prod(shape) * itemsize != len(items) + self.assertRaises(ValueError, c, [1,2,3,4,5], shape=[2,2], offset=3) + + # Invalid strides. + self.assertRaises(ValueError, c, [1,2,3], shape=[3], strides=['1']) + self.assertRaises(OverflowError, c, [1], shape=[1], + strides=[2**128]) + + # Invalid combination of strides and shape. + self.assertRaises(ValueError, c, [1,2], shape=[2,1], strides=[1]) + # Invalid combination of strides and format. + self.assertRaises(ValueError, c, [1,2,3,4], shape=[2], strides=[3], + fmt="L") + + # Invalid offset. + self.assertRaises(ValueError, c, [1,2,3], shape=[3], offset=4) + self.assertRaises(ValueError, c, [1,2,3], shape=[1], offset=3, + fmt="L") + + # Invalid format. + self.assertRaises(ValueError, c, [1,2,3], shape=[3], fmt="") + self.assertRaises(struct.error, c, [(1,2,3)], shape=[1], fmt="@#$") + + # Striding out of the memory bounds. + items = [1,2,3,4,5,6,7,8,9,10] + self.assertRaises(ValueError, c, items, shape=[2,3], + strides=[-3, -2], offset=5) + + # Constructing consumer: format argument invalid. + self.assertRaises(TypeError, c, bytearray(), fmt="Q") + + # Constructing original base object: getbuf argument invalid. + self.assertRaises(TypeError, c, [1], shape=[1], getbuf=PyBUF_FULL) + + # Shape argument is mandatory for original base objects. + self.assertRaises(TypeError, c, [1]) + + + # PyBUF_WRITABLE request to read-only provider. + self.assertRaises(BufferError, ndarray, b'123', getbuf=PyBUF_WRITABLE) + + # ND_VAREXPORT can only be specified during construction. + nd = ndarray([9], [1], flags=ND_VAREXPORT) + self.assertRaises(ValueError, nd.push, [1], [1], flags=ND_VAREXPORT) + + # Invalid operation for consumers: push/pop + nd = ndarray(b'123') + self.assertRaises(BufferError, nd.push, [1], [1]) + self.assertRaises(BufferError, nd.pop) + + # ND_VAREXPORT not set: push/pop fail with exported buffers + nd = ndarray([9], [1]) + nd.push([1], [1]) + m = memoryview(nd) + self.assertRaises(BufferError, nd.push, [1], [1]) + self.assertRaises(BufferError, nd.pop) + m.release() + nd.pop() + + # Single remaining buffer: pop fails + self.assertRaises(BufferError, nd.pop) + del nd + + def test_ndarray_linked_list(self): + for perm in permutations(range(5)): + m = [0]*5 + nd = ndarray([1,2,3], shape=[3], flags=ND_VAREXPORT) + m[0] = memoryview(nd) + + for i in range(1, 5): + nd.push([1,2,3], shape=[3]) + m[i] = memoryview(nd) + + for i in range(5): + m[perm[i]].release() + + self.assertRaises(BufferError, nd.pop) + del nd + + def test_ndarray_format_scalar(self): + # ndim = 0: scalar + for fmt, scalar, _ in iter_format(0): + itemsize = struct.calcsize(fmt) + nd = ndarray(scalar, shape=(), fmt=fmt) + self.verify(nd, flags=0, + obj=None, itemsize=itemsize, fmt=fmt, readonly=1, + ndim=0, shape=(), strides=(), + lst=scalar) + + def test_ndarray_format_shape(self): + # ndim = 1, shape = [n] + nitems = randrange(1, 10) + for fmt, items, _ in iter_format(nitems): + itemsize = struct.calcsize(fmt) + for flags in (0, ND_PIL): + nd = ndarray(items, shape=[nitems], fmt=fmt, flags=flags) + self.verify(nd, flags=flags, + obj=None, itemsize=itemsize, fmt=fmt, readonly=1, + ndim=1, shape=(nitems,), strides=(itemsize,), + lst=items) + + def test_ndarray_format_strides(self): + # ndim = 1, strides + nitems = randrange(1, 30) + for fmt, items, _ in iter_format(nitems): + itemsize = struct.calcsize(fmt) + for step in range(-5, 5): + if step == 0: + continue + + shape = [len(items[::step])] + strides = [step*itemsize] + offset = itemsize*(nitems-1) if step < 0 else 0 + + for flags in (0, ND_PIL): + nd = ndarray(items, shape=shape, strides=strides, fmt=fmt, + offset=offset, flags=flags) + self.verify(nd, flags=flags, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=1, shape=shape, strides=strides, + lst=items[::step]) + + def test_ndarray_fortran(self): + items = [1,2,3,4,5,6,7,8,9,10,11,12] + ex = ndarray(items, shape=(3, 4), strides=(1, 3)) + nd = ndarray(ex, getbuf=PyBUF_F_CONTIGUOUS) + self.assertEqual(nd.tolist(), farray(items, (3, 4))) + + def test_ndarray_multidim(self): + for ndim in range(5): + shape_t = [randrange(2, 6) for _ in range(ndim)] + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt = ''.join(sample(FMTCHRS, randrange(1, len(FMTCHRS)))) + items = randitems(fmt, nitems) + itemsize = struct.calcsize(fmt) + + for flags in (0, ND_PIL): + if ndim == 0 and flags == ND_PIL: + continue + + # C array + nd = ndarray(items, shape=shape, fmt=fmt, flags=flags) + + strides = strides_from_shape(ndim, shape, itemsize, 'C') + self.verify(nd, flags=flags, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=carray(items, shape)) + + # Fortran array + nd = ndarray(items, shape=shape, fmt=fmt, + flags=flags|ND_FORTRAN) + + strides = strides_from_shape(ndim, shape, itemsize, 'F') + self.verify(nd, flags=flags, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=farray(items, shape)) + + def test_ndarray_index_invalid(self): + # not writable + nd = ndarray([1], shape=[1]) + self.assertRaises(TypeError, nd.__setitem__, 1, 8) + + # cannot be deleted + nd = ndarray([1], shape=[1], flags=ND_WRITABLE) + self.assertRaises(TypeError, nd.__delitem__, 1) + + # overflow + nd = ndarray([1], shape=[1], flags=ND_WRITABLE) + self.assertRaises(OverflowError, nd.__getitem__, 1<<64) + self.assertRaises(OverflowError, nd.__setitem__, 1<<64, 8) + + # format + items = [1,2,3,4,5,6,7,8] + nd = ndarray(items, shape=[len(items)], fmt="B", flags=ND_WRITABLE) + self.assertRaises(struct.error, nd.__setitem__, 2, 300) + self.assertRaises(ValueError, nd.__setitem__, 1, (100, 200)) + + items = [(1,2), (3,4), (5,6)] + nd = ndarray(items, shape=[len(items)], fmt="LQ", flags=ND_WRITABLE) + self.assertRaises(ValueError, nd.__setitem__, 2, 300) + self.assertRaises(struct.error, nd.__setitem__, 1, (b'\x001', 200)) + + def test_ndarray_index_scalar(self): + # scalar + nd = ndarray(1, shape=(), flags=ND_WRITABLE) + + x = nd[()] + self.assertEqual(x.tolist(), nd.tolist()) + x = nd[...] + self.assertEqual(x.tolist(), nd.tolist()) + + self.assertRaises(TypeError, nd.__getitem__, 8) + self.assertRaises(TypeError, nd.__setitem__, 0, 8) + self.assertEqual(nd.tolist(), 1) + + def test_ndarray_index_null_strides(self): + items = [1,2,3,4,5,6,7,8] + ex = ndarray(items, shape=[2, 4], flags=ND_WRITABLE) + x = ndarray(ex, getbuf=PyBUF_CONTIG) + + # Subviews are only possible for full exporters. + self.assertRaises(BufferError, x.__getitem__, 1) + + # Same for slices. + self.assertRaises(BufferError, x.__getitem__, slice(3,5,1)) + + def test_ndarray_index_getitem_single(self): + # getitem + for fmt, items, _ in iter_format(5): + nd = ndarray(items, shape=[5], fmt=fmt) + for i in range(-5, 5): + self.assertEqual(nd[i], items[i]) + + self.assertRaises(IndexError, nd.__getitem__, -6) + self.assertRaises(IndexError, nd.__getitem__, 5) + + # getitem with null shape/strides + items = [1,2,3,4,5] + ex = ndarray(items, shape=[5], flags=ND_WRITABLE) + nd = ndarray(ex, getbuf=PyBUF_WRITABLE) + + for i in range(-5, 5): + self.assertEqual(nd[i], items[i]) + + def test_ndarray_index_setitem_single(self): + # assign single value + for fmt, items, single_item in iter_format(5): + nd = ndarray(items, shape=[5], fmt=fmt, flags=ND_WRITABLE) + for i in range(5): + items[i] = single_item + nd[i] = single_item + self.assertEqual(nd.tolist(), items) + + self.assertRaises(IndexError, nd.__setitem__, -6, single_item) + self.assertRaises(IndexError, nd.__setitem__, 5, single_item) + + # assign single value: lobject = robject + for fmt, items, single_item in iter_format(5): + nd = ndarray(items, shape=[5], fmt=fmt, flags=ND_WRITABLE) + for i in range(-5, 4): + items[i] = items[i+1] + nd[i] = nd[i+1] + self.assertEqual(nd.tolist(), items) + + def test_ndarray_index_getitem_multidim(self): + shape_t = (2, 3, 5) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt = ''.join(sample(FMTCHRS, randrange(1, len(FMTCHRS)))) + items = randitems(fmt, nitems) + itemsize = struct.calcsize(fmt) + + for flags in (0, ND_PIL): + # C array + nd = ndarray(items, shape=shape, fmt=fmt, flags=flags) + lst = carray(items, shape) + + for i in range(-shape[0], shape[0]): + self.assertEqual(lst[i], nd[i].tolist()) + for j in range(-shape[1], shape[1]): + self.assertEqual(lst[i][j], nd[i][j].tolist()) + for k in range(-shape[2], shape[2]): + self.assertEqual(lst[i][j][k], nd[i][j][k]) + + # Fortran array + nd = ndarray(items, shape=shape, fmt=fmt, + flags=flags|ND_FORTRAN) + lst = farray(items, shape) + + for i in range(-shape[0], shape[0]): + self.assertEqual(lst[i], nd[i].tolist()) + for j in range(-shape[1], shape[1]): + self.assertEqual(lst[i][j], nd[i][j].tolist()) + for k in range(shape[2], shape[2]): + self.assertEqual(lst[i][j][k], nd[i][j][k]) + + def test_ndarray_sequence(self): + nd = ndarray(1, shape=()) + self.assertRaises(TypeError, eval, "1 in nd", locals()) + + for fmt, items, _ in iter_format(5): + nd = ndarray(items, shape=[5], fmt=fmt) + for i, v in enumerate(nd): + self.assertEqual(v, items[i]) + self.assertTrue(v in nd) + + def test_ndarray_slice_invalid(self): + items = [1,2,3,4,5,6,7,8] + + # rvalue is not an exporter + xl = ndarray(items, shape=[8], flags=ND_WRITABLE) + self.assertRaises(TypeError, xl.__setitem__, slice(0,8,1), items) + + # rvalue is not a full exporter + xl = ndarray(items, shape=[8], flags=ND_WRITABLE) + ex = ndarray(items, shape=[8], flags=ND_WRITABLE) + xr = ndarray(ex, getbuf=PyBUF_ND) + self.assertRaises(BufferError, xl.__setitem__, slice(0,8,1), xr) + + # zero step + x = ndarray(items, shape=[8], fmt="L", flags=ND_WRITABLE) + self.assertRaises(ValueError, x.__getitem__, slice(0,1,0)) + + x = ndarray(items, shape=[2,4], fmt="L", flags=ND_WRITABLE) + self.assertRaises(ValueError, x.__getitem__, + (slice(0,1,1), slice(0,1,0))) + self.assertRaises(ValueError, x.__getitem__, + (slice(0,1,0), slice(0,1,1))) + self.assertRaises(TypeError, x.__getitem__, "@%$") + self.assertRaises(TypeError, x.__getitem__, ("@%$", slice(0,1,1))) + self.assertRaises(TypeError, x.__getitem__, (slice(0,1,1), {})) + + # differing format + x = ndarray(items, shape=[8], fmt="B", flags=ND_WRITABLE) + y = ndarray(items, shape=[8], fmt="b") + self.assertRaises(ValueError, x.__setitem__, slice(0,1,1), y[7:8]) + self.assertEqual(x.tolist(), items) + + # differing itemsize + x = ndarray(items, shape=[8], fmt="B", flags=ND_WRITABLE) + y = ndarray(items, shape=[8], fmt="L") + self.assertRaises(ValueError, x.__setitem__, slice(0,1,1), y[7:8]) + self.assertEqual(x.tolist(), items) + + # differing ndim + x = ndarray(items, shape=[2, 4], fmt="b", flags=ND_WRITABLE) + y = ndarray(items, shape=[8], fmt="b") + self.assertRaises(ValueError, x.__setitem__, slice(0,1,1), y[7:8]) + self.assertEqual(x.tolist(), [[1,2,3,4], [5,6,7,8]]) + + # differing shape + x = ndarray(items, shape=[8], fmt="b", flags=ND_WRITABLE) + y = ndarray(items, shape=[8], fmt="b") + self.assertRaises(ValueError, x.__setitem__, slice(0,2,1), y[7:8]) + self.assertEqual(x.tolist(), items) + + # _testbuffer module function + self.assertRaises(TypeError, slice_indices, slice(0,1,2), {}) + self.assertRaises(ValueError, slice_indices, slice(0,1,0), 4) + + # add_suboffsets() + x = ndarray(items, shape=[8], fmt="b", flags=ND_PIL) + self.assertRaises(TypeError, x.add_suboffsets) + + ex = ndarray(items, shape=[8], fmt="B") + x = ndarray(ex, getbuf=PyBUF_SIMPLE) + self.assertRaises(TypeError, x.add_suboffsets) + + def test_ndarray_slice_zero_shape(self): + items = [1,2,3,4,5,6,7,8,9,10,11,12] + + x = ndarray(items, shape=[3, 4], fmt="Q", flags=ND_WRITABLE) + y = ndarray(items, shape=[4, 3], fmt="Q") + + x[1:2, 2:2] = y[1:2, 3:3] + self.assertEqual(x.tolist(), carray(items, [3, 4])) + + def test_ndarray_slice_multidim(self): + shape_t = (2, 3, 5) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt = ''.join(sample(FMTCHRS, randrange(1, len(FMTCHRS)))) + items = randitems(fmt, nitems) + itemsize = struct.calcsize(fmt) + + for flags in (0, ND_PIL): + nd = ndarray(items, shape=shape, fmt=fmt, flags=flags) + lst = carray(items, shape) + + for slices in rslices_ndim(ndim, shape): + + listerr = None + try: + sliced = multislice(lst, slices) + except Exception as e: + listerr = e.__class__ + + nderr = None + try: + ndsliced = nd[slices] + except Exception as e: + nderr = e.__class__ + + if nderr or listerr: + self.assertIs(nderr, listerr) + else: + self.assertEqual(ndsliced.tolist(), sliced) + + def test_ndarray_slice_redundant_suboffsets(self): + shape_t = (2, 3, 5) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt = ''.join(sample(FMTCHRS, randrange(1, len(FMTCHRS)))) + items = randitems(fmt, nitems) + itemsize = struct.calcsize(fmt) + + nd = ndarray(items, shape=shape, fmt=fmt) + nd.add_suboffsets() + lst = carray(items, shape) + + for slices in rslices_ndim(ndim, shape): + + listerr = None + try: + sliced = multislice(lst, slices) + except Exception as e: + listerr = e.__class__ + + nderr = None + try: + ndsliced = nd[slices] + except Exception as e: + nderr = e.__class__ + + if nderr or listerr: + self.assertIs(nderr, listerr) + else: + self.assertEqual(ndsliced.tolist(), sliced) + + def test_ndarray_slice_assign_single(self): + for fmt, items, single_item in iter_format(5): + for lslice in genslices(5): + for rslice in genslices(5): + + nd = ndarray(items, shape=[5], fmt=fmt, flags=ND_WRITABLE) + + lsterr = None + diff_structure = None + lst = items[:] + try: + lval = lst[lslice] + rval = lst[rslice] + lst[lslice] = lst[rslice] + diff_structure = len(lval) != len(rval) + except Exception as e: + lsterr = e.__class__ + + nderr = None + try: + nd[lslice] = nd[rslice] + except Exception as e: + nderr = e.__class__ + + if diff_structure: # ndarray cannot change shape + self.assertIs(nderr, ValueError) + else: + self.assertEqual(nd.tolist(), lst) + self.assertIs(nderr, lsterr) + + def test_ndarray_slice_assign_multidim(self): + shape_t = (2, 3, 5) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt = ''.join(sample(FMTCHRS, randrange(1, len(FMTCHRS)))) + items = randitems(fmt, nitems) + itemsize = struct.calcsize(fmt) + + for flags in (0, ND_PIL): + for _ in range(ITERATIONS): + lslices, rslices = randslice_from_shape(ndim, shape) + + nd = ndarray(items, shape=shape, fmt=fmt, + flags=flags|ND_WRITABLE) + lst = carray(items, shape) + + listerr = None + try: + result = multislice_assign(lst, lst, lslices, rslices) + except Exception as e: + listerr = e.__class__ + + nderr = None + try: + nd[lslices] = nd[rslices] + except Exception as e: + nderr = e.__class__ + + if nderr or listerr: + self.assertIs(nderr, listerr) + else: + self.assertEqual(nd.tolist(), result) + + def test_ndarray_random(self): + # construction of valid arrays + for _ in range(ITERATIONS): + for fmt in FMTCHRS: + itemsize = struct.calcsize(fmt) + + t = rand_structure(itemsize, True, maxdim=MAXDIM, + maxshape=MAXSHAPE) + self.assertTrue(verify_structure(*t)) + items = randitems_from_structure(fmt, t) + + x = ndarray_from_structure(items, fmt, t) + xlist = x.tolist() + if t[2] > 0: # ndim + y = ndarray_from_structure(items, fmt, t, flags=ND_PIL) + ylist = y.tolist() + self.assertEqual(xlist, ylist) + + if numpy_array: + shape = t[3] + if 0 in shape: + continue # http://projects.scipy.org/numpy/ticket/1910 + z = numpy_array_from_structure(items, fmt, t) + self.assertEqual(xlist, z.tolist()) + + def test_ndarray_random_invalid(self): + # exceptions during construction of invalid arrays + for _ in range(ITERATIONS): + for fmt in FMTCHRS: + itemsize = struct.calcsize(fmt) + + t = rand_structure(itemsize, False, maxdim=MAXDIM, + maxshape=MAXSHAPE) + self.assertFalse(verify_structure(*t)) + items = randitems_from_structure(fmt, t) + + nderr = False + try: + x = ndarray_from_structure(items, fmt, t) + except Exception as e: + nderr = e.__class__ + self.assertTrue(nderr) + + if numpy_array: + numpy_err = False + try: + y = numpy_array_from_structure(items, fmt, t) + except Exception as e: + numpy_err = e.__class__ + + if 0: # http://projects.scipy.org/numpy/ticket/1910 + self.assertTrue(numpy_err) + + def test_ndarray_random_slice_assign(self): + # valid slice assignments + for _ in range(ITERATIONS): + for fmt in FMTCHRS: + itemsize = struct.calcsize(fmt) + + lshape, rshape, lslices, rslices = \ + rand_aligned_slices(maxdim=MAXDIM, maxshape=MAXSHAPE) + tl = rand_structure(itemsize, True, shape=lshape) + tr = rand_structure(itemsize, True, shape=rshape) + self.assertTrue(verify_structure(*tl)) + self.assertTrue(verify_structure(*tr)) + litems = randitems_from_structure(fmt, tl) + ritems = randitems_from_structure(fmt, tr) + + xl = ndarray_from_structure(litems, fmt, tl) + xr = ndarray_from_structure(ritems, fmt, tr) + xl[lslices] = xr[rslices] + xllist = xl.tolist() + xrlist = xr.tolist() + + if tl[2] > 0 and tr[2] > 0: # ndim + yl = ndarray_from_structure(litems, fmt, tl) + yr = ndarray_from_structure(ritems, fmt, tr) + yl[lslices] = yr[rslices] + yllist = yl.tolist() + yrlist = yr.tolist() + self.assertEqual(xllist, yllist) + self.assertEqual(xrlist, yrlist) + + if numpy_array: + if 0 in lshape or 0 in rshape: + continue # http://projects.scipy.org/numpy/ticket/1910 + + zl = numpy_array_from_structure(litems, fmt, tl) + zr = numpy_array_from_structure(ritems, fmt, tr) + zl[lslices] = zr[rslices] + + if not is_overlapping(tl) and not is_overlapping(tr): + # Slice assignment of overlapping structures + # is undefined in NumPy. + self.assertEqual(xllist, zl.tolist()) + + self.assertEqual(xrlist, zr.tolist()) + + def verify_getbuf(self, ex, req): + def simple_fmt(ex): + return ex.fmt == '' or ex.fmt == 'B' + def simple_buf(ex): + return ex.ndim == 1 and ex.c_contiguous and simple_fmt(ex) + def match(req, flag): + return ((req&flag) == flag) + + if (# cannot match writable request + (ex.readonly and match(req, PyBUF_WRITABLE)) or + # cannot match empty fmt request + (not simple_fmt(ex) and not match(req, PyBUF_FORMAT)) or + # cannot match simple unsigned bytes buffer request + ((req==PyBUF_SIMPLE or req==PyBUF_WRITABLE) and not simple_buf(ex)) or + # cannot match explicit contiguity request + (match(req, PyBUF_C_CONTIGUOUS) and not ex.c_contiguous) or + (match(req, PyBUF_F_CONTIGUOUS) and not ex.f_contiguous) or + (match(req, PyBUF_ANY_CONTIGUOUS) and not ex.contiguous) or + # buffer needs suboffsets + (not match(req, PyBUF_INDIRECT) and (ex.flags&ND_PIL)) or + # buffer without strides must be C-contiguous + (not match(req, PyBUF_STRIDES) and not ex.c_contiguous) or + # buffer without shape must be simple + (not match(req, PyBUF_ND) and not simple_buf(ex))): + + self.assertRaises(BufferError, ndarray, ex, getbuf=req) + return + + # The consumer may have requested default values. + ro = 0 if match(req, PyBUF_WRITABLE) else ex.readonly + ft = ex.fmt if match (req, PyBUF_FORMAT) else '' + sh = ex.shape if match(req, PyBUF_ND) else () + st = ex.strides if match(req, PyBUF_STRIDES) else () + + nd = ndarray(ex, getbuf=req) + self.verify(nd, flags=ex.flags, + obj=ex, itemsize=ex.itemsize, fmt=ft, readonly=ro, + ndim=ex.ndim, shape=sh, strides=st, + lst=ex.tolist()) + + def test_ndarray_getbuf(self): + requests = ( + # distinct flags + PyBUF_INDIRECT, PyBUF_STRIDES, PyBUF_ND, PyBUF_SIMPLE, + PyBUF_C_CONTIGUOUS, PyBUF_F_CONTIGUOUS, PyBUF_ANY_CONTIGUOUS, + # compound requests + PyBUF_FULL, PyBUF_FULL_RO, + PyBUF_RECORDS, PyBUF_RECORDS_RO, + PyBUF_STRIDED, PyBUF_STRIDED_RO, + PyBUF_CONTIG, PyBUF_CONTIG_RO, + ) + # items and format + items_fmt = ( + ([True if x % 2 else False for x in range(12)], '?'), + ([1,2,3,4,5,6,7,8,9,10,11,12], 'b'), + ([1,2,3,4,5,6,7,8,9,10,11,12], 'B'), + ([(2**31-x) if x % 2 else (-2**31+x) for x in range(12)], 'l') + ) + # shape, strides, offset + structure = ( + ([], [], 0), + ([12], [], 0), + ([12], [-1], 11), + ([6], [2], 0), + ([6], [-2], 11), + ([3, 4], [], 0), + ([3, 4], [-4, -1], 11), + ([2, 2], [4, 1], 4), + ([2, 2], [-4, -1], 8) + ) + # ndarray creation flags + ndflags = ( + 0, ND_WRITABLE, ND_FORTRAN, ND_FORTRAN|ND_WRITABLE, + ND_PIL, ND_PIL|ND_WRITABLE + ) + # flags that can actually be used as flags + real_flags = (0, PyBUF_WRITABLE, PyBUF_FORMAT, + PyBUF_WRITABLE|PyBUF_FORMAT) + + for items, fmt in items_fmt: + itemsize = struct.calcsize(fmt) + for shape, strides, offset in structure: + strides = [v * itemsize for v in strides] + offset *= itemsize + for flags in ndflags: + + if strides and (flags&ND_FORTRAN): + continue + if not shape and (flags&ND_PIL): + continue + + lst = items if shape else items[0] + ex = ndarray(lst, fmt=fmt, flags=flags, + shape=shape, strides=strides, offset=offset) + + for req in requests: + for bits in real_flags: + self.verify_getbuf(ex, req|bits) + + items = [1,2,3,4,5,6,7,8,9,10,11,12] + + # ND_GETBUF_FAIL + ex = ndarray(items, shape=[12], flags=ND_GETBUF_FAIL) + self.assertRaises(BufferError, ndarray, ex) + + # Request complex structure from a simple exporter. In this + # particular case the test object is not PEP-3118 compliant. + base = ndarray([9], [1]) + ex = ndarray(base, getbuf=PyBUF_SIMPLE) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_WRITABLE) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_ND) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_STRIDES) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_C_CONTIGUOUS) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_F_CONTIGUOUS) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_ANY_CONTIGUOUS) + nd = ndarray(ex, getbuf=PyBUF_SIMPLE) + + def test_ndarray_re_export(self): + items = [1,2,3,4,5,6,7,8,9,10,11,12] + + nd = ndarray(items, shape=[3,4], flags=ND_PIL) + ex = ndarray(nd) + + self.assertTrue(ex.flags & ND_PIL) + self.assertIs(ex.obj, nd) + self.assertEqual(ex.suboffsets, (0, -1)) + self.assertFalse(ex.c_contiguous) + self.assertFalse(ex.f_contiguous) + self.assertFalse(ex.contiguous) + + def test_ndarray_zero_shape(self): + # zeros in shape + nd = ndarray([1,2,3], shape=[0,3,3]) + self.assertEqual(nd.tolist(), []) + + nd = ndarray([1,2,3], shape=[3,0,3]) + self.assertEqual(nd.tolist(), [[], [], []]) + + nd = ndarray([1,2,3], shape=[3,3,0]) + self.assertEqual(nd.tolist(), + [[[], [], []], [[], [], []], [[], [], []]]) + + def test_ndarray_zero_strides(self): + # zero strides + nd = ndarray([1], shape=[5], strides=[0]) + self.assertEqual(nd.tolist(), [1, 1, 1, 1, 1]) + + def test_issue_7385(self): + x = ndarray([1,2,3], shape=[3], flags=ND_GETBUF_FAIL) + self.assertRaises(BufferError, memoryview, x) + + def test_memoryview_release(self): + + # Create re-exporter from getbuffer(memoryview), then release the view. + a = bytearray([1,2,3]) + m = memoryview(a) + nd = ndarray(m) # re-exporter + self.assertRaises(BufferError, m.release) + del nd + m.release() + + # chained views + a = bytearray([1,2,3]) + m1 = memoryview(a) + m2 = memoryview(m1) + nd = ndarray(m2) # re-exporter + m1.release() + self.assertRaises(BufferError, m2.release) + del nd + m2.release() + + # Allow changing layout while buffers are exported. + nd = ndarray([1,2,3], shape=[3], flags=ND_VAREXPORT) + m1 = memoryview(nd) + + nd.push([4,5,6,7,8], shape=[5]) # mutate nd + m2 = memoryview(nd) + + x = memoryview(m1) + self.assertEqual(x.tolist(), m1.tolist()) + + y = memoryview(m2) + self.assertEqual(y.tolist(), m2.tolist()) + self.assertEqual(y.tolist(), nd.tolist()) + m2.release() + y.release() + + nd.pop() # pop the current view + self.assertEqual(x.tolist(), nd.tolist()) + + del nd + m1.release() + x.release() + + # If multiple memoryviews share the same ManagedBuffer, implicit + # release() in the context manager's __exit__() method should still + # work. + def catch22(b): + with memoryview(b) as m2: + pass + + x = bytearray(b'123') + with memoryview(x) as m1: + catch22(m1) + self.assertEqual(m1[0], b'1') + + # XXX The same with a re-exporter of m1 will not work. + # x = bytearray(b'123') + # with memoryview(x) as m1: + # ex = ndarray(m1) + # self.assertEqual(m1[0], b'1') + + +def test_main(): + support.run_unittest(TestBufferProtocol) + + +if __name__ == "__main__": + test_main() diff -r 69b354a8c90f -r 78fb1181787a Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py Tue Aug 09 18:09:21 2011 +0200 +++ b/Lib/test/test_memoryview.py Tue Aug 09 20:28:59 2011 +0200 @@ -209,12 +209,16 @@ # If tp is a factory rather than a plain type, skip continue + class MyView(): + def __init__(self, base): + self.m = memoryview(base) class MySource(tp): pass class MyObject: pass - # Create a reference cycle through a memoryview object + # Create a reference cycle through a memoryview object. + # This exercises mbuf_clear(). b = MySource(tp(b'abc')) m = self._view(b) o = MyObject() @@ -226,6 +230,17 @@ gc.collect() self.assertTrue(wr() is None, wr()) + # This exercises memory_clear(). + m = MyView(tp(b'abc')) + o = MyObject() + m.x = m + m.o = o + wr = weakref.ref(o) + m = o = None + # The cycle must be broken + gc.collect() + self.assertTrue(wr() is None, wr()) + def _check_released(self, m, tp): check = self.assertRaisesRegex(ValueError, "released") with check: bytes(m) @@ -283,6 +298,10 @@ i = io.BytesIO(b'ZZZZ') self.assertRaises(TypeError, i.readinto, m) + def test_getbuf_fail(self): + self.assertRaises(TypeError, self._view, {}) + + # Variations on source objects for the buffer: bytes-like objects, then arrays # with itemsize > 1. # NOTE: support for multi-dimensional objects is unimplemented. diff -r 69b354a8c90f -r 78fb1181787a Lib/test/test_sys.py --- a/Lib/test/test_sys.py Tue Aug 09 18:09:21 2011 +0200 +++ b/Lib/test/test_sys.py Tue Aug 09 20:28:59 2011 +0200 @@ -770,7 +770,7 @@ check(int(PyLong_BASE**2-1), size(vh) + 2*self.longdigit) check(int(PyLong_BASE**2), size(vh) + 3*self.longdigit) # memory - check(memoryview(b''), size(h + 'PP2P2i7P')) + check(memoryview(b''), size(h + 'P3P3P3PiP4P2i5P')) # module check(unittest, size(h + '3P')) # None diff -r 69b354a8c90f -r 78fb1181787a Modules/_testbuffer.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/Modules/_testbuffer.c Tue Aug 09 20:28:59 2011 +0200 @@ -0,0 +1,2083 @@ +/* C Extension module to test all aspects of PEP-3118. + Written by Stefan Krah. */ + + +#define PY_SSIZE_T_CLEAN + +#include "Python.h" + + +/* struct module */ +PyObject *structmodule = NULL; +PyObject *Struct = NULL; +PyObject *calcsize = NULL; + +/* cache simple format string */ +static const char *simple_fmt = "B"; +PyObject *simple_format = NULL; +#define SIMPLE_FORMAT(fmt) (fmt == NULL || strcmp(fmt, "B") == 0) + + +/**************************************************************************/ +/* NDArray Object */ +/**************************************************************************/ + +#define CHECK_LIST_OR_TUPLE(v) \ + if (!PyList_Check(v) && !PyTuple_Check(v)) { \ + PyErr_SetString(PyExc_TypeError, \ + #v " must be a list or a tuple"); \ + return NULL; \ + } \ + +#define PyMem_XFree(v) \ + do { if (v) PyMem_Free(v); } while (0) + +/* User configurable flags for the ndarray */ +#define ND_VAREXPORT 0x001 /* change layout while buffers are exported */ + +/* User configurable flags for each base buffer */ +#define ND_WRITABLE 0x002 /* mark base buffer as writable */ +#define ND_FORTRAN 0x004 /* Fortran contiguous layout */ +#define ND_SCALAR 0x008 /* scalar: ndim = 0 */ +#define ND_PIL 0x010 /* convert to PIL-style array (suboffsets) */ +#define ND_GETBUF_FAIL 0x020 /* test issue 7385 */ + +/* Default: NumPy style (strides), read-only, no var-export, C-style layout */ +#define ND_DEFAULT 0x0 + +/* Internal flags for the base buffer */ +#define ND_C 0x040 /* C contiguous layout (default) */ +#define ND_OWN_ARRAYS 0x080 /* consumer owns arrays */ +#define ND_UNUSED 0x100 /* initializer */ + +/* ndarray properties */ +#define ND_IS_CONSUMER(nd) \ + (((NDArrayObject *)nd)->head == &((NDArrayObject *)nd)->staticbuf) + +/* ndbuf->flags properties */ +#define ND_C_CONTIGUOUS(flags) (!!(flags&(ND_SCALAR|ND_C))) +#define ND_FORTRAN_CONTIGUOUS(flags) (!!(flags&(ND_SCALAR|ND_FORTRAN))) +#define ND_ANY_CONTIGUOUS(flags) (!!(flags&(ND_SCALAR|ND_C|ND_FORTRAN))) + +/* getbuffer() requests */ +#define REQ_INDIRECT(flags) ((flags&PyBUF_INDIRECT) == PyBUF_INDIRECT) +#define REQ_C_CONTIGUOUS(flags) ((flags&PyBUF_C_CONTIGUOUS) == PyBUF_C_CONTIGUOUS) +#define REQ_F_CONTIGUOUS(flags) ((flags&PyBUF_F_CONTIGUOUS) == PyBUF_F_CONTIGUOUS) +#define REQ_ANY_CONTIGUOUS(flags) ((flags&PyBUF_ANY_CONTIGUOUS) == PyBUF_ANY_CONTIGUOUS) +#define REQ_STRIDES(flags) ((flags&PyBUF_STRIDES) == PyBUF_STRIDES) +#define REQ_SHAPE(flags) ((flags&PyBUF_ND) == PyBUF_ND) +#define REQ_WRITABLE(flags) (flags&PyBUF_WRITABLE) +#define REQ_FORMAT(flags) (flags&PyBUF_FORMAT) + + +/* Single node of a list of base buffers. The list is needed to implement + changes in memory layout while exported buffers are active. */ +static PyTypeObject NDArray_Type; + +struct ndbuf; +typedef struct ndbuf { + struct ndbuf *next; + struct ndbuf *prev; + Py_ssize_t len; /* length of data */ + Py_ssize_t offset; /* start of the array relative to data */ + char *data; /* raw data */ + int flags; /* capabilities of the base buffer */ + Py_ssize_t exports; /* number of exports */ + Py_buffer base; /* base buffer */ +} ndbuf_t; + +typedef struct { + PyObject_HEAD + int flags; /* ndarray flags */ + ndbuf_t staticbuf; /* static buffer for re-exporting mode */ + ndbuf_t *head; /* currently active base buffer */ +} NDArrayObject; + + +static ndbuf_t * +ndbuf_new(Py_ssize_t nitems, Py_ssize_t itemsize, Py_ssize_t offset, int flags) +{ + ndbuf_t *ndbuf; + Py_buffer *base; + Py_ssize_t len; + + len = nitems * itemsize; + if (offset % itemsize) { + PyErr_SetString(PyExc_ValueError, + "offset must be a multiple of itemsize"); + return NULL; + } + if (offset < 0 || offset+itemsize > len) { + PyErr_SetString(PyExc_ValueError, "offset out of bounds"); + return NULL; + } + + ndbuf = PyMem_Malloc(sizeof *ndbuf); + if (ndbuf == NULL) { + PyErr_NoMemory(); + return NULL; + } + + ndbuf->next = NULL; + ndbuf->prev = NULL; + ndbuf->len = len; + ndbuf->offset= offset; + + ndbuf->data = PyMem_Malloc(len); + if (ndbuf->data == NULL) { + PyErr_NoMemory(); + PyMem_Free(ndbuf); + return NULL; + } + + ndbuf->flags = flags; + ndbuf->exports = 0; + + base = &ndbuf->base; + base->obj = NULL; + base->buf = ndbuf->data; + base->len = len; + base->itemsize = 1; + base->readonly = 0; + base->format = NULL; + base->ndim = 1; + base->shape = NULL; + base->strides = NULL; + base->suboffsets = NULL; + base->internal = ndbuf; + + return ndbuf; +} + +static void +ndbuf_free(ndbuf_t *ndbuf) +{ + Py_buffer *base = &ndbuf->base; + + PyMem_XFree(ndbuf->data); + PyMem_XFree(base->format); + PyMem_XFree(base->shape); + PyMem_XFree(base->strides); + PyMem_XFree(base->suboffsets); + + PyMem_Free(ndbuf); +} + +static void +ndbuf_push(NDArrayObject *nd, ndbuf_t *elt) +{ + elt->next = nd->head; + if (nd->head) nd->head->prev = elt; + nd->head = elt; + elt->prev = NULL; +} + +static void +ndbuf_delete(NDArrayObject *nd, ndbuf_t *elt) +{ + if (elt->prev) + elt->prev->next = elt->next; + else + nd->head = elt->next; + + if (elt->next) + elt->next->prev = elt->prev; + + ndbuf_free(elt); +} + +static void +ndbuf_pop(NDArrayObject *nd) +{ + ndbuf_delete(nd, nd->head); +} + + +static PyObject * +ndarray_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +{ + NDArrayObject *nd; + + nd = PyObject_New(NDArrayObject, &NDArray_Type); + if (nd == NULL) + return NULL; + + nd->flags = 0; + nd->head = NULL; + return (PyObject *)nd; +} + +static void +ndarray_dealloc(NDArrayObject *self) +{ + if (self->head) { + if (ND_IS_CONSUMER(self)) { + Py_buffer *base = &self->head->base; + if (self->head->flags & ND_OWN_ARRAYS) { + PyMem_XFree(base->shape); + PyMem_XFree(base->strides); + PyMem_XFree(base->suboffsets); + } + PyBuffer_Release(base); + } + else { + while (self->head) + ndbuf_pop(self); + } + } + PyObject_Del(self); +} + +static int +ndarray_init_staticbuf(PyObject *exporter, NDArrayObject *nd, int flags) +{ + Py_buffer *base = &nd->staticbuf.base; + + if (PyObject_GetBuffer(exporter, base, flags) < 0) + return -1; + + nd->head = &nd->staticbuf; + + nd->head->next = NULL; + nd->head->prev = NULL; + nd->head->len = -1; + nd->head->offset = -1; + nd->head->data = NULL; + + nd->head->flags = base->readonly ? 0 : ND_WRITABLE; + nd->head->exports = 0; + + return 0; +} + +static void +ndbuf_init_flags(ndbuf_t *ndbuf) +{ + if (ndbuf->base.ndim == 0) + ndbuf->flags |= ND_SCALAR; + if (ndbuf->base.suboffsets) + ndbuf->flags |= ND_PIL; + if (PyBuffer_IsContiguous(&ndbuf->base, 'C')) + ndbuf->flags |= ND_C; + if (PyBuffer_IsContiguous(&ndbuf->base, 'F')) + ndbuf->flags |= ND_FORTRAN; +} + + +/****************************************************************************/ +/* Buffer/List conversions */ +/****************************************************************************/ + +static Py_ssize_t *strides_from_shape(const ndbuf_t *); + +/* Pack all items into the buffer of 'obj'. The 'format' parameter must be + in struct module syntax. For standard C types, a single item is an integer. + For compound types, a single item is a tuple of integers. */ +static int +pack_from_list(PyObject *obj, PyObject *items, PyObject *format, + Py_ssize_t itemsize) +{ + PyObject *structobj, *pack_into; + PyObject *args, *offset; + PyObject *item, *tmp; + Py_ssize_t nitems, fmtsize; + Py_ssize_t i, j; + int ret = 0; + + assert(PyObject_CheckBuffer(obj)); + assert(PyList_Check(items) || PyTuple_Check(items)); + + nitems = PySequence_Fast_GET_SIZE(items); + fmtsize = PyUnicode_GetSize(format); + if (fmtsize < 0) + return -1; + + structobj = PyObject_CallFunctionObjArgs(Struct, format, NULL); + if (structobj == NULL) + return -1; + + pack_into = PyObject_GetAttrString(structobj, "pack_into"); + if (pack_into == NULL) { + Py_DECREF(structobj); + return -1; + } + + /* fmtsize >= 1 */ + args = PyTuple_New(2 + fmtsize); + if (args == NULL) { + Py_DECREF(pack_into); + Py_DECREF(structobj); + return -1; + } + + offset = NULL; + for (i = 0; i < nitems; i++) { + /* Loop invariant: args[j] are borrowed references or NULL. */ + PyTuple_SET_ITEM(args, 0, obj); + for (j = 1; j < 2+fmtsize; j++) + PyTuple_SET_ITEM(args, j, NULL); + + Py_XDECREF(offset); + offset = PyLong_FromSsize_t(i*itemsize); + if (offset == NULL) { + ret = -1; + break; + } + PyTuple_SET_ITEM(args, 1, offset); + + item = PySequence_Fast_GET_ITEM(items, i); + if ((PyBytes_Check(item) || PyLong_Check(item) || + PyFloat_Check(item)) && fmtsize == 1) { + PyTuple_SET_ITEM(args, 2, item); + } + else if ((PyList_Check(item) || PyTuple_Check(item)) && + PySequence_Length(item) == fmtsize) { + for (j = 0; j < fmtsize; j++) { + tmp = PySequence_Fast_GET_ITEM(item, j); + PyTuple_SET_ITEM(args, 2+j, tmp); + } + } + else { + PyErr_SetString(PyExc_ValueError, + "mismatch between initializer element and format string"); + ret = -1; + break; + } + + tmp = PyObject_CallObject(pack_into, args); + if (tmp == NULL) { + ret = -1; + break; + } + Py_DECREF(tmp); + } + + Py_INCREF(obj); /* args[0] */ + /* args[1]: offset is either NULL or should be dealloc'd */ + for (i = 2; i < 2+fmtsize; i++) { + tmp = PyTuple_GET_ITEM(args, i); + Py_XINCREF(tmp); + } + Py_DECREF(args); + + Py_DECREF(pack_into); + Py_DECREF(structobj); + return ret; + +} + +/* Pack single element */ +static int +pack_single(char *ptr, PyObject *item, const char *fmt, Py_ssize_t itemsize) +{ + PyObject *pack_into = NULL, *args = NULL; + PyObject *format = NULL, *mview = NULL, *zero = NULL; + Py_ssize_t i, fmtsize; + int ret = -1; + PyObject *x; + + if (fmt == NULL) fmt = "B"; + fmtsize = strlen(fmt); + assert(fmtsize >= 1); + + format = PyUnicode_FromString(fmt); + if (format == NULL) + goto out; + + mview = PyMemoryView_FromBytes(ptr, itemsize, PyBUF_WRITE); + if (mview == NULL) + goto out; + + zero = PyLong_FromLong(0); + if (zero == NULL) + goto out; + + pack_into = PyObject_GetAttrString(structmodule, "pack_into"); + if (pack_into == NULL) + goto out; + + args = PyTuple_New(3+fmtsize); + if (args == NULL) + goto out; + + PyTuple_SET_ITEM(args, 0, format); + PyTuple_SET_ITEM(args, 1, mview); + PyTuple_SET_ITEM(args, 2, zero); + + if ((PyBytes_Check(item) || PyLong_Check(item) || + PyFloat_Check(item)) && fmtsize == 1) { + PyTuple_SET_ITEM(args, 3, item); + } + else if ((PyList_Check(item) || PyTuple_Check(item)) && + PySequence_Length(item) == fmtsize) { + for (i = 0; i < fmtsize; i++) { + x = PySequence_Fast_GET_ITEM(item, i); + PyTuple_SET_ITEM(args, 3+i, x); + } + } + else { + PyErr_SetString(PyExc_ValueError, + "mismatch between initializer element and format string"); + goto args_out; + } + + x = PyObject_CallObject(pack_into, args); + if (x != NULL) { + Py_DECREF(x); + ret = 0; + } + + +args_out: + for (i = 0; i < 3+fmtsize; i++) + Py_XINCREF(PyTuple_GET_ITEM(args, i)); + Py_XDECREF(args); +out: + Py_XDECREF(pack_into); + Py_XDECREF(zero); + Py_XDECREF(mview); + Py_XDECREF(format); + return ret; +} + +static void +copy_rec(const Py_ssize_t *shape, Py_ssize_t ndim, Py_ssize_t itemsize, + char *dptr, const Py_ssize_t *dstrides, const Py_ssize_t *dsuboffsets, + char *sptr, const Py_ssize_t *sstrides, const Py_ssize_t *ssuboffsets, + char *mem) +{ + Py_ssize_t i; + + assert(ndim >= 1); + + if (ndim == 1) { + if (dstrides[0] == itemsize && sstrides[0] == itemsize) + memmove(dptr, sptr, shape[0] * itemsize); + else { + char *p; + assert(mem != NULL); + for (i=0, p=mem; i < shape[0]; p+=itemsize, sptr+=sstrides[0], i++) + memcpy(p, sptr, itemsize); + for (i=0, p=mem; i < shape[0]; p+=itemsize, dptr+=dstrides[0], i++) + memcpy(dptr, p, itemsize); + } + return; + } + + for (i = 0; i < shape[0]; dptr+=dstrides[0], sptr+=sstrides[0], i++) { + char *xdptr = dptr; + char *xsptr = sptr; + + if (dsuboffsets && dsuboffsets[0] >= 0) + xdptr = *((char**)dptr) + dsuboffsets[0]; + if (ssuboffsets && ssuboffsets[0] >= 0) + xsptr = *((char**)sptr) + ssuboffsets[0]; + + copy_rec(shape+1, ndim-1, itemsize, + xdptr, dstrides+1, dsuboffsets ? dsuboffsets+1 : NULL, + xsptr, sstrides+1, ssuboffsets ? ssuboffsets+1 : NULL, + mem); + } +} + +static int +cmp_structure(Py_buffer *dest, Py_buffer *src) +{ + Py_ssize_t i; + + if (strcmp(dest->format, src->format) != 0 || + dest->itemsize != src->itemsize || + dest->ndim != src->ndim) + return -1; + + for (i = 0; i < dest->ndim; i++) { + if (dest->shape[i] != src->shape[i]) + return -1; + if (dest->shape[i] == 0) + break; + } + + return 0; +} + +/* Copy src to dest. Both buffers must have the same format, itemsize, + ndim and shape. Copying is atomic, the function never fails with + a partial copy. */ +static int +copy_buffer(Py_buffer *dest, Py_buffer *src) +{ + char *mem = NULL; + + assert(dest->ndim > 0); + + if (cmp_structure(dest, src) < 0) { + PyErr_SetString(PyExc_ValueError, + "ndarray assignment: lvalue and rvalue have different structures"); + return -1; + } + + if (dest->strides[dest->ndim-1] != dest->itemsize || + src->strides[src->ndim-1] != src->itemsize) { + mem = PyMem_Malloc(dest->shape[dest->ndim-1] * dest->itemsize); + if (mem == NULL) { + PyErr_NoMemory(); + return -1; + } + } + + copy_rec(dest->shape, dest->ndim, dest->itemsize, + dest->buf, dest->strides, dest->suboffsets, + src->buf, src->strides, src->suboffsets, + mem); + + PyMem_XFree(mem); + return 0; +} + + +/* Unpack single element */ +static PyObject * +unpack_single(char *ptr, const char *fmt, Py_ssize_t itemsize) +{ + PyObject *x, *unpack_from, *mview; + + unpack_from = PyObject_GetAttrString(structmodule, "unpack_from"); + if (unpack_from == NULL) + return NULL; + + mview = PyMemoryView_FromBytes(ptr, itemsize, PyBUF_READ); + if (mview == NULL) { + Py_DECREF(unpack_from); + return NULL; + } + + x = PyObject_CallFunction(unpack_from, "sO", fmt, mview); + Py_DECREF(unpack_from); + Py_DECREF(mview); + if (x == NULL) + return NULL; + + if (PyTuple_GET_SIZE(x) == 1) { + PyObject *tmp = PyTuple_GET_ITEM(x, 0); + Py_INCREF(tmp); + Py_DECREF(x); + return tmp; + } + + return x; +} + +/* Unpack a multi-dimensional matrix into a nested list. Return a scalar + for ndim = 0. */ +static PyObject * +unpack_rec(PyObject *unpack_from, char *ptr, PyObject *mview, char *item, + const Py_ssize_t *shape, const Py_ssize_t *strides, + const Py_ssize_t *suboffsets, Py_ssize_t ndim, Py_ssize_t itemsize) +{ + PyObject *lst, *x; + Py_ssize_t i; + + assert(ndim >= 0); + assert(shape != NULL); + assert(strides != NULL); + + if (ndim == 0) { + memcpy(item, ptr, itemsize); + x = PyObject_CallFunctionObjArgs(unpack_from, mview, NULL); + if (x == NULL) + return NULL; + if (PyTuple_GET_SIZE(x) == 1) { + PyObject *tmp = PyTuple_GET_ITEM(x, 0); + Py_INCREF(tmp); + Py_DECREF(x); + return tmp; + } + return x; + } + + lst = PyList_New(shape[0]); + if (lst == NULL) + return NULL; + + for (i = 0; i < shape[0]; ptr+=strides[0], i++) { + char *nextptr = ptr; + if (suboffsets && suboffsets[0] >= 0) + nextptr = *((char**)ptr) + suboffsets[0]; + + x = unpack_rec(unpack_from, nextptr, mview, item, + shape+1, strides+1, suboffsets ? suboffsets+1 : NULL, + ndim-1, itemsize); + if (x == NULL) { + Py_DECREF(lst); + return NULL; + } + + PyList_SET_ITEM(lst, i, x); + } + + return lst; +} + + +static PyObject * +ndarray_as_list(NDArrayObject *nd) +{ + PyObject *structobj = NULL, *unpack_from = NULL; + PyObject *lst = NULL, *mview = NULL; + Py_buffer *base = &nd->head->base; + Py_ssize_t *shape = base->shape; + Py_ssize_t *strides = base->strides; + Py_ssize_t simple_shape[1]; + Py_ssize_t simple_strides[1]; + char *item = NULL; + PyObject *format; + char *fmt = base->format; + + base = &nd->head->base; + + if (fmt == NULL) { + assert(base->itemsize == 1); + fmt = (char *)simple_fmt; + } + if (shape == NULL) { + assert(ND_C_CONTIGUOUS(nd->head->flags)); + assert(base->strides == NULL); + assert(base->ndim <= 1); + shape = simple_shape; + shape[0] = base->len; + strides = simple_strides; + strides[0] = base->itemsize; + } + else if (strides == NULL) { + assert(ND_C_CONTIGUOUS(nd->head->flags)); + strides = strides_from_shape(nd->head); + if (strides == NULL) + return NULL; + } + + format = PyUnicode_FromString(fmt); + if (format == NULL) + goto out; + + structobj = PyObject_CallFunctionObjArgs(Struct, format, NULL); + Py_DECREF(format); + if (structobj == NULL) + goto out; + + unpack_from = PyObject_GetAttrString(structobj, "unpack_from"); + if (unpack_from == NULL) + goto out; + + item = PyMem_Malloc(base->itemsize); + if (item == NULL) { + PyErr_NoMemory(); + goto out; + } + + mview = PyMemoryView_FromBytes(item, base->itemsize, PyBUF_WRITE); + if (mview == NULL) + goto out; + + lst = unpack_rec(unpack_from, base->buf, mview, item, + shape, strides, base->suboffsets, + base->ndim, base->itemsize); + +out: + Py_XDECREF(mview); + PyMem_XFree(item); + Py_XDECREF(unpack_from); + Py_XDECREF(structobj); + if (strides != base->strides && strides != simple_strides) + PyMem_XFree(strides); + + return lst; +} + + +/****************************************************************************/ +/* Initialize ndbuf */ +/****************************************************************************/ + +/* + State of a new ndbuf during initialization. 'OK' means that initialization + is complete. 'PTR' means that a pointer has been initialized, but the + state of the memory is still undefined and ndbuf->offset is disregarded. + + +-----------------+-----------+-------------+----------------+ + | | ndbuf_new | init_simple | init_structure | + +-----------------+-----------+-------------+----------------+ + | next | OK (NULL) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | prev | OK (NULL) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | len | OK | OK | OK | + +-----------------+-----------+-------------+----------------+ + | offset | OK | OK | OK | + +-----------------+-----------+-------------+----------------+ + | data | PTR | OK | OK | + +-----------------+-----------+-------------+----------------+ + | flags | user | user | OK | + +-----------------+-----------+-------------+----------------+ + | exports | OK (0) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.obj | OK (NULL) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.buf | PTR | PTR | OK | + +-----------------+-----------+-------------+----------------+ + | base.len | len(data) | len(data) | OK | + +-----------------+-----------+-------------+----------------+ + | base.itemsize | 1 | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.readonly | 0 | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.format | NULL | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.ndim | 1 | 1 | OK | + +-----------------+-----------+-------------+----------------+ + | base.shape | NULL | NULL | OK | + +-----------------+-----------+-------------+----------------+ + | base.strides | NULL | NULL | OK | + +-----------------+-----------+-------------+----------------+ + | base.suboffsets | NULL | NULL | OK | + +-----------------+-----------+-------------+----------------+ + | base.internal | OK | OK | OK | + +-----------------+-----------+-------------+----------------+ + +*/ + +static Py_ssize_t +get_itemsize(PyObject *format) +{ + PyObject *tmp; + Py_ssize_t itemsize; + + tmp = PyObject_CallFunctionObjArgs(calcsize, format, NULL); + if (tmp == NULL) + return -1; + itemsize = PyLong_AsSsize_t(tmp); + Py_DECREF(tmp); + + return itemsize; +} + +static char * +get_format(PyObject *format) +{ + PyObject *tmp; + char *fmt; + + tmp = PyUnicode_AsASCIIString(format); + if (tmp == NULL) + return NULL; + fmt = PyMem_Malloc(PyBytes_GET_SIZE(tmp)+1); + if (fmt == NULL) { + PyErr_NoMemory(); + Py_DECREF(tmp); + return NULL; + } + strcpy(fmt, PyBytes_AS_STRING(tmp)); + Py_DECREF(tmp); + + return fmt; +} + +static int +init_simple(ndbuf_t *ndbuf, PyObject *items, PyObject *format, + Py_ssize_t itemsize) +{ + PyObject *mview; + Py_buffer *base = &ndbuf->base; + int ret; + + mview = PyMemoryView_FromBuffer(base); + if (mview == NULL) + return -1; + + ret = pack_from_list(mview, items, format, itemsize); + Py_DECREF(mview); + if (ret < 0) + return -1; + + base->readonly = !(ndbuf->flags & ND_WRITABLE); + base->itemsize = itemsize; + base->format = get_format(format); + if (base->format == NULL) + return -1; + + return 0; +} + +static Py_ssize_t * +seq_as_ssize_array(PyObject *seq, Py_ssize_t len, int is_shape) +{ + Py_ssize_t *dest; + Py_ssize_t x, i; + + dest = PyMem_Malloc(len * (sizeof *dest)); + if (dest == NULL) { + PyErr_NoMemory(); + return NULL; + } + + for (i = 0; i < len; i++) { + PyObject *tmp = PySequence_Fast_GET_ITEM(seq, i); + if (!PyLong_Check(tmp)) { + PyErr_Format(PyExc_ValueError, + "elements of %s must be integers", + is_shape ? "shape" : "strides"); + PyMem_Free(dest); + return NULL; + } + x = PyLong_AsSsize_t(tmp); + if (PyErr_Occurred()) { + PyMem_Free(dest); + return NULL; + } + if (is_shape && x < 0) { + PyErr_Format(PyExc_ValueError, + "elements of shape must be integers >= 0"); + PyMem_Free(dest); + return NULL; + } + dest[i] = x; + } + + return dest; +} + +static Py_ssize_t * +strides_from_shape(const ndbuf_t *ndbuf) +{ + const Py_buffer *base = &ndbuf->base; + Py_ssize_t *s, i; + char layout; + + s = PyMem_Malloc(base->ndim * (sizeof *s)); + if (s == NULL) { + PyErr_NoMemory(); + return NULL; + } + + if (ndbuf->flags & ND_FORTRAN) { + s[0] = base->itemsize; + for (i = 1; i < base->ndim; i++) + s[i] = s[i-1] * base->shape[i-1]; + layout = 'F'; + } + else { + s[base->ndim-1] = base->itemsize; + for (i = base->ndim-2; i >= 0; i--) + s[i] = s[i+1] * base->shape[i+1]; + layout = 'C'; + } + + return s; +} + +/* Bounds check: + + len := complete length of allocated memory + offset := start of the array + + A single array element is indexed by: + + i = indices[0] * strides[0] + indices[1] * strides[1] + ... + + imin is reached when all indices[n] combined with positive strides are 0 + and all indices combined with negative strides are shape[n]-1, which is + the maximum index for the nth dimension. + + imax is reached when all indices[n] combined with negative strides are 0 + and all indices combined with positive strides are shape[n]-1. +*/ +static int +verify_structure(Py_ssize_t len, Py_ssize_t itemsize, Py_ssize_t offset, + const Py_ssize_t *shape, const Py_ssize_t *strides, + Py_ssize_t ndim) +{ + Py_ssize_t imin, imax; + Py_ssize_t n; + + assert(ndim >= 0); + + if (ndim == 0 && (offset < 0 || offset+itemsize > len)) + goto invalid_combination; + + for (n = 0; n < ndim; n++) + if (strides[n] % itemsize) { + PyErr_SetString(PyExc_ValueError, + "strides must be a multiple of itemsize"); + return -1; + } + + for (n = 0; n < ndim; n++) + if (shape[n] == 0) + return 0; + + imin = imax = 0; + for (n = 0; n < ndim; n++) + if (strides[n] <= 0) + imin += (shape[n]-1) * strides[n]; + else + imax += (shape[n]-1) * strides[n]; + + if (imin + offset < 0 || imax + offset + itemsize > len) + goto invalid_combination; + + return 0; + + +invalid_combination: + PyErr_SetString(PyExc_ValueError, + "invalid combination of buffer, shape and strides"); + return -1; +} + +/* Convert a NumPy-style array to an array using suboffsets to stride in + the first dimension. Requirements: ndim > 0. + + Input: + + shape = {2, 2, 3}; + strides = {6, 3, 1}; + suboffsets = NULL; + buf = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}; + + Output: + + sizep = sizeof(char *) + + shape = {2, 2, 3}; + strides = {sizep, 3, 1}; + suboffsets = {0, -1, -1}; + buf = {p1, p2, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}; + | | ^ ^ + | | | | + `---'---' | + | | + `---------------------' + + So, in the example the input resembles the three-dimensional array + char v[2][2][3], while the output resembles an array of two pointers + to two-dimensional arrays: char (*v[2])[2][3]. +*/ +static int +init_suboffsets(ndbuf_t *ndbuf) +{ + Py_buffer *base = &ndbuf->base; + Py_ssize_t addsize; + Py_ssize_t i; + char *data; + + assert(base->ndim > 0); + assert(base->suboffsets == NULL); + + addsize = base->shape[0] * (sizeof (char *)); + + data = PyMem_Malloc(ndbuf->len + addsize); + if (data == NULL) { + PyErr_NoMemory(); + return -1; + } + + memcpy(data + addsize, ndbuf->data, ndbuf->len); + + PyMem_Free(ndbuf->data); + base->buf = ndbuf->data = data; + + for (i = 0; i < base->shape[0]; i++) + ((char **)base->buf)[i] = (char *)base->buf + + addsize + ndbuf->offset + + i * base->strides[0]; + + base->strides[0] = (sizeof (char *)); + + base->suboffsets = PyMem_Malloc(base->ndim * (sizeof *base->suboffsets)); + if (base->suboffsets == NULL) { + PyErr_NoMemory(); + return -1; + } + base->suboffsets[0] = 0; + for (i = 1; i < base->ndim; i++) + base->suboffsets[i] = -1; + + ndbuf->flags &= ~(ND_C|ND_FORTRAN); + return 0; +} + +static void +init_len(Py_buffer *base) +{ + Py_ssize_t i; + + base->len = 1; + for (i = 0; i < base->ndim; i++) + base->len *= base->shape[i]; + base->len *= base->itemsize; +} + +static int +init_structure(ndbuf_t *ndbuf, PyObject *shape, PyObject *strides, + Py_ssize_t ndim) +{ + Py_buffer *base = &ndbuf->base; + + base->ndim = (int)ndim; + if (ndim == 0) { + if (ndbuf->flags & ND_PIL) { + PyErr_SetString(PyExc_TypeError, + "ndim = 0 cannot be used in conjunction with ND_PIL"); + return -1; + } + ndbuf->flags |= ND_SCALAR; + return 0; + } + + /* shape */ + base->shape = seq_as_ssize_array(shape, ndim, 1); + if (base->shape == NULL) + return -1; + + /* strides */ + if (strides) { + base->strides = seq_as_ssize_array(strides, ndim, 0); + } + else { + base->strides = strides_from_shape(ndbuf); + } + if (base->strides == NULL) + return -1; + if (verify_structure(base->len, base->itemsize, ndbuf->offset, + base->shape, base->strides, ndim) < 0) + return -1; + + /* buf */ + base->buf = ndbuf->data + ndbuf->offset; + + /* len */ + init_len(base); + + /* ndbuf->flags */ + if (PyBuffer_IsContiguous(base, 'C')) + ndbuf->flags |= ND_C; + if (PyBuffer_IsContiguous(base, 'F')) + ndbuf->flags |= ND_FORTRAN; + + + /* convert numpy array to suboffset representation */ + if (ndbuf->flags & ND_PIL) { + /* modifies base->buf, base->strides and base->suboffsets **/ + return init_suboffsets(ndbuf); + } + + return 0; +} + +static ndbuf_t * +init_ndbuf(PyObject *items, PyObject *shape, PyObject *strides, + Py_ssize_t offset, PyObject *format, int flags) +{ + ndbuf_t *ndbuf; + Py_ssize_t ndim; + Py_ssize_t nitems; + Py_ssize_t itemsize; + + /* ndim = len(shape) */ + CHECK_LIST_OR_TUPLE(shape) + ndim = PySequence_Fast_GET_SIZE(shape); + if (ndim > INT_MAX) { + PyErr_SetString(PyExc_ValueError, "ndim must not exceed INT_MAX"); + return NULL; + } + + /* len(strides) = len(shape) */ + if (strides) { + CHECK_LIST_OR_TUPLE(strides) + if (PySequence_Fast_GET_SIZE(strides) == 0) + strides = NULL; + else if (flags & ND_FORTRAN) { + PyErr_SetString(PyExc_TypeError, + "ND_FORTRAN cannot be used together with strides"); + return NULL; + } + else if (PySequence_Fast_GET_SIZE(strides) != ndim) { + PyErr_SetString(PyExc_ValueError, + "len(shape) != len(strides)"); + return NULL; + } + } + + /* itemsize */ + itemsize = get_itemsize(format); + if (itemsize <= 0) { + if (itemsize == 0) { + PyErr_SetString(PyExc_ValueError, + "itemsize must not be zero"); + } + return NULL; + } + + /* convert scalar to list */ + if (ndim == 0) { + items = Py_BuildValue("(O)", items); + if (items == NULL) + return NULL; + } + else { + CHECK_LIST_OR_TUPLE(items) + Py_INCREF(items); + } + + /* number of items */ + nitems = PySequence_Fast_GET_SIZE(items); + if (nitems == 0) { + PyErr_SetString(PyExc_ValueError, + "initializer list or tuple must not be empty"); + Py_DECREF(items); + return NULL; + } + + ndbuf = ndbuf_new(nitems, itemsize, offset, flags); + if (ndbuf == NULL) { + Py_DECREF(items); + return NULL; + } + + + if (init_simple(ndbuf, items, format, itemsize) < 0) + goto error; + if (init_structure(ndbuf, shape, strides, ndim) < 0) + goto error; + + Py_DECREF(items); + return ndbuf; + +error: + Py_DECREF(items); + ndbuf_free(ndbuf); + return NULL; +} + +/* initialize and push a new base onto the linked list */ +static int +ndarray_push_base(NDArrayObject *nd, PyObject *items, + PyObject *shape, PyObject *strides, + Py_ssize_t offset, PyObject *format, int flags) +{ + ndbuf_t *ndbuf; + + ndbuf = init_ndbuf(items, shape, strides, offset, format, flags); + if (ndbuf == NULL) + return -1; + + ndbuf_push(nd, ndbuf); + return 0; +} + +#define PyBUF_UNUSED 0x10000 +static int +ndarray_init(PyObject *self, PyObject *args, PyObject *kwds) +{ + NDArrayObject *nd = (NDArrayObject *)self; + static char *kwlist[] = { + "obj", "shape", "strides", "offset", "fmt", "flags", "getbuf", NULL + }; + PyObject *v = NULL; /* initializer: scalar, list, tuple or base object */ + PyObject *shape = NULL; /* size of each dimension */ + PyObject *strides = NULL; /* number of bytes to the next elt in each dim */ + Py_ssize_t offset = 0; /* buffer offset */ + PyObject *format = simple_format; /* struct module specifier: "B" */ + int flags = ND_UNUSED; /* base buffer and ndarray flags */ + + int getbuf = PyBUF_UNUSED; /* re-exporter: getbuffer request flags */ + + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|OOnOii", kwlist, + &v, &shape, &strides, &offset, &format, &flags, &getbuf)) + return -1; + + /* NDArrayObject is re-exporter */ + if (PyObject_CheckBuffer(v) && shape == NULL) { + if (strides || offset || format != simple_format || + flags != ND_UNUSED) { + PyErr_SetString(PyExc_TypeError, + "construction from exporter object only takes a single " + "additional getbuf argument"); + return -1; + } + + getbuf = (getbuf == PyBUF_UNUSED) ? PyBUF_FULL_RO : getbuf; + + if (ndarray_init_staticbuf(v, nd, getbuf) < 0) + return -1; + + ndbuf_init_flags(nd->head); + + return 0; + } + + /* NDArrayObject is the original base object. */ + if (getbuf != PyBUF_UNUSED) { + PyErr_SetString(PyExc_TypeError, + "getbuf argument only valid for construction from exporter " + "object"); + return -1; + } + if (shape == NULL) { + PyErr_SetString(PyExc_TypeError, + "shape is a required argument when constructing from " + "list, tuple or scalar"); + return -1; + } + + if (flags == ND_UNUSED) + flags = ND_DEFAULT; + if (flags & ND_VAREXPORT) { + nd->flags |= ND_VAREXPORT; + flags &= ~ND_VAREXPORT; + } + + /* Initialize and push the first base buffer onto the linked list. */ + return ndarray_push_base(nd, v, shape, strides, offset, format, flags); +} + +/* Push an additional base onto the linked list. */ +static PyObject * +ndarray_push(PyObject *self, PyObject *args, PyObject *kwds) +{ + NDArrayObject *nd = (NDArrayObject *)self; + static char *kwlist[] = { + "items", "shape", "strides", "offset", "fmt", "flags", NULL + }; + PyObject *items = NULL; /* initializer: scalar, list or tuple */ + PyObject *shape = NULL; /* size of each dimension */ + PyObject *strides = NULL; /* number of bytes to the next elt in each dim */ + PyObject *format = simple_format; /* struct module specifier: "B" */ + Py_ssize_t offset = 0; /* buffer offset */ + int flags = ND_UNUSED; /* base buffer flags */ + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO|OnOi", kwlist, + &items, &shape, &strides, &offset, &format, &flags)) + return NULL; + + if (flags & ND_VAREXPORT) { + PyErr_SetString(PyExc_ValueError, + "ND_VAREXPORT flag can only be used during object creation"); + return NULL; + } + if (ND_IS_CONSUMER(nd)) { + PyErr_SetString(PyExc_BufferError, + "structure of re-exporting object is immutable"); + return NULL; + } + if (!(nd->flags&ND_VAREXPORT) && nd->head->exports > 0) { + PyErr_Format(PyExc_BufferError, + "cannot change structure: %zd exported buffer%s", + nd->head->exports, nd->head->exports==1 ? "" : "s"); + return NULL; + } + + if (ndarray_push_base(nd, items, shape, strides, + offset, format, flags) < 0) + return NULL; + Py_RETURN_NONE; +} + +/* Pop a base from the linked list (if possible). */ +static PyObject * +ndarray_pop(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + if (ND_IS_CONSUMER(nd)) { + PyErr_SetString(PyExc_BufferError, + "structure of re-exporting object is immutable"); + return NULL; + } + if (nd->head->exports > 0) { + PyErr_Format(PyExc_BufferError, + "cannot change structure: %zd exported buffer%s", + nd->head->exports, nd->head->exports==1 ? "" : "s"); + return NULL; + } + if (nd->head->next == NULL) { + PyErr_SetString(PyExc_BufferError, + "list only has a single base"); + return NULL; + } + + ndbuf_pop(nd); + Py_RETURN_NONE; +} + +/**************************************************************************/ +/* getbuffer */ +/**************************************************************************/ + +static int +ndarray_getbuf(NDArrayObject *self, Py_buffer *view, int flags) +{ + ndbuf_t *ndbuf = self->head; + Py_buffer *base = &ndbuf->base; + int baseflags = ndbuf->flags; + + /* start with complete information */ + *view = *base; + view->obj = NULL; + + + if (base->ndim != 0 && + ((REQ_FORMAT(flags) && base->format == NULL) || + (REQ_SHAPE(flags) && base->shape == NULL) || + (REQ_STRIDES(flags) && base->strides == NULL))) { + /* The ndarray is a re-exporter that has been created without full + information for testing purposes. In this particular case the + ndarray is not a PEP-3118 compliant buffer provider. */ + PyErr_SetString(PyExc_BufferError, + "re-exporter does not provide format, shape or strides"); + return -1; + } + + if (baseflags & ND_GETBUF_FAIL) { + PyErr_SetString(PyExc_BufferError, + "ND_GETBUF_FAIL: forced test exception"); + return -1; + } + + if (REQ_WRITABLE(flags) && base->readonly) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not writable"); + return -1; + } + if (!REQ_FORMAT(flags)) { + if (!SIMPLE_FORMAT(base->format)) { + PyErr_SetString(PyExc_BufferError, + "ndarray requires format information"); + return -1; + } + view->format = NULL; + } + + if (REQ_C_CONTIGUOUS(flags) && !ND_C_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not C-contiguous"); + return -1; + } + if (REQ_F_CONTIGUOUS(flags) && !ND_FORTRAN_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not Fortran contiguous"); + return -1; + } + if (REQ_ANY_CONTIGUOUS(flags) && !ND_ANY_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not contiguous"); + return -1; + } + if (!REQ_INDIRECT(flags) && (baseflags & ND_PIL)) { + PyErr_SetString(PyExc_BufferError, + "ndarray cannot be represented without suboffsets"); + return -1; + } + if (!REQ_STRIDES(flags)) { + if (!ND_C_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not C-contiguous"); + return -1; + } + view->strides = NULL; + } + if (!REQ_SHAPE(flags)) { + /* PyBUF_SIMPLE or PyBUF_WRITABLE: at this point buf is C-contiguous, + so base->buf = ndbuf->data. */ + if (base->ndim != 1 || !SIMPLE_FORMAT(base->format)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not a simple sequence of unsigned bytes"); + return -1; + } + view->shape = NULL; + } + + view->obj = (PyObject *)self; + Py_INCREF(view->obj); + self->head->exports++; + + return 0; +} + +static int +ndarray_releasebuf(NDArrayObject *self, Py_buffer *view) +{ + if (!ND_IS_CONSUMER(self)) { + ndbuf_t *ndbuf = view->internal; + if (--ndbuf->exports == 0 && ndbuf != self->head) + ndbuf_delete(self, ndbuf); + } + + return 0; +} + +static PyBufferProcs ndarray_as_buffer = { + (getbufferproc)ndarray_getbuf, /* bf_getbuffer */ + (releasebufferproc)ndarray_releasebuf /* bf_releasebuffer */ +}; + + +/**************************************************************************/ +/* indexing/slicing */ +/**************************************************************************/ + +static char * +ptr_from_index(Py_buffer *base, Py_ssize_t index) +{ + char *ptr; + Py_ssize_t nitems; /* items in the first dimension */ + + if (base->shape) + nitems = base->shape[0]; + else { + assert(base->ndim == 1 && SIMPLE_FORMAT(base->format)); + nitems = base->len; + } + + if (index < 0) { + index += nitems; + } + if (index < 0 || index >= nitems) { + PyErr_SetString(PyExc_IndexError, "index out of bounds"); + return NULL; + } + + ptr = (char *)base->buf; + + if (base->strides == NULL) + ptr += base->itemsize * index; + else + ptr += base->strides[0] * index; + if (base->suboffsets != NULL && base->suboffsets[0] >= 0) + ptr = *((char **)ptr) + base->suboffsets[0]; + + return ptr; +} + +static PyObject * +ndarray_item(NDArrayObject *self, Py_ssize_t index) +{ + ndbuf_t *ndbuf = self->head; + Py_buffer *base = &ndbuf->base; + char *ptr; + + if (base->ndim == 0) { + PyErr_SetString(PyExc_TypeError, "invalid indexing of scalar"); + return NULL; + } + + ptr = ptr_from_index(base, index); + if (ptr == NULL) + return NULL; + + if (base->ndim == 1) { + return unpack_single(ptr, base->format ? base->format : "B", + base->itemsize); + } + else { + NDArrayObject *nd; + Py_buffer *subview; + + nd = (NDArrayObject *)ndarray_new(&NDArray_Type, NULL, NULL); + if (nd == NULL) + return NULL; + + if (ndarray_init_staticbuf((PyObject *)self, nd, PyBUF_FULL_RO) < 0) { + Py_DECREF(nd); + return NULL; + } + + subview = &nd->staticbuf.base; + + subview->buf = ptr; + subview->len /= subview->shape[0]; + + subview->ndim--; + subview->shape++; + if (subview->strides) subview->strides++; + if (subview->suboffsets) subview->suboffsets++; + + ndbuf_init_flags(&nd->staticbuf); + + return (PyObject *)nd; + } +} + +static int +init_slice(Py_buffer *base, PyObject *key, int dim) +{ + Py_ssize_t start, stop, step, slicelength; + + if (PySlice_GetIndicesEx(key, base->shape[dim], + &start, &stop, &step, &slicelength) < 0) { + return -1; + } + + + if (base->suboffsets == NULL || dim == 0) { + adjust_buf: + base->buf = (char *)base->buf + base->strides[dim] * start; + } + else { + Py_ssize_t n = dim-1; + while (n >= 0 && base->suboffsets[n] < 0) + n--; + if (n < 0) + goto adjust_buf; + base->suboffsets[n] = base->suboffsets[n] + base->strides[dim] * start; + } + base->shape[dim] = slicelength; + base->strides[dim] = base->strides[dim] * step; + + return 0; +} + +static int +copy_structure(Py_buffer *base) +{ + Py_ssize_t *shape = NULL, *strides = NULL, *suboffsets = NULL; + Py_ssize_t i; + + shape = PyMem_Malloc(base->ndim * (sizeof *shape)); + strides = PyMem_Malloc(base->ndim * (sizeof *strides)); + if (shape == NULL || strides == NULL) + goto err_nomem; + + suboffsets = NULL; + if (base->suboffsets) { + suboffsets = PyMem_Malloc(base->ndim * (sizeof *suboffsets)); + if (suboffsets == NULL) + goto err_nomem; + } + + for (i = 0; i < base->ndim; i++) { + shape[i] = base->shape[i]; + strides[i] = base->strides[i]; + if (suboffsets) + suboffsets[i] = base->suboffsets[i]; + } + + base->shape = shape; + base->strides = strides; + base->suboffsets = suboffsets; + + return 0; + +err_nomem: + PyErr_NoMemory(); + PyMem_XFree(shape); + PyMem_XFree(strides); + PyMem_XFree(suboffsets); + return -1; +} + +static PyObject * +ndarray_subscript(NDArrayObject *self, PyObject *key) +{ + NDArrayObject *nd; + ndbuf_t *ndbuf; + Py_buffer *base; + + if (self->head->base.ndim == 0) { + if (key == Py_Ellipsis || + (PyTuple_Check(key) && PyTuple_GET_SIZE(key) == 0)) { + Py_INCREF(self); + return (PyObject *)self; + } + else { + PyErr_SetString(PyExc_TypeError, "invalid indexing of scalar"); + return NULL; + } + } + if (PyIndex_Check(key)) { + Py_ssize_t index = PyLong_AsSsize_t(key); + if (index == -1 && PyErr_Occurred()) + return NULL; + return ndarray_item(self, index); + } + + nd = (NDArrayObject *)ndarray_new(&NDArray_Type, NULL, NULL); + if (nd == NULL) + return NULL; + + /* new ndarray is a consumer */ + if (ndarray_init_staticbuf((PyObject *)self, nd, PyBUF_FULL_RO) < 0) { + Py_DECREF(nd); + return NULL; + } + + /* copy shape, strides and suboffsets */ + ndbuf = nd->head; + base = &ndbuf->base; + if (copy_structure(base) < 0) { + Py_DECREF(nd); + return NULL; + } + ndbuf->flags |= ND_OWN_ARRAYS; + + if (PySlice_Check(key)) { + /* one-dimensional slice */ + if (init_slice(base, key, 0) < 0) + goto err_occurred; + } + else if PyTuple_Check(key) { + /* multi-dimensional slice */ + PyObject *tuple = key; + Py_ssize_t i, n; + + n = PyTuple_GET_SIZE(tuple); + for (i = 0; i < n; i++) { + key = PyTuple_GET_ITEM(tuple, i); + if (!PySlice_Check(key)) + goto type_error; + if (init_slice(base, key, (int)i) < 0) + goto err_occurred; + } + } + else { + goto type_error; + } + + init_len(base); + + return (PyObject *)nd; + + +type_error: + PyErr_Format(PyExc_TypeError, + "cannot index memory using \"%.200s\"", + key->ob_type->tp_name); +err_occurred: + Py_DECREF(nd); + return NULL; +} + + +static int +ndarray_ass_subscript(NDArrayObject *self, PyObject *key, PyObject *value) +{ + NDArrayObject *nd; + Py_buffer *dest = &self->head->base; + Py_buffer src; + char *ptr; + Py_ssize_t index; + int ret = -1; + + if (dest->readonly) { + PyErr_SetString(PyExc_TypeError, "ndarray is not writable"); + return -1; + } + if (value == NULL) { + PyErr_SetString(PyExc_TypeError, "ndarray data cannot be deleted"); + return -1; + } + + if (dest->ndim == 0) { + PyErr_SetString(PyExc_TypeError, "invalid indexing of scalar"); + return -1; + } + if (dest->ndim == 1 && PyIndex_Check(key)) { + /* rvalue must be a single item */ + index = PyLong_AsSsize_t(key); + if (index == -1 && PyErr_Occurred()) + return -1; + ptr = ptr_from_index(dest, index); + if (ptr == NULL) + return -1; + if (pack_single(ptr, value, dest->format, dest->itemsize) < 0) + return -1; + return 0; + } + + /* rvalue must be an exporter */ + if (PyObject_GetBuffer(value, &src, PyBUF_FULL_RO) == -1) + return -1; + + nd = (NDArrayObject *)ndarray_subscript(self, key); + if (nd != NULL) { + dest = &nd->head->base; + ret = copy_buffer(dest, &src); + Py_DECREF(nd); + } + + PyBuffer_Release(&src); + return ret; +} + +static PyObject *slice_indices(PyObject *self, PyObject *args) +{ + PyObject *ret, *key, *tmp; + Py_ssize_t s[4]; /* start, stop, step, slicelength */ + Py_ssize_t i, len; + + if (!PyArg_ParseTuple(args, "On", &key, &len)) { + return NULL; + } + if (PySlice_GetIndicesEx(key, len, &s[0], &s[1], &s[2], &s[3]) < 0) { + return NULL; + } + + ret = PyTuple_New(4); + if (ret == NULL) + return NULL; + + for (i = 0; i < 4; i++) { + tmp = PyLong_FromSsize_t(s[i]); + if (tmp == NULL) + goto error; + PyTuple_SET_ITEM(ret, i, tmp); + } + + return ret; + +error: + Py_DECREF(ret); + return NULL; +} + + +static PyMappingMethods ndarray_as_mapping = { + NULL, /* mp_length */ + (binaryfunc)ndarray_subscript, /* mp_subscript */ + (objobjargproc)ndarray_ass_subscript /* mp_ass_subscript */ +}; + +static PySequenceMethods ndarray_as_sequence = { + 0, /* sq_length */ + 0, /* sq_concat */ + 0, /* sq_repeat */ + (ssizeargfunc)ndarray_item, /* sq_item */ +}; + + +/**************************************************************************/ +/* getters */ +/**************************************************************************/ + +static PyObject * +ssize_array_as_tuple(Py_ssize_t *array, Py_ssize_t len) +{ + PyObject *tuple, *x; + Py_ssize_t i; + + if (array == NULL) + return PyTuple_New(0); + + tuple = PyTuple_New(len); + if (tuple == NULL) + return NULL; + + for (i = 0; i < len; i++) { + x = PyLong_FromSsize_t(array[i]); + if (x == NULL) { + Py_DECREF(tuple); + return NULL; + } + PyTuple_SET_ITEM(tuple, i, x); + } + + return tuple; +} + +static PyObject * +ndarray_getflags(NDArrayObject *self, void *closure) +{ + return PyLong_FromLong(self->head->flags); +} + +static PyObject * +ndarray_getobj(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + + if (base->obj == NULL) { + Py_RETURN_NONE; + } + Py_INCREF(base->obj); + return base->obj; +} + +static PyObject * +ndarray_getlen(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromSsize_t(base->len); +} + +static PyObject * +ndarray_getreadonly(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromLong(base->readonly); +} + +static PyObject * +ndarray_getitemsize(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromSsize_t(base->itemsize); +} + +static PyObject * +ndarray_getformat(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + char *fmt = base->format ? base->format : ""; + return PyUnicode_FromString(fmt); +} + +static PyObject * +ndarray_getndim(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromSsize_t(base->ndim); +} + +static PyObject * +ndarray_getshape(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return ssize_array_as_tuple(base->shape, base->ndim); +} + +static PyObject * +ndarray_getstrides(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return ssize_array_as_tuple(base->strides, base->ndim); +} + +static PyObject * +ndarray_getsuboffsets(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return ssize_array_as_tuple(base->suboffsets, base->ndim); +} + +static PyObject * +ndarray_c_contig(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + int ret = PyBuffer_IsContiguous(&nd->head->base, 'C'); + + if (ret != ND_C_CONTIGUOUS(nd->head->flags)) { + PyErr_SetString(PyExc_RuntimeError, + "results from PyBuffer_IsContiguous() and flags differ"); + return NULL; + } + return PyBool_FromLong(ret); +} + +static PyObject * +ndarray_fortran_contig(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + int ret = PyBuffer_IsContiguous(&nd->head->base, 'F'); + + if (ret != ND_FORTRAN_CONTIGUOUS(nd->head->flags)) { + PyErr_SetString(PyExc_RuntimeError, + "results from PyBuffer_IsContiguous() and flags differ"); + return NULL; + } + return PyBool_FromLong(ret); +} + +static PyObject * +ndarray_contig(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + int ret = PyBuffer_IsContiguous(&nd->head->base, 'A'); + + if (ret != ND_ANY_CONTIGUOUS(nd->head->flags)) { + PyErr_SetString(PyExc_RuntimeError, + "results from PyBuffer_IsContiguous() and flags differ"); + return NULL; + } + return PyBool_FromLong(ret); +} + + +static PyGetSetDef ndarray_getset [] = +{ + /* ndbuf */ + { "flags", (getter)ndarray_getflags, NULL, NULL, NULL}, + /* ndbuf.base */ + { "obj", (getter)ndarray_getobj, NULL, NULL, NULL}, + { "len", (getter)ndarray_getlen, NULL, NULL, NULL}, + { "readonly", (getter)ndarray_getreadonly, NULL, NULL, NULL}, + { "itemsize", (getter)ndarray_getitemsize, NULL, NULL, NULL}, + { "fmt", (getter)ndarray_getformat, NULL, NULL, NULL}, + { "ndim", (getter)ndarray_getndim, NULL, NULL, NULL}, + { "shape", (getter)ndarray_getshape, NULL, NULL, NULL}, + { "strides", (getter)ndarray_getstrides, NULL, NULL, NULL}, + { "suboffsets", (getter)ndarray_getsuboffsets, NULL, NULL, NULL}, + { "c_contiguous", (getter)ndarray_c_contig, NULL, NULL, NULL }, + { "f_contiguous", (getter)ndarray_fortran_contig, NULL, NULL, NULL }, + { "contiguous", (getter)ndarray_contig, NULL, NULL, NULL }, + {NULL} +}; + +static PyObject * +ndarray_tolist(PyObject *self, PyObject *dummy) +{ + return ndarray_as_list((NDArrayObject *)self); +} + +/* add redundant (negative) suboffsets for testing */ +static PyObject * +ndarray_add_suboffsets(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + Py_buffer *base = &nd->head->base; + Py_ssize_t i; + + if (base->suboffsets != NULL) { + PyErr_SetString(PyExc_TypeError, + "cannot add suboffsets to PIL-style array"); + return NULL; + } + if (base->strides == NULL) { + PyErr_SetString(PyExc_TypeError, + "cannot add suboffsets to array without strides"); + return NULL; + } + + base->suboffsets = PyMem_Malloc(base->ndim * (sizeof *base->suboffsets)); + if (base->suboffsets == NULL) { + PyErr_NoMemory(); + return NULL; + } + + for (i = 0; i < base->ndim; i++) + base->suboffsets[i] = -1; + + Py_RETURN_NONE; +} + +static PyMethodDef ndarray_methods [] = +{ + { "tolist", ndarray_tolist, METH_NOARGS, NULL }, + { "push", (PyCFunction)ndarray_push, METH_VARARGS|METH_KEYWORDS, NULL }, + { "pop", ndarray_pop, METH_NOARGS, NULL }, + { "add_suboffsets", ndarray_add_suboffsets, METH_NOARGS, NULL }, + {NULL} +}; + +static PyTypeObject NDArray_Type = { + PyVarObject_HEAD_INIT(NULL, 0) + "ndarray", /* Name of this type */ + sizeof(NDArrayObject), /* Basic object size */ + 0, /* Item size for varobject */ + (destructor)ndarray_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + &ndarray_as_sequence, /* tp_as_sequence */ + &ndarray_as_mapping, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + &ndarray_as_buffer, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + 0, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + ndarray_methods, /* tp_methods */ + 0, /* tp_members */ + ndarray_getset, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + ndarray_init, /* tp_init */ + 0, /* tp_alloc */ + ndarray_new, /* tp_new */ +}; + +static struct PyMethodDef _testbuffer_functions[] = { + {"slice_indices", slice_indices, METH_VARARGS, NULL}, + {NULL, NULL} +}; + +static struct PyModuleDef _testbuffermodule = { + PyModuleDef_HEAD_INIT, + "_testbuffer", + NULL, + -1, + _testbuffer_functions, + NULL, + NULL, + NULL, + NULL +}; + +PyMODINIT_FUNC +PyInit__testbuffer(void) +{ + PyObject *m; + + m = PyModule_Create(&_testbuffermodule); + if (m == NULL) + return NULL; + + Py_TYPE(&NDArray_Type)=&PyType_Type; + Py_INCREF(&NDArray_Type); + PyModule_AddObject(m, "ndarray", (PyObject *)&NDArray_Type); + + structmodule = PyImport_ImportModule("struct"); + if (structmodule == NULL) + return NULL; + + Struct = PyObject_GetAttrString(structmodule, "Struct"); + calcsize = PyObject_GetAttrString(structmodule, "calcsize"); + if (Struct == NULL || calcsize == NULL) + return NULL; + + simple_format = PyUnicode_FromString(simple_fmt); + if (simple_format == NULL) + return NULL; + + PyModule_AddIntConstant(m, "ND_VAREXPORT", ND_VAREXPORT); + PyModule_AddIntConstant(m, "ND_WRITABLE", ND_WRITABLE); + PyModule_AddIntConstant(m, "ND_FORTRAN", ND_FORTRAN); + PyModule_AddIntConstant(m, "ND_SCALAR", ND_SCALAR); + PyModule_AddIntConstant(m, "ND_PIL", ND_PIL); + PyModule_AddIntConstant(m, "ND_GETBUF_FAIL", ND_GETBUF_FAIL); + + PyModule_AddIntConstant(m, "PyBUF_SIMPLE", PyBUF_SIMPLE); + PyModule_AddIntConstant(m, "PyBUF_WRITABLE", PyBUF_WRITABLE); + PyModule_AddIntConstant(m, "PyBUF_FORMAT", PyBUF_FORMAT); + PyModule_AddIntConstant(m, "PyBUF_ND", PyBUF_ND); + PyModule_AddIntConstant(m, "PyBUF_STRIDES", PyBUF_STRIDES); + PyModule_AddIntConstant(m, "PyBUF_INDIRECT", PyBUF_INDIRECT); + PyModule_AddIntConstant(m, "PyBUF_C_CONTIGUOUS", PyBUF_C_CONTIGUOUS); + PyModule_AddIntConstant(m, "PyBUF_F_CONTIGUOUS", PyBUF_F_CONTIGUOUS); + PyModule_AddIntConstant(m, "PyBUF_ANY_CONTIGUOUS", PyBUF_ANY_CONTIGUOUS); + PyModule_AddIntConstant(m, "PyBUF_FULL", PyBUF_FULL); + PyModule_AddIntConstant(m, "PyBUF_FULL_RO", PyBUF_FULL_RO); + PyModule_AddIntConstant(m, "PyBUF_RECORDS", PyBUF_RECORDS); + PyModule_AddIntConstant(m, "PyBUF_RECORDS_RO", PyBUF_RECORDS_RO); + PyModule_AddIntConstant(m, "PyBUF_STRIDED", PyBUF_STRIDED); + PyModule_AddIntConstant(m, "PyBUF_STRIDED_RO", PyBUF_STRIDED_RO); + PyModule_AddIntConstant(m, "PyBUF_CONTIG", PyBUF_CONTIG); + PyModule_AddIntConstant(m, "PyBUF_CONTIG_RO", PyBUF_CONTIG_RO); + + return m; +} + + + diff -r 69b354a8c90f -r 78fb1181787a Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c Tue Aug 09 18:09:21 2011 +0200 +++ b/Modules/_testcapimodule.c Tue Aug 09 20:28:59 2011 +0200 @@ -275,95 +275,6 @@ } -/* Issue #7385: Check that memoryview() does not crash - * when bf_getbuffer returns an error - */ - -static int -broken_buffer_getbuffer(PyObject *self, Py_buffer *view, int flags) -{ - PyErr_SetString( - TestError, - "test_broken_memoryview: expected error in bf_getbuffer"); - return -1; -} - -static PyBufferProcs memoryviewtester_as_buffer = { - (getbufferproc)broken_buffer_getbuffer, /* bf_getbuffer */ - 0, /* bf_releasebuffer */ -}; - -static PyTypeObject _MemoryViewTester_Type = { - PyVarObject_HEAD_INIT(NULL, 0) - "memoryviewtester", /* Name of this type */ - sizeof(PyObject), /* Basic object size */ - 0, /* Item size for varobject */ - (destructor)PyObject_Del, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - &memoryviewtester_as_buffer, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - 0, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - PyType_GenericNew, /* tp_new */ -}; - -static PyObject* -test_broken_memoryview(PyObject* self) -{ - PyObject *obj = PyObject_New(PyObject, &_MemoryViewTester_Type); - PyObject *res; - - if (obj == NULL) { - PyErr_Clear(); - PyErr_SetString( - TestError, - "test_broken_memoryview: failed to create object"); - return NULL; - } - - res = PyMemoryView_FromObject(obj); - if (res || !PyErr_Occurred()){ - PyErr_SetString( - TestError, - "test_broken_memoryview: memoryview() didn't raise an Exception"); - Py_XDECREF(res); - Py_DECREF(obj); - return NULL; - } - - PyErr_Clear(); - Py_DECREF(obj); - Py_RETURN_NONE; -} - - /* Tests of PyLong_{As, From}{Unsigned,}Long(), and (#ifdef HAVE_LONG_LONG) PyLong_{As, From}{Unsigned,}LongLong(). @@ -2263,7 +2174,6 @@ {"test_list_api", (PyCFunction)test_list_api, METH_NOARGS}, {"test_dict_iteration", (PyCFunction)test_dict_iteration,METH_NOARGS}, {"test_lazy_hash_inheritance", (PyCFunction)test_lazy_hash_inheritance,METH_NOARGS}, - {"test_broken_memoryview", (PyCFunction)test_broken_memoryview,METH_NOARGS}, {"test_long_api", (PyCFunction)test_long_api, METH_NOARGS}, {"test_long_and_overflow", (PyCFunction)test_long_and_overflow, METH_NOARGS}, @@ -2521,7 +2431,6 @@ return NULL; Py_TYPE(&_HashInheritanceTester_Type)=&PyType_Type; - Py_TYPE(&_MemoryViewTester_Type)=&PyType_Type; Py_TYPE(&test_structmembersType)=&PyType_Type; Py_INCREF(&test_structmembersType); diff -r 69b354a8c90f -r 78fb1181787a Objects/abstract.c --- a/Objects/abstract.c Tue Aug 09 18:09:21 2011 +0200 +++ b/Objects/abstract.c Tue Aug 09 20:28:59 2011 +0200 @@ -651,7 +651,7 @@ PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int flags) { - if (view == NULL) return 0; + if (view == NULL) return 0; /* XXX why not -1? */ if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && (readonly == 1)) { PyErr_SetString(PyExc_BufferError, diff -r 69b354a8c90f -r 78fb1181787a Objects/memoryobject.c --- a/Objects/memoryobject.c Tue Aug 09 18:09:21 2011 +0200 +++ b/Objects/memoryobject.c Tue Aug 09 20:28:59 2011 +0200 @@ -1,25 +1,180 @@ - /* Memoryview object implementation */ #include "Python.h" -#define IS_RELEASED(memobj) \ - (((PyMemoryViewObject *) memobj)->view.buf == NULL) +/* + ManagedBuffer Object: + --------------------- + + The purpose of this object is to facilitate the handling of chained + memoryviews that have the same underlying exporting object. PEP-3118 + allows the underlying object to change while a view is exported. This + could lead to unexpected results when constructing a new memoryview + from an existing memoryview. + + Rather than repeatedly redirecting buffer requests to the original base + object, all chained memoryviews use a single buffer snapshot. This + snapshot is generated by the constructor PyManagedBuffer_FromObject(). + + Ownership rules: + ---------------- + + The master buffer inside a ManagedBuffer is filled in by the original + base object and is read-only for all consumers. + + A MemoryView's buffer is a private copy of the ancestor's buffer. The + shape, strides and suboffsets arrays belong to the MemoryView and are + thus writable. + + If a MemoryView exports several buffers via memory_getbuf(), all + buffer copies share shape, strides and suboffsets. In this case, + the arrays are NOT writable. + + Reference count assumptions: + ---------------------------- + + The 'obj' member of a Py_buffer must either be NULL or refer to the + exporting base object. In the Python codebase, all getbufferprocs + return a new reference to view.obj (example: bytes_buffer_getbuffer). + + PyBuffer_Release() decrements view.obj (if non-NULL), so the + releasebufferprocs should NOT decrement view.obj. +*/ + +#define CHECK_MBUF_RELEASED(mbuf) \ + if (((PyManagedBufferObject *) mbuf)->released) { \ + PyErr_SetString(PyExc_ValueError, \ + "operation forbidden on released managedbuffer object"); \ + return NULL; \ + } + +PyObject * +PyManagedBuffer_FromObject(PyObject *base) +{ + PyManagedBufferObject *mbuf; + + mbuf = (PyManagedBufferObject *) + PyObject_GC_New(PyManagedBufferObject, &PyManagedBuffer_Type); + if (mbuf == NULL) + return NULL; + mbuf->released = 0; + mbuf->exports = 0; + mbuf->master.obj = NULL; + _PyObject_GC_TRACK(mbuf); + + if (PyObject_GetBuffer(base, &mbuf->master, PyBUF_FULL_RO) < 0) { + /* mbuf->master.obj must be NULL. */ + Py_DECREF(mbuf); + return NULL; + } + + /* Assume that master.obj is a new reference to base. */ + assert(mbuf->master.obj == base); + + return (PyObject *)mbuf; +} + +static void +mbuf_release(PyManagedBufferObject *self) +{ + if (self->released) + return; + + /* NOTE: at this point self->exports still can be > 0 if this function + is called from mbuf_clear() to break up a reference cycle. */ + self->released = 1; + + /* PyBuffer_Release() decrements master->obj and sets it to NULL. */ + _PyObject_GC_UNTRACK(self); + PyBuffer_Release(&self->master); +} + +static void +mbuf_dealloc(PyManagedBufferObject *self) +{ + assert(self->exports == 0); + mbuf_release(self); + PyObject_GC_Del(self); +} + +static int +mbuf_traverse(PyManagedBufferObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->master.obj); + return 0; +} + +static int +mbuf_clear(PyManagedBufferObject *self) +{ + assert(self->exports >= 0); + mbuf_release(self); + return 0; +} + +PyTypeObject PyManagedBuffer_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "managedbuffer", + sizeof(PyManagedBufferObject), + 0, + (destructor)mbuf_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ + 0, /* tp_doc */ + (traverseproc)mbuf_traverse, /* tp_traverse */ + (inquiry)mbuf_clear /* tp_clear */ +}; + + +/* Actual MemoryView Object */ + +/* In the process of breaking reference cycles mbuf_release() can be + called before memory_release(). */ +#define BASE_INACCESSIBLE(memobj) \ + (((PyMemoryViewObject *) memobj)->released || \ + ((PyMemoryViewObject *) memobj)->mbuf->released) #define CHECK_RELEASED(memobj) \ - if (IS_RELEASED(memobj)) { \ + if (BASE_INACCESSIBLE(memobj)) { \ PyErr_SetString(PyExc_ValueError, \ "operation forbidden on released memoryview object"); \ return NULL; \ } #define CHECK_RELEASED_INT(memobj) \ - if (IS_RELEASED(memobj)) { \ + if (BASE_INACCESSIBLE(memobj)) { \ PyErr_SetString(PyExc_ValueError, \ "operation forbidden on released memoryview object"); \ return -1; \ } +#define CHECK_SHAPE(buf) \ + if (buf->shape == NULL) { \ + PyErr_SetString(PyExc_TypeError, \ + "shape information missing"); \ + return NULL; \ + } + +#define CHECK_SHAPE_INT(buf) \ + if (buf->shape == NULL) { \ + PyErr_SetString(PyExc_TypeError, \ + "shape information missing"); \ + return -1; \ + } + static Py_ssize_t get_shape0(Py_buffer *buf) { @@ -33,86 +188,110 @@ return -1; } -static void -dup_buffer(Py_buffer *dest, Py_buffer *src) -{ - *dest = *src; - if (src->ndim == 1 && src->shape != NULL) { - dest->shape = &(dest->smalltable[0]); - dest->shape[0] = get_shape0(src); - } - if (src->ndim == 1 && src->strides != NULL) { - dest->strides = &(dest->smalltable[1]); - dest->strides[0] = src->strides[0]; - } -} - -static int -memory_getbuf(PyMemoryViewObject *self, Py_buffer *view, int flags) -{ - int res = 0; - CHECK_RELEASED_INT(self); - if (self->view.obj != NULL) - res = PyObject_GetBuffer(self->view.obj, view, flags); - if (view) - dup_buffer(view, &self->view); - return res; -} - -static void -memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view) -{ - PyBuffer_Release(view); -} - PyDoc_STRVAR(memory_doc, "memoryview(object)\n\ \n\ Create a new memoryview object which references the given object."); PyObject * +PyMemoryView_FromBytes(char *bytes, Py_ssize_t size, int flags) +{ + PyManagedBufferObject *mbuf; + PyObject *mview; + int readonly; + + assert(bytes != NULL); + assert(flags == PyBUF_READ || flags == PyBUF_WRITE); + + mbuf = (PyManagedBufferObject *) + PyObject_GC_New(PyManagedBufferObject, &PyManagedBuffer_Type); + if (mbuf == NULL) + return NULL; + + mbuf->released = 0; + mbuf->exports = 0; + + readonly = (flags == PyBUF_WRITE) ? 0 : 1; + (void)PyBuffer_FillInfo(&mbuf->master, NULL, bytes, size, readonly, + PyBUF_FULL_RO); + mbuf->master.format = "c"; + + _PyObject_GC_TRACK(mbuf); + mview = PyMemoryView_FromObject((PyObject *)mbuf); + Py_DECREF(mbuf); + return mview; +} + +PyObject * PyMemoryView_FromBuffer(Py_buffer *info) { - PyMemoryViewObject *mview; + PyManagedBufferObject *mbuf; + PyObject *mview; if (info->buf == NULL) { PyErr_SetString(PyExc_ValueError, "cannot make memory view from a buffer with a NULL data pointer"); return NULL; } - mview = (PyMemoryViewObject *) - PyObject_GC_New(PyMemoryViewObject, &PyMemoryView_Type); - if (mview == NULL) + + mbuf = (PyManagedBufferObject *) + PyObject_GC_New(PyManagedBufferObject, &PyManagedBuffer_Type); + if (mbuf == NULL) return NULL; - dup_buffer(&mview->view, info); - /* NOTE: mview->view.obj should already have been incref'ed as - part of PyBuffer_FillInfo(). */ - _PyObject_GC_TRACK(mview); - return (PyObject *)mview; + + mbuf->released = 0; + mbuf->exports = 0; + /* info->obj is either NULL or we own the reference. */ + mbuf->master = *info; + + _PyObject_GC_TRACK(mbuf); + mview = PyMemoryView_FromObject((PyObject *)mbuf); + Py_DECREF(mbuf); + return mview; } PyObject * -PyMemoryView_FromObject(PyObject *base) +PyMemoryView_FromObject(PyObject *v) { PyMemoryViewObject *mview; - Py_buffer view; + PyManagedBufferObject *mbuf; - if (!PyObject_CheckBuffer(base)) { + if (PyMemoryView_Check(v)) { + CHECK_RELEASED(v); + mbuf = ((PyMemoryViewObject *)v)->mbuf; + Py_INCREF(mbuf); + } + else if (PyObject_CheckBuffer(v)) { + mbuf = (PyManagedBufferObject *)PyManagedBuffer_FromObject(v); + if (mbuf == NULL) + return NULL; + } + else if (PyManagedBuffer_Check(v)) { + CHECK_MBUF_RELEASED(v); + mbuf = (PyManagedBufferObject *)v; + Py_INCREF(mbuf); + } + else { PyErr_SetString(PyExc_TypeError, "cannot make memory view because object does " "not have the buffer interface"); return NULL; } - if (PyObject_GetBuffer(base, &view, PyBUF_FULL_RO) < 0) - return NULL; - - mview = (PyMemoryViewObject *)PyMemoryView_FromBuffer(&view); + mview = (PyMemoryViewObject *) + PyObject_GC_New(PyMemoryViewObject, &PyMemoryView_Type); if (mview == NULL) { - PyBuffer_Release(&view); + Py_DECREF(mbuf); return NULL; } + mview->released = 0; + mview->exports = 0; + mview->mbuf = mbuf; + mview->view = mbuf->master; + mbuf->exports++; + + _PyObject_GC_TRACK(mview); return (PyObject *)mview; } @@ -130,6 +309,102 @@ return PyMemoryView_FromObject(obj); } +/* Buffer methods */ +static int +memory_getbuf(PyMemoryViewObject *self, Py_buffer *view, int flags) +{ + CHECK_RELEASED_INT(self); + + /* XXX All flags are ignored, except for this basic check. */ + if ((flags&PyBUF_WRITABLE) && self->view.readonly) + return -1; + + *view = self->view; + view->obj = (PyObject *)self; + Py_INCREF(view->obj); + + self->exports++; + return 0; +} + +static void +memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view) +{ + self->exports--; + return; + /* PyBuffer_Release() decrements view->obj after this function returns. */ +} + +static int +mview_release(PyMemoryViewObject *self) +{ + if (self->released) + return 0; + + if (self->exports == 0) { + self->released = 1; + assert(self->mbuf->exports > 0); + if (--self->mbuf->exports == 0) + mbuf_release(self->mbuf); + return 0; + } + if (self->exports > 0) { + (void)PyErr_Format(PyExc_BufferError, + "memoryview has %zd exported buffer%s", self->exports, + self->exports==1 ? "" : "s"); + return -1; + } + + Py_FatalError("mview_release: negative export count"); + return -1; +} + +static PyObject * +memory_release(PyMemoryViewObject *self) +{ + if (mview_release(self) < 0) + return NULL; + Py_RETURN_NONE; +} + +static void +memory_dealloc(PyMemoryViewObject *self) +{ + assert(self->exports == 0); + _PyObject_GC_UNTRACK(self); + (void)mview_release(self); + Py_CLEAR(self->mbuf); + PyObject_GC_Del(self); +} + +static int +memory_traverse(PyMemoryViewObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->mbuf); + return 0; +} + +static int +memory_clear(PyMemoryViewObject *self) +{ + (void)mview_release(self); + Py_CLEAR(self->mbuf); + return 0; +} + +static PyObject * +memory_enter(PyObject *self, PyObject *args) +{ + CHECK_RELEASED(self); + Py_INCREF(self); + return self; +} + +static PyObject * +memory_exit(PyObject *self, PyObject *args) +{ + return memory_release((PyMemoryViewObject *)self); +} static void _strided_copy_nd(char *dest, char *src, int nd, Py_ssize_t *shape, @@ -262,7 +537,6 @@ PyMemoryViewObject *mem; PyObject *bytes; Py_buffer *view; - int flags; char *dest; if (!PyObject_CheckBuffer(obj)) { @@ -271,26 +545,20 @@ return NULL; } - mem = PyObject_GC_New(PyMemoryViewObject, &PyMemoryView_Type); + mem = (PyMemoryViewObject *)PyMemoryView_FromObject(obj); if (mem == NULL) return NULL; view = &mem->view; - flags = PyBUF_FULL_RO; - switch(buffertype) { - case PyBUF_WRITE: - flags = PyBUF_FULL; - break; - } - - if (PyObject_GetBuffer(obj, view, flags) != 0) { + if (buffertype == PyBUF_WRITE && view->readonly) { + PyErr_SetString(PyExc_BufferError, + "underlying buffer is not writable"); Py_DECREF(mem); return NULL; } if (PyBuffer_IsContiguous(view, fort)) { /* no copy needed */ - _PyObject_GC_TRACK(mem); return (PyObject *)mem; } /* otherwise a copy is needed */ @@ -298,9 +566,18 @@ Py_DECREF(mem); PyErr_SetString(PyExc_BufferError, "writable contiguous buffer requested " - "for a non-contiguousobject."); + "for a non-contiguous object."); return NULL; } + + PyErr_SetString(PyExc_NotImplementedError, + "feature is not implemented"); + Py_DECREF(mem); + return NULL; + /* XXX NOT REACHED: The following lines need a test case. Currently + 'dest' is filled in but not used: 'mem' is returned unchanged. + */ + bytes = PyBytes_FromStringAndSize(NULL, view->len); if (bytes == NULL) { Py_DECREF(mem); @@ -323,7 +600,6 @@ } PyBuffer_Release(view); /* XXX ? */ } - _PyObject_GC_TRACK(mem); return (PyObject *)mem; } @@ -460,63 +736,32 @@ return res; } -static void -do_release(PyMemoryViewObject *self) -{ - if (self->view.obj != NULL) { - PyBuffer_Release(&(self->view)); - } - self->view.obj = NULL; - self->view.buf = NULL; -} - -static PyObject * -memory_enter(PyObject *self, PyObject *args) -{ - CHECK_RELEASED(self); - Py_INCREF(self); - return self; -} - -static PyObject * -memory_exit(PyObject *self, PyObject *args) -{ - do_release((PyMemoryViewObject *) self); - Py_RETURN_NONE; -} - static PyMethodDef memory_methods[] = { - {"release", memory_exit, METH_NOARGS}, - {"tobytes", (PyCFunction)memory_tobytes, METH_NOARGS, NULL}, - {"tolist", (PyCFunction)memory_tolist, METH_NOARGS, NULL}, + {"release", (PyCFunction)memory_release, METH_NOARGS}, + {"tobytes", (PyCFunction)memory_tobytes, METH_NOARGS, NULL}, + {"tolist", (PyCFunction)memory_tolist, METH_NOARGS, NULL}, {"__enter__", memory_enter, METH_NOARGS}, - {"__exit__", memory_exit, METH_VARARGS}, - {NULL, NULL} /* sentinel */ + {"__exit__", memory_exit, METH_VARARGS}, + {NULL, NULL} }; -static void -memory_dealloc(PyMemoryViewObject *self) -{ - _PyObject_GC_UNTRACK(self); - do_release(self); - PyObject_GC_Del(self); -} - static PyObject * memory_repr(PyMemoryViewObject *self) { - if (IS_RELEASED(self)) + if (self->released) return PyUnicode_FromFormat("", self); else return PyUnicode_FromFormat("", self); } + /* Sequence methods */ static Py_ssize_t memory_length(PyMemoryViewObject *self) { CHECK_RELEASED_INT(self); + /* XXX handle ndim > 1 */ return get_shape0(&self->view); } @@ -536,12 +781,14 @@ } if (view->ndim == 1) { /* Return a bytes object */ - char *ptr; - ptr = (char *)view->buf; + char *ptr = (char *)view->buf; + + CHECK_SHAPE(view); + if (result < 0) { - result += get_shape0(view); + result += view->shape[0]; } - if ((result < 0) || (result >= get_shape0(view))) { + if ((result < 0) || (result >= view->shape[0])) { PyErr_SetString(PyExc_IndexError, "index out of bounds"); return NULL; @@ -556,14 +803,26 @@ } return PyBytes_FromStringAndSize(ptr, view->itemsize); } else { + PyErr_SetString(PyExc_NotImplementedError, + "multi-dimensional sub-views are not implemented"); + return NULL; +#if 0 /* Return a new memory-view object */ Py_buffer newview; memset(&newview, 0, sizeof(newview)); /* XXX: This needs to be fixed so it actually returns a sub-view */ return PyMemoryView_FromBuffer(&newview); +#endif } } +static void +init_arrays(PyMemoryViewObject *m) +{ + m->view.shape = m->shape; + m->view.strides = m->strides; +} + /* mem[obj] returns a bytes object holding the data for one element if obj fully indexes the memory view or another memory-view object @@ -601,35 +860,40 @@ else if (PySlice_Check(key)) { Py_ssize_t start, stop, step, slicelength; - if (PySlice_GetIndicesEx(key, get_shape0(view), + CHECK_SHAPE(view); + + if (PySlice_GetIndicesEx(key, view->shape[0], &start, &stop, &step, &slicelength) < 0) { return NULL; } if (step == 1 && view->ndim == 1) { - Py_buffer newview; - void *newbuf = (char *) view->buf - + start * view->itemsize; - int newflags = view->readonly - ? PyBUF_CONTIG_RO : PyBUF_CONTIG; - - /* XXX There should be an API to create a subbuffer */ - if (view->obj != NULL) { - if (PyObject_GetBuffer(view->obj, &newview, newflags) == -1) - return NULL; + + PyMemoryViewObject *sliced; + void *newbuf = (char *) view->buf + start * view->itemsize; + + if (!PyBuffer_IsContiguous(&self->view, 'A')) { + PyErr_SetString(PyExc_BufferError, + "sliced view requires a contiguous base"); + return NULL; } - else { - newview = *view; - } - newview.buf = newbuf; - newview.len = slicelength * newview.itemsize; - newview.format = view->format; - newview.shape = &(newview.smalltable[0]); - newview.shape[0] = slicelength; - newview.strides = &(newview.itemsize); - return PyMemoryView_FromBuffer(&newview); + + sliced = (PyMemoryViewObject *) + PyMemoryView_FromObject((PyObject *)self); + if (sliced == NULL) + return NULL; + + init_arrays(sliced); + + sliced->view.buf = newbuf; + sliced->view.len = slicelength * view->itemsize; + sliced->view.format = view->format; + sliced->view.shape[0] = slicelength; + sliced->view.strides[0] = view->itemsize; + return (PyObject *)sliced; } - PyErr_SetNone(PyExc_NotImplementedError); + PyErr_SetString(PyExc_NotImplementedError, + "multi-dimensional slices are not implemented"); return NULL; } PyErr_Format(PyExc_TypeError, @@ -663,14 +927,17 @@ PyErr_SetNone(PyExc_NotImplementedError); return -1; } + + CHECK_SHAPE_INT(view); + if (PyIndex_Check(key)) { start = PyNumber_AsSsize_t(key, NULL); if (start == -1 && PyErr_Occurred()) return -1; if (start < 0) { - start += get_shape0(view); + start += view->shape[0]; } - if ((start < 0) || (start >= get_shape0(view))) { + if ((start < 0) || (start >= view->shape[0])) { PyErr_SetString(PyExc_IndexError, "index out of bounds"); return -1; @@ -680,7 +947,7 @@ else if (PySlice_Check(key)) { Py_ssize_t stop, step; - if (PySlice_GetIndicesEx(key, get_shape0(view), + if (PySlice_GetIndicesEx(key, view->shape[0], &start, &stop, &step, &len) < 0) { return -1; } @@ -741,8 +1008,8 @@ ww.obj = NULL; if (op != Py_EQ && op != Py_NE) goto _notimpl; - if ((PyMemoryView_Check(v) && IS_RELEASED(v)) || - (PyMemoryView_Check(w) && IS_RELEASED(w))) { + if ((PyMemoryView_Check(v) && BASE_INACCESSIBLE(v)) || + (PyMemoryView_Check(w) && BASE_INACCESSIBLE(w))) { equal = (v == w); goto _end; } @@ -778,22 +1045,6 @@ } -static int -memory_traverse(PyMemoryViewObject *self, visitproc visit, void *arg) -{ - if (self->view.obj != NULL) - Py_VISIT(self->view.obj); - return 0; -} - -static int -memory_clear(PyMemoryViewObject *self) -{ - PyBuffer_Release(&self->view); - return 0; -} - - /* As mapping */ static PyMappingMethods memory_as_mapping = { (lenfunc)memory_length, /* mp_length */ diff -r 69b354a8c90f -r 78fb1181787a Objects/object.c --- a/Objects/object.c Tue Aug 09 18:09:21 2011 +0200 +++ b/Objects/object.c Tue Aug 09 20:28:59 2011 +0200 @@ -1517,6 +1517,9 @@ if (PyType_Ready(&PyProperty_Type) < 0) Py_FatalError("Can't initialize property type"); + if (PyType_Ready(&PyManagedBuffer_Type) < 0) + Py_FatalError("Can't initialize managed buffer type"); + if (PyType_Ready(&PyMemoryView_Type) < 0) Py_FatalError("Can't initialize memoryview type"); diff -r 69b354a8c90f -r 78fb1181787a PCbuild/_testbuffer.vcproj --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/PCbuild/_testbuffer.vcproj Tue Aug 09 20:28:59 2011 +0200 @@ -0,0 +1,521 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff -r 69b354a8c90f -r 78fb1181787a PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln Tue Aug 09 18:09:21 2011 +0200 +++ b/PCbuild/pcbuild.sln Tue Aug 09 20:28:59 2011 +0200 @@ -137,6 +137,11 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "xxlimited", "xxlimited.vcproj", "{F749B822-B489-4CA5-A3AD-CE078F5F338A}" EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testbuffer", "_testbuffer.vcproj", "{A2697BD3-28C1-4AEC-9106-8B748639FD16}" + ProjectSection(ProjectDependencies) = postProject + {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} + EndProjectSection +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 @@ -588,6 +593,22 @@ {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|Win32.Build.0 = Release|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|x64.ActiveCfg = Release|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|x64.Build.0 = Release|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|Win32.ActiveCfg = Debug|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|Win32.Build.0 = Debug|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|x64.ActiveCfg = Debug|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|x64.Build.0 = Debug|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|Win32.ActiveCfg = Release|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|Win32.Build.0 = Release|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|x64.ActiveCfg = Release|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff -r 69b354a8c90f -r 78fb1181787a PCbuild/readme.txt --- a/PCbuild/readme.txt Tue Aug 09 18:09:21 2011 +0200 +++ b/PCbuild/readme.txt Tue Aug 09 20:28:59 2011 +0200 @@ -92,6 +92,9 @@ _testcapi tests of the Python C API, run via Lib/test/test_capi.py, and implemented by module Modules/_testcapimodule.c +_testbuffer + buffer protocol tests, run via Lib/test/test_buffer.py, and + implemented by module Modules/_testbuffer.c pyexpat Python wrapper for accelerated XML parsing, which incorporates stable code from the Expat project: http://sourceforge.net/projects/expat/ diff -r 69b354a8c90f -r 78fb1181787a setup.py --- a/setup.py Tue Aug 09 18:09:21 2011 +0200 +++ b/setup.py Tue Aug 09 20:28:59 2011 +0200 @@ -525,6 +525,8 @@ # Python C API test module exts.append( Extension('_testcapi', ['_testcapimodule.c'], depends=['testcapi_long.h']) ) + # Python PEP-3118 (buffer protocol) test module + exts.append( Extension('_testbuffer', ['_testbuffer.c']) ) # profiler (_lsprof is for cProfile.py) exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) ) # static Unicode character database