diff -r 92842e347d98 -r 51cedae5acfc Doc/c-api/buffer.rst --- a/Doc/c-api/buffer.rst Thu Sep 08 00:56:17 2011 +0200 +++ b/Doc/c-api/buffer.rst Thu Sep 08 15:02:41 2011 +0200 @@ -7,6 +7,7 @@ .. sectionauthor:: Greg Stein .. sectionauthor:: Benjamin Peterson +.. sectionauthor:: Stefan Krah .. index:: @@ -20,7 +21,7 @@ While each of these types have their own semantics, they share the common characteristic of being backed by a possibly large memory buffer. It is -then desireable, in some situations, to access that buffer directly and +then desirable, in some situations, to access that buffer directly and without intermediate copying. Python provides such a facility at the C level in the form of the *buffer @@ -60,8 +61,10 @@ resource leaks. -The buffer structure -==================== +.. _buffer-structure: + +Buffer structure +================ Buffer structures (or simply "buffers") are useful as a way to expose the binary data from another object to the Python programmer. They can also be @@ -81,93 +84,326 @@ .. c:type:: Py_buffer - .. c:member:: void *buf + .. c:member:: void \*obj - A pointer to the start of the memory for the object. + A new reference to the exporting object or *NULL*. The reference is owned + by the consumer and automatically decremented and set to *NULL* by + :c:func:`PyBuffer_Release`. + + For temporary buffers that are wrapped by :c:func:`PyMemoryView_FromBuffer` + this field must be *NULL*. + + .. c:member:: void \*buf + + A pointer to the start of the logical structure described by the buffer + fields. This can be any location within the underlying physical memory + block of the exporter. For example, with negative :c:member:`~Py_buffer.strides` + the value may point to the end of the memory block. + + For contiguous arrays, the value points to the beginning of the memory + block. .. c:member:: Py_ssize_t len - :noindex: - The total length of the memory in bytes. + ``product(shape) * itemsize``. For contiguous arrays, this is the length + of the underlying memory block. For non-contiguous arrays, it is the length + that the logical structure would have if it were copied to a contiguous + representation. + + Accessing ``((char *)buf)[0] up to ((char *)buf)[len-1]`` is only valid + if the buffer has been obtained by a request that guarantees contiguity. In + most cases such a request will be :c:macro:`PyBUF_SIMPLE` or :c:macro:`PyBUF_WRITABLE`. .. c:member:: int readonly - An indicator of whether the buffer is read only. + An indicator of whether the buffer is read-only. This field is controlled + by the :c:macro:`PyBUF_WRITABLE` flag. - .. c:member:: const char *format - :noindex: + .. c:member:: Py_ssize_t itemsize - A *NULL* terminated string in :mod:`struct` module style syntax giving - the contents of the elements available through the buffer. If this is - *NULL*, ``"B"`` (unsigned bytes) is assumed. + Item size in bytes of a single element. Same as the value of :func:`struct.calcsize` + called on non-NULL :c:member:`~Py_buffer.format` values. + + Important exception: If a consumer requests a buffer without the + :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_Buffer.format` will + be set to *NULL*, but :c:member:`~Py_buffer.itemsize` still has + the value for the original format. + + If :c:member:`~Py_Buffer.shape` is present, the equality + ``product(shape) * itemsize = len`` still holds and the consumer + can use :c:member:`~Py_buffer.itemsize` to navigate the buffer. + + If :c:member:`~Py_Buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE` + or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard + :c:member:`~Py_buffer.itemsize` and assume ``itemsize = 1``. + + .. c:member:: const char \*format + + A *NUL* terminated string in :mod:`struct` module style syntax describing + the contents of a single item. If this is *NULL*, ``"B"`` (unsigned bytes) + is assumed. + + This field is controlled by the :c:macro:`PyBUF_FORMAT` flag. .. c:member:: int ndim - The number of dimensions the memory represents as a multi-dimensional - array. If it is 0, :c:data:`strides` and :c:data:`suboffsets` must be - *NULL*. + The number of dimensions the memory represents as an n-dimensional array. + If it is 0, :c:member:`~Py_Buffer.buf` points to a single item representing + a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides` + and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*. - .. c:member:: Py_ssize_t *shape + .. c:member:: Py_ssize_t \*shape - An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the - shape of the memory as a multi-dimensional array. Note that - ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to - :c:data:`len`. + An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` + indicating the shape of the memory as an n-dimensional array. Note that + ``shape[0] * ... * shape[ndim-1] * itemsize`` MUST be equal to + :c:member:`~Py_buffer.len`. - .. c:member:: Py_ssize_t *strides + Shape values are restricted to ``shape[n] >= 0``. The case + ``shape[n] = 0`` requires special attention. See `complex arrays`_ + for further information. - An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the - number of bytes to skip to get to a new element in each dimension. + The shape array is read-only for the consumer. - .. c:member:: Py_ssize_t *suboffsets + .. c:member:: Py_ssize_t \*strides - An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim`. If these - suboffset numbers are greater than or equal to 0, then the value stored - along the indicated dimension is a pointer and the suboffset value - dictates how many bytes to add to the pointer after de-referencing. A - suboffset value that it negative indicates that no de-referencing should - occur (striding in a contiguous memory block). + An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim` + giving the number of bytes to skip to get to a new element in each + dimension. - Here is a function that returns a pointer to the element in an N-D array - pointed to by an N-dimensional index when there are both non-NULL strides - and suboffsets:: + Stride values can be any integer. For regular arrays, strides are + usually positive, but a consumer MUST be able to handle the case + ``strides[n] <= 0``. See `complex arrays`_ for further information. - void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, - Py_ssize_t *suboffsets, Py_ssize_t *indices) { - char *pointer = (char*)buf; - int i; - for (i = 0; i < ndim; i++) { - pointer += strides[i] * indices[i]; - if (suboffsets[i] >=0 ) { - pointer = *((char**)pointer) + suboffsets[i]; - } - } - return (void*)pointer; - } + The strides array is read-only for the consumer. + .. c:member:: Py_ssize_t \*suboffsets - .. c:member:: Py_ssize_t itemsize + An array of :c:type:`Py_ssize_t` of length :c:member:`~Py_buffer.ndim`. + If ``suboffsets[n] >= 0``, the values stored along the nth dimension are + pointers and the suboffset value dictates how many bytes to add to each + pointer after de-referencing. A suboffset value that is negative + indicates that no de-referencing should occur (striding in a contiguous + memory block). - This is a storage for the itemsize (in bytes) of each element of the - shared memory. It is technically un-necessary as it can be obtained - using :c:func:`PyBuffer_SizeFromFormat`, however an exporter may know - this information without parsing the format string and it is necessary - to know the itemsize for proper interpretation of striding. Therefore, - storing it is more convenient and faster. + This type of array representation is used by the Python Imaging Library + (PIL). See `complex arrays`_ for further information how to access elements + of such an array. - .. c:member:: void *internal + The suboffsets array is read-only for the consumer. + + .. c:member:: void \*internal This is for use internally by the exporting object. For example, this might be re-cast as an integer by the exporter and used to store flags about whether or not the shape, strides, and suboffsets arrays must be - freed when the buffer is released. The consumer should never alter this + freed when the buffer is released. The consumer MUST NOT alter this value. +.. _buffer-request-types: + +Buffer request types +==================== + +Buffers are usually obtained by sending a buffer request to an exporting +object via :c:func:`PyObject_GetBuffer`. Since the complexity of the logical +structure of the memory can vary drastically, the consumer uses the *flags* +argument to specify the exact buffer type it can handle. + +All :c:data:`Py_buffer` fields are unambiguously defined by the request +type. + +request-independent fields +~~~~~~~~~~~~~~~~~~~~~~~~~~ +The following fields are not influenced by *flags* and must always be filled in +with the correct values: :c:member:`~Py_buffer.obj`, :c:member:`~Py_buffer.buf`, +:c:member:`~Py_buffer.len`, :c:member:`~Py_buffer.itemsize`, :c:member:`~Py_buffer.ndim`. + + +readonly, format +~~~~~~~~~~~~~~~~ + + .. c:macro:: PyBUF_WRITABLE + + Controls the :c:member:`~Py_buffer.readonly` field. If set, the exporter + MUST return a writable buffer or failure. Otherwise, the exporter MAY + return a readable or writable buffer, but the choice MUST be consistent + for all consumers. + + .. c:macro:: PyBUF_FORMAT + + Controls the :c:member:`~Py_buffer.format` field. If set, this field MUST + be filled in correctly. Otherwise, this field MUST be *NULL*. + + +:c:macro:`PyBUF_WRITABLE` can be \|'d to any of the flags in the next section. +Since :c:macro:`PyBUF_SIMPLE` is defined as 0, :c:macro:`PyBUF_WRITABLE` +can be used as a stand-alone flag to request a simple writable buffer. + +:c:macro:`PyBUF_FORMAT` can be \|'d to any of the flags except :c:macro:`PyBUF_SIMPLE`. +The latter already implies format ``B`` (unsigned bytes). + + +shape, strides, suboffsets +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The flags that control the logical structure of the memory are listed +in decreasing order of complexity. Note that each flag contains all bits +of the flags below it. + + ++-----------------------------+-------+---------+------------+ +| Request | shape | strides | suboffsets | ++=============================+=======+=========+============+ +| .. c:macro:: PyBUF_INDIRECT | yes | yes | if needed | ++-----------------------------+-------+---------+------------+ +| .. c:macro:: PyBUF_STRIDES | yes | yes | NULL | ++-----------------------------+-------+---------+------------+ +| .. c:macro:: PyBUF_ND | yes | NULL | NULL | ++-----------------------------+-------+---------+------------+ +| .. c:macro:: PyBUF_SIMPLE | NULL | NULL | NULL | ++-----------------------------+-------+---------+------------+ + + +contiguity requests +~~~~~~~~~~~~~~~~~~~ + +C or Fortran contiguity can be explicitly requested, with and without stride +information. Without stride information, the buffer must be C-contiguous. + ++-----------------------------------+-------+---------+------------+--------+ +| Request | shape | strides | suboffsets | contig | ++===================================+=======+=========+============+========+ +| .. c:macro:: PyBUF_C_CONTIGUOUS | yes | yes | NULL | C | ++-----------------------------------+-------+---------+------------+--------+ +| .. c:macro:: PyBUF_F_CONTIGUOUS | yes | yes | NULL | F | ++-----------------------------------+-------+---------+------------+--------+ +| .. c:macro:: PyBUF_ANY_CONTIGUOUS | yes | yes | NULL | C or F | ++-----------------------------------+-------+---------+------------+--------+ +| .. c:macro:: PyBUF_ND | yes | NULL | NULL | C | ++-----------------------------------+-------+---------+------------+--------+ + + +compound requests +~~~~~~~~~~~~~~~~~ + +All possible requests are fully defined by some combination of the flags in +the previous section. For convenience, the buffer protocol provides frequently +used combinations as single flags. + +In the following table *U* stands for undefined contiguity. The consumer would +have to call :c:func:`PyBuffer_IsContiguous` to determine contiguity. + + + ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| Request | shape | strides | suboffsets | contig | readonly | format | ++===============================+=======+=========+============+========+==========+========+ +| .. c:macro:: PyBUF_FULL | yes | yes | if needed | U | 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_FULL_RO | yes | yes | if needed | U | 1 or 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_RECORDS | yes | yes | NULL | U | 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_RECORDS_RO | yes | yes | NULL | U | 1 or 0 | yes | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_STRIDED | yes | yes | NULL | U | 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_STRIDED_RO | yes | yes | NULL | U | 1 or 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_CONTIG | yes | NULL | NULL | C | 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ +| .. c:macro:: PyBUF_CONTIG_RO | yes | NULL | NULL | C | 1 or 0 | NULL | ++-------------------------------+-------+---------+------------+--------+----------+--------+ + + +Complex arrays +============== + +NumPy-style: shape and strides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The logical structure of NumPy-style arrays is defined by :c:member:`~Py_buffer.itemsize`, +:c:member:`~Py_buffer.ndim`, :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides`. + +If ``ndim = 0``, the memory location pointed to by :c:member:`~Py_buffer.buf` is +interpreted as a scalar of size :c:member:`~Py_buffer.itemsize`. In that case, +both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are *NULL*. + +If :c:member:`~Py_buffer.strides` is *NULL*, the array is interpreted as +as standard n-dimensional C-array. Otherwise, the consumer must access an +n-dimensional array as follows: + + ``ptr = (char *)buf + indices[0] * strides[0] + ... + indices[n-1] * strides[n-1]`` + ``item = *((typeof(item) *)ptr);`` + + +As noted above, :c:member:`~Py_buffer.buf` can point to any location within +the actual memory block. An exporter can check the validity of a buffer with +this function: + +.. code-block:: python + + def verify_structure(memlen, itemsize, ndim, shape, strides, offset): + """Verify that the parameters represent a valid array within + the bounds of the allocated memory: + char *mem: start of the physical memory block + memlen: length of the physical memory block + offset: (char *)buf - mem + """ + if offset % itemsize: + return False + if offset < 0 or offset+itemsize > memlen: + return False + if any(v % itemsize for v in strides): + return False + + if ndim <= 0: + return ndim == 0 and not shape and not strides + if 0 in shape: + return True + + imin = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] <= 0) + imax = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] > 0) + + return 0 <= offset+imin and offset+imax+itemsize <= memlen + + +PIL-style: shape, strides and suboffsets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In addition to the regular items, PIL-style arrays can contain pointers +that must be followed in order to get to the next element in a dimension. +For example, the regular three-dimensional C-array ``char v[2][2][3]`` can +also be viewed as an array of 2 pointers to 2 two-dimensional arrays: +``char (*v[2])[2][3]``. In suboffsets representation, those two pointers +can be embedded at the start of :c:member:`~Py_buffer.buf`, pointing +to two ``char x[2][3]`` arrays that can be located anywhere in memory. + + +Here is a function that returns a pointer to the element in an N-D array +pointed to by an N-dimensional index when there are both non-NULL strides +and suboffsets:: + + void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, + Py_ssize_t *suboffsets, Py_ssize_t *indices) { + char *pointer = (char*)buf; + int i; + for (i = 0; i < ndim; i++) { + pointer += strides[i] * indices[i]; + if (suboffsets[i] >=0 ) { + pointer = *((char**)pointer) + suboffsets[i]; + } + } + return (void*)pointer; + } + Buffer-related functions ======================== - .. c:function:: int PyObject_CheckBuffer(PyObject *obj) Return 1 if *obj* supports the buffer interface otherwise 0. When 1 is @@ -175,133 +411,36 @@ succeed. -.. c:function:: int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) +.. c:function:: int PyObject_GetBuffer(PyObject *exporter, Py_buffer *view, int flags) - Export a view over some internal data from the target object *obj*. - *obj* must not be NULL, and *view* must point to an existing - :c:type:`Py_buffer` structure allocated by the caller (most uses of - this function will simply declare a local variable of type - :c:type:`Py_buffer`). The *flags* argument is a bit field indicating - what kind of buffer is requested. The buffer interface allows - for complicated memory layout possibilities; however, some callers - won't want to handle all the complexity and instead request a simple - view of the target object (using :c:macro:`PyBUF_SIMPLE` for a read-only - view and :c:macro:`PyBUF_WRITABLE` for a read-write view). + Send a request to *exporter* to fill in *view* as specified by *flags*. + If the exporter cannot provide a buffer of the exact type, it MUST raise + :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to *NULL* and + return -1. - Some exporters may not be able to share memory in every possible way and - may need to raise errors to signal to some consumers that something is - just not possible. These errors should be a :exc:`BufferError` unless - there is another error that is actually causing the problem. The - exporter can use flags information to simplify how much of the - :c:data:`Py_buffer` structure is filled in with non-default values and/or - raise an error if the object can't support a simpler view of its memory. + On success, fill in *view*, set :c:member:`view->obj` to a new reference + to *exporter* and return 0. - On success, 0 is returned and the *view* structure is filled with useful - values. On error, -1 is returned and an exception is raised; the *view* - is left in an undefined state. - - The following are the possible values to the *flags* arguments. - - .. c:macro:: PyBUF_SIMPLE - - This is the default flag. The returned buffer exposes a read-only - memory area. The format of data is assumed to be raw unsigned bytes, - without any particular structure. This is a "stand-alone" flag - constant. It never needs to be '|'d to the others. The exporter will - raise an error if it cannot provide such a contiguous buffer of bytes. - - .. c:macro:: PyBUF_WRITABLE - - Like :c:macro:`PyBUF_SIMPLE`, but the returned buffer is writable. If - the exporter doesn't support writable buffers, an error is raised. - - .. c:macro:: PyBUF_STRIDES - - This implies :c:macro:`PyBUF_ND`. The returned buffer must provide - strides information (i.e. the strides cannot be NULL). This would be - used when the consumer can handle strided, discontiguous arrays. - Handling strides automatically assumes you can handle shape. The - exporter can raise an error if a strided representation of the data is - not possible (i.e. without the suboffsets). - - .. c:macro:: PyBUF_ND - - The returned buffer must provide shape information. The memory will be - assumed C-style contiguous (last dimension varies the fastest). The - exporter may raise an error if it cannot provide this kind of - contiguous buffer. If this is not given then shape will be *NULL*. - - .. c:macro:: PyBUF_C_CONTIGUOUS - PyBUF_F_CONTIGUOUS - PyBUF_ANY_CONTIGUOUS - - These flags indicate that the contiguity returned buffer must be - respectively, C-contiguous (last dimension varies the fastest), Fortran - contiguous (first dimension varies the fastest) or either one. All of - these flags imply :c:macro:`PyBUF_STRIDES` and guarantee that the - strides buffer info structure will be filled in correctly. - - .. c:macro:: PyBUF_INDIRECT - - This flag indicates the returned buffer must have suboffsets - information (which can be NULL if no suboffsets are needed). This can - be used when the consumer can handle indirect array referencing implied - by these suboffsets. This implies :c:macro:`PyBUF_STRIDES`. - - .. c:macro:: PyBUF_FORMAT - - The returned buffer must have true format information if this flag is - provided. This would be used when the consumer is going to be checking - for what 'kind' of data is actually stored. An exporter should always - be able to provide this information if requested. If format is not - explicitly requested then the format must be returned as *NULL* (which - means ``'B'``, or unsigned bytes). - - .. c:macro:: PyBUF_STRIDED - - This is equivalent to ``(PyBUF_STRIDES | PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_STRIDED_RO - - This is equivalent to ``(PyBUF_STRIDES)``. - - .. c:macro:: PyBUF_RECORDS - - This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT | - PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_RECORDS_RO - - This is equivalent to ``(PyBUF_STRIDES | PyBUF_FORMAT)``. - - .. c:macro:: PyBUF_FULL - - This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT | - PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_FULL_RO - - This is equivalent to ``(PyBUF_INDIRECT | PyBUF_FORMAT)``. - - .. c:macro:: PyBUF_CONTIG - - This is equivalent to ``(PyBUF_ND | PyBUF_WRITABLE)``. - - .. c:macro:: PyBUF_CONTIG_RO - - This is equivalent to ``(PyBUF_ND)``. + Calls to :c:func:`PyObject_GetBuffer` and :c:func:`PyBuffer_Release` must + always be paired similar to to :c:func:`malloc` and :c:func:`free`. Thus, + after the consumer is done with the buffer, :c:func:`PyBuffer_Release` must + be called exactly once. .. c:function:: void PyBuffer_Release(Py_buffer *view) - Release the buffer *view*. This should be called when the buffer is no - longer being used as it may free memory from it. + Release the buffer *view* and decrement :c:member:`view->obj`. This function + MUST be called when the buffer is no longer being used, otherwise reference + leaks may occur. + + It is an error to call this function on a buffer that was not obtained via + :c:func:`PyObject_GetBuffer`. .. c:function:: Py_ssize_t PyBuffer_SizeFromFormat(const char *) - Return the implied :c:data:`~Py_buffer.itemsize` from the struct-stype - :c:data:`~Py_buffer.format`. + Return the implied :c:data:`~Py_buffer.itemsize` from :c:data:`~Py_buffer.format`. + This function is not yet implemented. .. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) @@ -318,9 +457,23 @@ given shape with the given number of bytes per element. -.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags) +.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *exporter, void *buf, Py_ssize_t len, int readonly, int flags) - Fill in a buffer-info structure, *view*, correctly for an exporter that can - only share a contiguous chunk of memory of "unsigned bytes" of the given - length. Return 0 on success and -1 (with raising an error) on error. + Handle buffer requests for an exporter that wants to expose *buf* of size *len* + with writability set according to *readonly*. *buf* is interpreted a a sequence + of unsigned bytes. + The *flags* argument indicates the request type. This function always fills in + *view* as specified by flags, unless *buf* has been designated as read-only + and :c:macro:`PyBUF_WRITABLE` is set in *flags*. + + On success, set :c:member:`view->obj` to a new reference to *exporter* and + return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set + :c:member:`view->obj` to *NULL* and return -1; + + If this function is used as part of a :ref:`getbufferproc `, + *exporter* MUST be set to the exporting object. Otherwise, *exporter* MUST + be NULL. + + + diff -r 92842e347d98 -r 51cedae5acfc Doc/c-api/memoryview.rst --- a/Doc/c-api/memoryview.rst Thu Sep 08 00:56:17 2011 +0200 +++ b/Doc/c-api/memoryview.rst Thu Sep 08 15:02:41 2011 +0200 @@ -17,16 +17,19 @@ Create a memoryview object from an object that provides the buffer interface. If *obj* supports writable buffer exports, the memoryview object will be - readable and writable, other it will be read-only. + read/write, otherwise it may be either read-only or read/write at the + discretion of the exporter. +.. c:function:: PyObject *PyMemoryView_FromMemory(char *mem, Py_ssize_t size, int flags) + + Create a memoryview object using *mem* as the underlying buffer. + *flags* can be one of :c:macro:`PyBUF_READ` or :c:macro:`PyBUF_WRITE`. .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view) Create a memoryview object wrapping the given buffer structure *view*. - The memoryview object then owns the buffer represented by *view*, which - means you shouldn't try to call :c:func:`PyBuffer_Release` yourself: it - will be done on deallocation of the memoryview object. - + For simple byte buffers, :c:func:`PyMemoryView_FromMemory` is the preferred + function. .. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) diff -r 92842e347d98 -r 51cedae5acfc Doc/c-api/typeobj.rst --- a/Doc/c-api/typeobj.rst Thu Sep 08 00:56:17 2011 +0200 +++ b/Doc/c-api/typeobj.rst Thu Sep 08 15:02:41 2011 +0200 @@ -1198,46 +1198,69 @@ .. sectionauthor:: Greg J. Stein .. sectionauthor:: Benjamin Peterson - - -The :ref:`buffer interface ` exports a model where an object can expose its internal -data. - -If an object does not export the buffer interface, then its :attr:`tp_as_buffer` -member in the :c:type:`PyTypeObject` structure should be *NULL*. Otherwise, the -:attr:`tp_as_buffer` will point to a :c:type:`PyBufferProcs` structure. - +.. sectionauthor:: Stefan Krah .. c:type:: PyBufferProcs - Structure used to hold the function pointers which define an implementation of - the buffer protocol. + This structure holds pointers to the functions required by the + :ref:`Buffer protocol `. The protocol defines how + an exporter object can expose its internal data to consumer objects. - .. c:member:: getbufferproc bf_getbuffer +.. c:member:: getbufferproc PyBufferProcs.bf_getbuffer - This should fill a :c:type:`Py_buffer` with the necessary data for - exporting the type. The signature of :data:`getbufferproc` is ``int - (PyObject *obj, Py_buffer *view, int flags)``. *obj* is the object to - export, *view* is the :c:type:`Py_buffer` struct to fill, and *flags* gives - the conditions the caller wants the memory under. (See - :c:func:`PyObject_GetBuffer` for all flags.) :c:member:`bf_getbuffer` is - responsible for filling *view* with the appropriate information. - (:c:func:`PyBuffer_FillView` can be used in simple cases.) See - :c:type:`Py_buffer`\s docs for what needs to be filled in. + The signature of this function is:: + int (PyObject *exporter, Py_buffer *view, int flags); - .. c:member:: releasebufferproc bf_releasebuffer + Handle a request to *exporter* to fill in *view* as specified by *flags*. + A standard implementation of this function will take these steps: - This should release the resources of the buffer. The signature of - :c:data:`releasebufferproc` is ``void (PyObject *obj, Py_buffer *view)``. - If the :c:data:`bf_releasebuffer` function is not provided (i.e. it is - *NULL*), then it does not ever need to be called. + - Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`, + set :c:data:`view->obj` to *NULL* and return -1. - The exporter of the buffer interface must make sure that any memory - pointed to in the :c:type:`Py_buffer` structure remains valid until - releasebuffer is called. Exporters will need to define a - :c:data:`bf_releasebuffer` function if they can re-allocate their memory, - strides, shape, suboffsets, or format variables which they might share - through the struct bufferinfo. + - Fill in the requested fields. - See :c:func:`PyBuffer_Release`. + - Increment an internal counter for the number of exports. + + - Set :c:data:`view->obj` to *exporter* and increment :c:data:`view->obj`. + + - Return 0. + + The individual fields of *view* are described in section + :ref:`Buffer structure `, the rules how an exporter + must react to specific requests are in section + :ref:`Buffer request types `. + + All memory pointed to in the :c:type:`Py_buffer` structure belongs to + the exporter and must remain valid until there are no consumers left. + :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`, + :c:member:`~Py_buffer.suboffsets` and :c:member:`~Py_buffer.internal` + are read-only for the consumer. + + :c:func:`PyBuffer_FillInfo` provides an easy way of exposing a simple + bytes buffer while dealing correctly with all request types. + + :c:func:`PyObject_GetBuffer` is the interface for the consumer that + wraps this function. + +.. c:member:: releasebufferproc PyBufferProcs.bf_releasebuffer + + The signature of this function is:: + + void (PyObject *exporter, Py_buffer *view); + + Handle a request to release the resources of the buffer. If no resources + need to be released, this field may be *NULL*. A standard implementation + of this function will take these steps: + + - Decrement an internal counter for the number of exports. + + - If the counter is 0, free all memory associated with *view*. + + + This function MUST NOT decrement :c:data:`view->obj`, since that is + done automatically in :c:func:`PyBuffer_Release`. + + + :c:func:`PyBuffer_Release` is the interface for the consumer that + wraps this function. diff -r 92842e347d98 -r 51cedae5acfc Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst Thu Sep 08 00:56:17 2011 +0200 +++ b/Doc/library/stdtypes.rst Thu Sep 08 15:02:41 2011 +0200 @@ -2339,13 +2339,13 @@ :class:`memoryview` objects allow Python code to access the internal data of an object that supports the :ref:`buffer protocol ` without -copying. Memory is generally interpreted as simple bytes. +copying. .. class:: memoryview(obj) Create a :class:`memoryview` that references *obj*. *obj* must support the buffer protocol. Built-in objects that support the buffer protocol include - :class:`bytes` and :class:`bytearray`. + :class:`bytes`, :class:`bytearray` and :class:`array.array`. A :class:`memoryview` has the notion of an *element*, which is the atomic memory unit handled by the originating object *obj*. For many @@ -2353,43 +2353,65 @@ is a single byte, but other types such as :class:`array.array` may have bigger elements. - ``len(view)`` returns the total number of elements in the memoryview, - *view*. The :class:`~memoryview.itemsize` attribute will give you the + ``len(view)`` is equal to the length of :class:`~memoryview.tolist`. + If ``view.ndim = 0``, the length is 1. If ``view.ndim = 1``, the length + is equal to the number of elements in the view. For higher dimensions, + the length is equal to the length of the nested list representation of + the view. The :class:`~memoryview.itemsize` attribute will give you the number of bytes in a single element. - A :class:`memoryview` supports slicing to expose its data. Taking a single - index will return a single element as a :class:`bytes` object. Full - slicing will result in a subview:: - - >>> v = memoryview(b'abcefg') - >>> v[1] - b'b' - >>> v[-1] - b'g' - >>> v[1:4] - - >>> bytes(v[1:4]) - b'bce' - - If the object the memoryview is over supports changing its data, the - memoryview supports slice assignment:: + A :class:`memoryview` supports slicing to expose its data. If + :class:`~memoryview.format` is one of the native format specifiers + from the :mod:`struct` module, index will return a single element + with the correct type. Full slicing will result in a subview:: + + >>> v = memoryview(b'abcefg') + >>> v[1] + 98 + >>> v[-1] + 103 + >>> v[1:4] + + >>> bytes(v[1:4]) + b'bce' + + Other native formats:: + + >>> import array + >>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444]) + >>> a[0] + -11111111 + >>> a[-1] + 44444444 + >>> a[2:3].tolist() + [-33333333] + >>> a[::2].tolist() + [-11111111, -33333333] + >>> a[::-1].tolist() + [44444444, -33333333, 22222222, -11111111] + + .. versionadded:: 3.3 + + If the underlying object is writable, the memoryview supports slice + assignment. Resizing is not allowed:: >>> data = bytearray(b'abcefg') >>> v = memoryview(data) >>> v.readonly False - >>> v[0] = b'z' + >>> v[0] = ord(b'z') >>> data bytearray(b'zbcefg') >>> v[1:4] = b'123' >>> data - bytearray(b'a123fg') - >>> v[2] = b'spam' + bytearray(b'z123fg') + >>> v[2:3] = b'spam' Traceback (most recent call last): File "", line 1, in - ValueError: cannot modify size of memoryview object - - Notice how the size of the memoryview object cannot be changed. + ValueError: memoryview assignment: lvalue and rvalue have different structures + >>> v[2:6] = b'spam' + >>> data + bytearray(b'z1spam') :class:`memoryview` has several methods: @@ -2404,12 +2426,21 @@ >>> bytes(m) b'abc' + For non-contiguous arrays the result is equal to the flattened list + representation with all elements converted to bytes. This is + currently not implemented. + .. method:: tolist() - Return the data in the buffer as a list of integers. :: + Return the data in the buffer as a list of elements. :: >>> memoryview(b'abc').tolist() [97, 98, 99] + >>> import array + >>> a = array.array('d', [1.1, 2.2, 3.3]) + >>> m = memoryview(a) + >>> m.tolist() + [1.1, 2.2, 3.3] .. method:: release() @@ -2436,7 +2467,7 @@ >>> with memoryview(b'abc') as m: ... m[0] ... - b'a' + 97 >>> m[0] Traceback (most recent call last): File "", line 1, in @@ -2444,45 +2475,195 @@ .. versionadded:: 3.2 + .. method:: cast() + + Cast a memoryview to a new format or shape. The return value is + a new memoryview, but the buffer itself is not copied. Supported + casts are 1D -> C-contiguous and C-contiguous -> 1D. One of the + formats must be a byte format ('B', 'b' or 'c'). The byte length + of the result must be the same as the original length. + + Cast 1D/long to 1D/unsigned bytes:: + + >>> import array + >>> a = array.array('l', [1,2,3]) + >>> m1 = memoryview(a) + >>> m1.format + 'l' + >>> m1.itemsize + 8 + >>> len(m1) + 3 + >>> m1.len + 24 + >>> m2 = m1.cast('B') + >>> m2.format + 'B' + >>> m2.itemsize + 1 + >>> len(m2) + 24 + >>> m2.len + 24 + + Cast 1D/unsigned bytes to 1D/char:: + + >>> b = bytearray(b'zyz') + >>> m1 = memoryview(b) + >>> m1[0] = b'a' + Traceback (most recent call last): + File "", line 1, in + ValueError: memoryview: invalid value for format "B" + >>> m2 = m1.cast('c') + >>> m2[0] = b'a' + >>> b + bytearray(b'ayz') + + Cast 3D/ints to 1D/signed char:: + + >>> from _testbuffer import * + >>> nd = ndarray(list(range(12)), shape=[2,2,3], format='i') + >>> m1 = memoryview(nd) + >>> m1.format + 'i' + >>> m1.itemsize + 4 + >>> len(m1) + 2 + >>> m1.len + 48 + >>> m2 = m1.cast('b') + >>> m2.format + 'b' + >>> m2.itemsize + 1 + >>> len(m2) + 48 + >>> m2.len + 48 + + Cast 1D/unsigned char to to 2D/unsigned long:: + + >>> from _testbuffer import * + >>> import struct + >>> size = struct.calcsize('L') + >>> b = bytearray(list(range(2*3*size))) + >>> m1 = memoryview(b) + >>> m2 = m1.cast('L', shape=[2,3]) + >>> len(m2) + 2 + >>> m2.len + 48 + >>> nd = ndarray(m2) + >>> nd.tolist() + [[506097522914230528, 1084818905618843912, 1663540288323457296], [2242261671028070680, 2820983053732684064, 3399704436437297448]] + + .. versionadded:: 3.3 + There are also several readonly attributes available: + .. attribute:: obj + + The underlying object of the memoryview:: + + >>> b = bytearray(b'xyz') + >>> m = memoryview(b) + >>> m.obj is b + True + + .. versionadded:: 3.3 + + .. attribute:: len + + product(shape) * itemsize. This is the amount of space in bytes that + the array would use in a contiguous representation. It is not neccesarily + equal to len(memoryview):: + + >>> import array + >>> a = array.array('i', [1,2,3,4,5]) + >>> x = memoryview(a) + >>> len(x) + 5 + >>> x.len + 20 + >>> y = x[::2] + >>> len(y) + 3 + + Multi-dimensional arrays:: + + >>> from _testbuffer import * + >>> nd = ndarray(list(range(12)), shape=[3,4], format='L') + >>> nd.tolist() + [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]] + >>> m = memoryview(nd) + >>> len(m) + 3 + >>> m.len + 96 + + .. attribute:: readonly + + A bool indicating whether the memory is read only. + .. attribute:: format A string containing the format (in :mod:`struct` module style) for each - element in the view. This defaults to ``'B'``, a simple bytestring. + element in the view. A memoryview can be created from exporters with + arbitrary format strings, but all methods only work with native + single element formats. .. attribute:: itemsize The size in bytes of each element of the memoryview:: - >>> m = memoryview(array.array('H', [1,2,3])) + >>> import struct + >>> m = memoryview(array.array('H', [32000, 32001, 32002])) >>> m.itemsize 2 >>> m[0] - b'\x01\x00' - >>> len(m[0]) == m.itemsize + 32000 + >>> struct.calcsize('H') == m.itemsize True + .. attribute:: ndim + + An integer indicating how many dimensions of a multi-dimensional array the + memory represents. + .. attribute:: shape A tuple of integers the length of :attr:`ndim` giving the shape of the memory as a N-dimensional array. - .. attribute:: ndim - - An integer indicating how many dimensions of a multi-dimensional array the - memory represents. - .. attribute:: strides A tuple of integers the length of :attr:`ndim` giving the size in bytes to access each element for each dimension of the array. - .. attribute:: readonly - - A bool indicating whether the memory is read only. - - .. memoryview.suboffsets isn't documented because it only seems useful for C + .. attribute:: suboffsets + + A value >= 0 indicates that elements in that dimension are pointers to + sub-arrays. Values > 0 are used for slicing. Suboffsets are only + relevant for PIL-style arrays. + + .. attribute:: c_contiguous + + A bool indicating whether the memory is C-contiguous. + + .. versionadded:: 3.3 + + .. attribute:: f_contiguous + + A bool indicating whether the memory is Fortran contiguous. + + .. versionadded:: 3.3 + + .. attribute:: contiguous + + A bool indicating whether the memory is contiguous. + + .. versionadded:: 3.3 .. _typecontextmanager: diff -r 92842e347d98 -r 51cedae5acfc Include/abstract.h --- a/Include/abstract.h Thu Sep 08 00:56:17 2011 +0200 +++ b/Include/abstract.h Thu Sep 08 15:02:41 2011 +0200 @@ -547,7 +547,7 @@ /* Copy the data from the src buffer to the buffer of destination */ - PyAPI_FUNC(int) PyBuffer_IsContiguous(Py_buffer *view, char fort); + PyAPI_FUNC(int) PyBuffer_IsContiguous(const Py_buffer *view, char fort); PyAPI_FUNC(void) PyBuffer_FillContiguousStrides(int ndims, diff -r 92842e347d98 -r 51cedae5acfc Include/memoryobject.h --- a/Include/memoryobject.h Thu Sep 08 00:56:17 2011 +0200 +++ b/Include/memoryobject.h Thu Sep 08 15:02:41 2011 +0200 @@ -6,8 +6,10 @@ extern "C" { #endif +PyAPI_DATA(PyTypeObject) PyManagedBuffer_Type; PyAPI_DATA(PyTypeObject) PyMemoryView_Type; +#define PyManagedBuffer_Check(op) (Py_TYPE(op) == &PyManagedBuffer_Type) #define PyMemoryView_Check(op) (Py_TYPE(op) == &PyMemoryView_Type) #ifndef Py_LIMITED_API @@ -57,18 +59,40 @@ #ifndef Py_LIMITED_API PyAPI_FUNC(PyObject *) PyMemoryView_FromBuffer(Py_buffer *info); - /* create new if bufptr is NULL - will be a new bytesobject in base */ +PyAPI_FUNC(PyObject *) PyMemoryView_FromMemory(char *mem, Py_ssize_t size, + int flags); #endif -/* The struct is declared here so that macros can work, but it shouldn't - be considered public. Don't access those fields directly, use the macros +/* The structs are declared here so that macros can work, but they shouldn't + be considered public. Don't access their fields directly, use the macros and functions instead! */ #ifndef Py_LIMITED_API typedef struct { PyObject_HEAD - Py_buffer view; + int released; + Py_ssize_t exports; /* number of direct memoryview exports */ + Py_buffer master; /* snapshot buffer obtained from the original exporter */ +} PyManagedBufferObject; + +/* static storage used for casting between formats */ +#define Py_MEMORYVIEW_MAX_FORMAT 3 /* must be >= 3 */ + +/* memoryview state flags */ +#define Py_MEMORYVIEW_RELEASED 0x001 /* access to master buffer blocked */ +#define Py_MEMORYVIEW_C 0x002 /* C-contiguous layout */ +#define Py_MEMORYVIEW_FORTRAN 0x004 /* Fortran contiguous layout */ +#define Py_MEMORYVIEW_SCALAR 0x008 /* scalar: ndim = 0 */ +#define Py_MEMORYVIEW_PIL 0x010 /* PIL-style layout */ + +typedef struct { + PyObject_VAR_HEAD + PyManagedBufferObject *mbuf; /* managed buffer */ + int flags; /* state flags */ + Py_ssize_t exports; /* number of buffer re-exports */ + Py_buffer view; /* private copy of the exporter's view */ + char format[3]; /* used for casting */ + Py_ssize_t ob_array[1]; /* shape, strides, suboffsets */ } PyMemoryViewObject; #endif diff -r 92842e347d98 -r 51cedae5acfc Include/object.h --- a/Include/object.h Thu Sep 08 00:56:17 2011 +0200 +++ b/Include/object.h Thu Sep 08 15:02:41 2011 +0200 @@ -157,8 +157,6 @@ Py_ssize_t *shape; Py_ssize_t *strides; Py_ssize_t *suboffsets; - Py_ssize_t smalltable[2]; /* static store for shape and strides of - mono-dimensional buffers. */ void *internal; } Py_buffer; diff -r 92842e347d98 -r 51cedae5acfc Lib/ctypes/test/test_pep3118.py --- a/Lib/ctypes/test/test_pep3118.py Thu Sep 08 00:56:17 2011 +0200 +++ b/Lib/ctypes/test/test_pep3118.py Thu Sep 08 15:02:41 2011 +0200 @@ -25,14 +25,17 @@ v = memoryview(ob) try: self.assertEqual(normalize(v.format), normalize(fmt)) - if shape is not None: + if shape: self.assertEqual(len(v), shape[0]) else: self.assertEqual(len(v) * sizeof(itemtp), sizeof(ob)) self.assertEqual(v.itemsize, sizeof(itemtp)) self.assertEqual(v.shape, shape) - # ctypes object always have a non-strided memory block - self.assertEqual(v.strides, None) + # XXX Issue #12851: PyCData_NewGetBuffer() must provide strides + # if requested. memoryview currently reconstructs missing + # stride information, so this assert will fail. + # self.assertEqual(v.strides, ()) + # they are always read/write self.assertFalse(v.readonly) @@ -52,14 +55,15 @@ v = memoryview(ob) try: self.assertEqual(v.format, fmt) - if shape is not None: + if shape: self.assertEqual(len(v), shape[0]) else: self.assertEqual(len(v) * sizeof(itemtp), sizeof(ob)) self.assertEqual(v.itemsize, sizeof(itemtp)) self.assertEqual(v.shape, shape) - # ctypes object always have a non-strided memory block - self.assertEqual(v.strides, None) + # XXX Issue #12851 + # self.assertEqual(v.strides, ()) + # they are always read/write self.assertFalse(v.readonly) @@ -110,34 +114,34 @@ ## simple types - (c_char, "l:x:>l:y:}", None, BEPoint), - (LEPoint, "T{l:x:>l:y:}", None, POINTER(BEPoint)), - (POINTER(LEPoint), "&T{l:x:>l:y:}", (), BEPoint), + (LEPoint, "T{l:x:>l:y:}", (), POINTER(BEPoint)), + (POINTER(LEPoint), "&T{':STANDARD, + '=':STANDARD, + '!':STANDARD +} + +if struct: + for fmt in fmtdict['@']: + fmtdict['@'][fmt] = native_type_range(fmt) + +MEMORYVIEW = NATIVE.copy() +ARRAY = NATIVE.copy() +for k in NATIVE: + if not k in "bBhHiIlLfd": + del ARRAY[k] + +if sys.float_repr_style != 'short': + # memoryview: 'f', 'd' only supported for 'short' repr + del MEMORYVIEW['f'] + del MEMORYVIEW['d'] + del ARRAY['f'] + del ARRAY['d'] + +BYTEFMT = NATIVE.copy() +for k in NATIVE: + if not k in "Bbc": + del BYTEFMT[k] + +fmtdict['m'] = MEMORYVIEW +fmtdict['@m'] = MEMORYVIEW +fmtdict['a'] = ARRAY +fmtdict['b'] = BYTEFMT +fmtdict['@b'] = BYTEFMT + +# Capabilities of the test objects: +MODE = 0 +MULT = 1 +cap = { # format chars # multiplier + 'ndarray': (['', '@', '<', '>', '=', '!'], ['', '1', '2', '3']), + 'array': (['a'], ['']), + 'numpy': ([''], ['']), + 'memoryview': (['@m', 'm'], ['']), + 'bytefmt': (['@b', 'b'], ['']), +} + +def randrange_fmt(mode, char, obj): + """Return random item for a type specified by a mode and a single + format character.""" + x = randrange(*fmtdict[mode][char]) + if char == 'c': + x = bytes(chr(x), 'latin1') + if char == '?': + x = bool(x) + if char == 'f' or char == 'd': + x = struct.pack(char, x) + x = struct.unpack(char, x)[0] + if obj == 'numpy' and x == b'\x00': + # http://projects.scipy.org/numpy/ticket/1925 + x = b'\x01' + return x + +def gen_item(fmt, obj): + """Return single random item.""" + mode, chars = fmt.split('#') + x = [] + for c in chars: + x.append(randrange_fmt(mode, c, obj)) + return x[0] if len(x) == 1 else tuple(x) + +def gen_items(n, fmt, obj): + """Return a list of random items (or a scalar).""" + if n == 0: + return gen_item(fmt, obj) + lst = [0] * n + for i in range(n): + lst[i] = gen_item(fmt, obj) + return lst + +def struct_items(n, obj): + mode = choice(cap[obj][MODE]) + xfmt = mode + '#' + fmt = mode.strip('amb') + nmemb = randrange(2, 10) # number of struct members + for _ in range(nmemb): + char = choice(tuple(fmtdict[mode])) + multiplier = choice(cap[obj][MULT]) + xfmt += (char * int(multiplier if multiplier else 1)) + fmt += (multiplier + char) + items = gen_items(n, xfmt, obj) + item = gen_item(xfmt, obj) + return fmt, items, item + +def randitems(n, obj='ndarray', mode=None, char=None): + """Return random format, items, item.""" + if mode is None: + mode = choice(cap[obj][MODE]) + if char is None: + char = choice(tuple(fmtdict[mode])) + multiplier = choice(cap[obj][MULT]) + fmt = mode + '#' + char * int(multiplier if multiplier else 1) + items = gen_items(n, fmt, obj) + item = gen_item(fmt, obj) + fmt = mode.strip('amb') + multiplier + char + return fmt, items, item + +def iter_mode(n, obj='ndarray'): + """Iterate through supported mode/char combinations.""" + for mode in cap[obj][MODE]: + for char in fmtdict[mode]: + yield randitems(n, obj, mode, char) + +def iter_format(nitems, testobj='ndarray'): + """Yield (format, items, item) for all possible modes and format + characters plus one random compound format string.""" + for t in iter_mode(nitems, testobj): + yield t + if testobj != 'ndarray': + raise StopIteration + for t in iter_mode(nitems, testobj): + yield t + yield struct_items(nitems, testobj) + + +def is_byte_format(fmt): + return 'c' in fmt or 'b' in fmt or 'B' in fmt + +def is_memoryview_format(fmt): + """format suitable for memoryview""" + x = len(fmt) + return ((x == 1 or (x == 2 and fmt[0] == '@')) and + fmt[x-1] in MEMORYVIEW) + +NON_BYTE_FORMAT = [c for c in fmtdict['@'] if not is_byte_format(c)] + + +# ====================================================================== +# Multi-dimensional tolist(), slicing and slice assignments +# ====================================================================== + +def atomp(lst): + """Tuple items (representing structs) are regarded as atoms.""" + return not isinstance(lst, list) + +def listp(lst): + return isinstance(lst, list) + +def prod(lst): + """Product of list elements.""" + if len(lst) == 0: + return 0 + x = lst[0] + for v in lst[1:]: + x *= v + return x + +def strides_from_shape(ndim, shape, itemsize, layout): + """Calculate strides of a contiguous array. Layout is 'C' or + 'F' (Fortran).""" + if ndim == 0: + return () + if layout == 'C': + strides = list(shape[1:]) + [itemsize] + for i in range(ndim-2, -1, -1): + strides[i] *= strides[i+1] + else: + strides = [itemsize] + list(shape[:-1]) + for i in range(1, ndim): + strides[i] *= strides[i-1] + return strides + +def _ca(items, s): + """Convert flat item list to the nested list representation of a + multidimensional C array with shape 's'.""" + if atomp(items): + return items + if len(s) == 0: + return items[0] + lst = [0] * s[0] + stride = len(items) // s[0] if s[0] else 0 + for i in range(s[0]): + start = i*stride + lst[i] = _ca(items[start:start+stride], s[1:]) + return lst + +def _fa(items, s): + """Convert flat item list to the nested list representation of a + multidimensional Fortran array with shape 's'.""" + if atomp(items): + return items + if len(s) == 0: + return items[0] + lst = [0] * s[0] + stride = s[0] + for i in range(s[0]): + lst[i] = _fa(items[i::stride], s[1:]) + return lst + +def carray(items, shape): + if listp(items) and not 0 in shape and prod(shape) != len(items): + raise ValueError("prod(shape) != len(items)") + return _ca(items, shape) + +def farray(items, shape): + if listp(items) and not 0 in shape and prod(shape) != len(items): + raise ValueError("prod(shape) != len(items)") + return _fa(items, shape) + +def slice_shape(lst, slices): + """Get the shape of lst after slicing: slices is a list of slice + objects.""" + if atomp(lst): + return [] + return [len(lst[slices[0]])] + slice_shape(lst[0], slices[1:]) + +def multislice(lst, slices): + """Multi-dimensional slicing: slices is a list of slice objects.""" + if atomp(lst): + return lst + return [multislice(sublst, slices[1:]) for sublst in lst[slices[0]]] + +def m_assign(llst, rlst, lslices, rslices): + """Multi-dimensional slice assignment: llst and rlst are the operands, + lslices and rslices are lists of slice objects. llst and rlst must + have the same structure. + + For a two-dimensional example, this is not implemented in Python: + + llst[0:3:2, 0:3:2] = rlst[1:3:1, 1:3:1] + + Instead we write: + + lslices = [slice(0,3,2), slice(0,3,2)] + rslices = [slice(1,3,1), slice(1,3,1)] + multislice_assign(llst, rlst, lslices, rslices) + """ + if atomp(rlst): + return rlst + rlst = [m_assign(l, r, lslices[1:], rslices[1:]) + for l, r in zip(llst[lslices[0]], rlst[rslices[0]])] + llst[lslices[0]] = rlst + return llst + +def cmp_structure(llst, rlst, lslices, rslices): + """Compare the structure of llst[lslices] and rlst[rslices].""" + lshape = slice_shape(llst, lslices) + rshape = slice_shape(rlst, rslices) + if (len(lshape) != len(rshape)): + return -1 + for i in range(len(lshape)): + if lshape[i] != rshape[i]: + return -1 + if lshape[i] == 0: + return 0 + return 0 + +def multislice_assign(llst, rlst, lslices, rslices): + """Return llst after assigning: llst[lslices] = rlst[rslices]""" + if cmp_structure(llst, rlst, lslices, rslices) < 0: + raise ValueError("lvalue and rvalue have different structures") + return m_assign(llst, rlst, lslices, rslices) + + +# ====================================================================== +# Random structures +# ====================================================================== + +# +# PEP-3118 is very permissive with respect to the contents of a +# Py_buffer. In particular: +# +# - shape can be zero +# - strides can be any integer, including zero +# - offset can point to any location in the underlying +# memory block, provided that it is a multiple of +# itemsize. +# +# The functions in this section test and verify random structures +# in full generality. A structure is valid iff it fits in the +# underlying memory block. +# +# The structure 't' (short for 'tuple') is fully defined by: +# +# t = (memlen, itemsize, ndim, shape, strides, offset) +# + +def verify_structure(memlen, itemsize, ndim, shape, strides, offset): + """Verify that the parameters represent a valid array within + the bounds of the allocated memory: + char *mem: start of the physical memory block + memlen: length of the physical memory block + offset: (char *)buf - mem + """ + if offset % itemsize: + return False + if offset < 0 or offset+itemsize > memlen: + return False + if any(v % itemsize for v in strides): + return False + + if ndim <= 0: + return ndim == 0 and not shape and not strides + if 0 in shape: + return True + + imin = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] <= 0) + imax = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] > 0) + + return 0 <= offset+imin and offset+imax+itemsize <= memlen + +def indices(shape): + """Generate all possible tuples of indices.""" + iterables = [range(v) for v in shape] + return product(*iterables) + +def get_item(lst, indices): + for i in indices: + lst = lst[i] + return lst + +def memory_index(indices, t): + """Location of an item in the underlying memory.""" + memlen, itemsize, ndim, shape, strides, offset = t + p = offset + for i in range(ndim): + p += strides[i]*indices[i] + return p + +def is_overlapping(t): + """The structure 't' is overlapping if at least one memory location + is visited twice while iterating through all possible tuples of + indices.""" + memlen, itemsize, ndim, shape, strides, offset = t + visited = 1<= 95 and valid: + minshape = 0 + elif n >= 90: + minshape = 1 + shape = [0] * ndim + + for i in range(ndim): + shape[i] = randrange(minshape, maxshape+1) + else: + ndim = len(shape) + + maxstride = 5 + n = randrange(100) + zero_stride = True if n >= 95 and n & 1 else False + + strides = [0] * ndim + strides[ndim-1] = itemsize * randrange(-maxstride, maxstride+1) + if not zero_stride and strides[ndim-1] == 0: + strides[ndim-1] = itemsize + + for i in range(ndim-2, -1, -1): + maxstride *= shape[i+1] if shape[i+1] else 1 + if zero_stride: + strides[i] = itemsize * randrange(-maxstride, maxstride+1) + else: + strides[i] = ((1,-1)[randrange(2)] * + itemsize * randrange(1, maxstride+1)) + + imin = imax = 0 + if not 0 in shape: + imin = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] <= 0) + imax = sum(strides[j]*(shape[j]-1) for j in range(ndim) + if strides[j] > 0) + + nitems = imax - imin + if valid: + offset = -imin * itemsize + memlen = offset + (imax+1) * itemsize + else: + memlen = (-imin + imax) * itemsize + offset = -imin-itemsize if randrange(2) == 0 else memlen + return memlen, itemsize, ndim, shape, strides, offset + +def randslice_from_slicelen(slicelen, listlen): + """Create a random slice of len slicelen that fits into listlen.""" + maxstart = listlen - slicelen + start = randrange(maxstart+1) + maxstep = (listlen - start) // slicelen if slicelen else 1 + step = randrange(1, maxstep+1) + stop = start + slicelen * step + s = slice(start, stop, step) + _, _, _, control = slice_indices(s, listlen) + if control != slicelen: + raise RuntimeError + return s + +def randslice_from_shape(ndim, shape): + """Create two sets of slices for an array x with shape 'shape' + such that shapeof(x[lslices]) == shapeof(x[rslices]).""" + lslices = [0] * ndim + rslices = [0] * ndim + for n in range(ndim): + l = shape[n] + slicelen = randrange(1, l+1) if l > 0 else 0 + lslices[n] = randslice_from_slicelen(slicelen, l) + rslices[n] = randslice_from_slicelen(slicelen, l) + return tuple(lslices), tuple(rslices) + +def rand_aligned_slices(maxdim=5, maxshape=16): + """Create (lshape, rshape, tuple(lslices), tuple(rslices)) such that + shapeof(x[lslices]) == shapeof(y[rslices]), where x is an array + with shape 'lshape' and y is an array with shape 'rshape'.""" + ndim = randrange(1, maxdim+1) + minshape = 2 + n = randrange(100) + if n >= 95: + minshape = 0 + elif n >= 90: + minshape = 1 + all_random = True if randrange(100) >= 80 else False + lshape = [0]*ndim; rshape = [0]*ndim + lslices = [0]*ndim; rslices = [0]*ndim + + for n in range(ndim): + small = randrange(minshape, maxshape+1) + big = randrange(minshape, maxshape+1) + if big < small: + big, small = small, big + + # Create a slice that fits the smaller value. + if all_random: + start = randrange(-small, small+1) + stop = randrange(-small, small+1) + step = (1,-1)[randrange(2)] * randrange(1, small+2) + s_small = slice(start, stop, step) + _, _, _, slicelen = slice_indices(s_small, small) + else: + slicelen = randrange(1, small+1) if small > 0 else 0 + s_small = randslice_from_slicelen(slicelen, small) + + # Create a slice of the same length for the bigger value. + s_big = randslice_from_slicelen(slicelen, big) + if randrange(2) == 0: + rshape[n], lshape[n] = big, small + rslices[n], lslices[n] = s_big, s_small + else: + rshape[n], lshape[n] = small, big + rslices[n], lslices[n] = s_small, s_big + + return lshape, rshape, tuple(lslices), tuple(rslices) + +def randitems_from_structure(fmt, t): + """Return a list of random items for structure 't' with format + 'fmtchar'.""" + memlen, itemsize, _, _, _, _ = t + return gen_items(memlen//itemsize, '#'+fmt, 'numpy') + +def ndarray_from_structure(items, fmt, t, flags=0): + """Return ndarray from the tuple returned by rand_structure()""" + memlen, itemsize, ndim, shape, strides, offset = t + return ndarray(items, shape=shape, strides=strides, format=fmt, + offset=offset, flags=ND_WRITABLE|flags) + +def numpy_array_from_structure(items, fmt, t): + """Return numpy_array from the tuple returned by rand_structure()""" + memlen, itemsize, ndim, shape, strides, offset = t + buf = bytearray(memlen) + for j, v in enumerate(items): + struct.pack_into(fmt, buf, j*itemsize, v) + return numpy_array(buffer=buf, shape=shape, strides=strides, + dtype=fmt, offset=offset) + + +# ====================================================================== +# memoryview casts +# ====================================================================== + +def cast_items(exporter, fmt, itemsize, shape=None): + """Interpret the raw memory of 'exporter' as a list of items with + size 'itemsize'. If shape=None, the new structure is assumed to + be 1-D with n * itemsize = bytelen. If shape is given, the usual + constraint for contiguous arrays prod(shape) * itemsize = bytelen + applies. On success, return (items, shape). If the constraints + cannot be met, return (None, None). If a chunk of bytes is interpreted + as NaN as a result of float conversion, return ('nan', None).""" + bytelen = exporter.len + if shape: + if prod(shape) * itemsize != bytelen: + return None, shape + elif shape == []: + if exporter.ndim == 0 or itemsize != bytelen: + return None, shape + else: + n, r = divmod(bytelen, itemsize) + shape = [n] + if r != 0: + return None, shape + + mem = exporter.tobytes() + byteitems = [mem[i:i+itemsize] for i in range(0, len(mem), itemsize)] + + items = [] + for v in byteitems: + item = struct.unpack(fmt, v)[0] + if item != item: + return 'nan', shape + items.append(item) + + return (items, shape) if shape != [] else (items[0], shape) + +def gencastshapes(): + """Generate shapes to test casting.""" + for n in range(32): + yield [n] + ndim = randrange(4, 6) + minshape = 1 if randrange(100) > 80 else 2 + yield [randrange(minshape, 5) for _ in range(ndim)] + ndim = randrange(2, 4) + minshape = 1 if randrange(100) > 80 else 2 + yield [randrange(minshape, 5) for _ in range(ndim)] + + +# ====================================================================== +# Actual tests +# ====================================================================== + +def genslices(n): + """Generate all possible slices for a single dimension.""" + return product(range(-n, n+1), range(-n, n+1), range(-n, n+1)) + +def genslices_ndim(ndim, shape): + """Generate all possible slice tuples for 'shape'.""" + iterables = [genslices(shape[n]) for n in range(ndim)] + return product(*iterables) + +def rslice(n, allow_empty=False): + """Generate random slice for a single dimension of length n. + If zero=True, the slices may be empty, otherwise they will + be non-empty.""" + minlen = 0 if allow_empty or n == 0 else 1 + slicelen = randrange(minlen, n+1) + return randslice_from_slicelen(slicelen, n) + +def rslices(n, allow_empty=False): + """Generate random slices for a single dimension.""" + for _ in range(5): + yield rslice(n, allow_empty) + +def rslices_ndim(ndim, shape, iterations=5): + """Generate random slice tuples for 'shape'.""" + # non-empty slices + for _ in range(iterations): + yield tuple(rslice(shape[n]) for n in range(ndim)) + # possibly empty slices + for _ in range(iterations): + yield tuple(rslice(shape[n], allow_empty=True) for n in range(ndim)) + # invalid slices + yield tuple(slice(0,1,0) for _ in range(ndim)) + +def rpermutation(iterable, r=None): + pool = tuple(iterable) + r = len(pool) if r is None else r + yield tuple(sample(pool, r)) + +def ndarray_print(nd): + """Print ndarray for debugging.""" + try: + x = nd.tolist() + except (TypeError, NotImplementedError): + x = nd.tobytes() + if isinstance(nd, ndarray): + offset = nd.offset + flags = nd.flags + else: + offset = 'unknown' + flags = 'unknown' + print("ndarray(%s, shape=%s, strides=%s, suboffsets=%s, offset=%s, " + "format='%s', itemsize=%s, flags=%s)" % + (x, nd.shape, nd.strides, nd.suboffsets, offset, + nd.format, nd.itemsize, flags)) + sys.stdout.flush() + + +ITERATIONS = 100 +MAXDIM = 5 +MAXSHAPE = 10 + +if SHORT_TEST: + ITERATIONS = 10 + MAXDIM = 3 + MAXSHAPE = 4 + genslices = rslices + genslices_ndim = rslices_ndim + permutations = rpermutation + + +@unittest.skipUnless(struct, 'struct module required for this test.') +@unittest.skipUnless(ndarray, 'ndarray object required for this test') +class TestBufferProtocol(unittest.TestCase): + + def setUp(self): + self.sizeof_void_p = get_config_var('SIZEOF_VOID_P') + if not self.sizeof_void_p: + self.sizeof_void_p = 8 if architecture()[0] == '64bit' else 4 + + def verify(self, bufobj, obj=-1, + itemsize={1}, fmt=-1, readonly={1}, + ndim={1}, shape=-1, strides=-1, + lst=-1, sliced=False, cast=False): + # Verify buffer contents against expected values. Default values + # are deliberately initialized to invalid types. + if shape: + expected_len = prod(shape)*itemsize + else: + if not fmt: # array has been implicitly cast to unsigned bytes + expected_len = len(lst) + else: # ndim = 0 + expected_len = itemsize + + # Reconstruct suboffsets from strides. Support for slicing + # could be added, but is currently only needed for test_getbuf(). + suboffsets = () + if bufobj.suboffsets: + self.assertGreater(ndim, 0) + + suboffset0 = 0 + for n in range(1, ndim): + if shape[n] == 0: + break + if strides[n] <= 0: + suboffset0 += -strides[n] * (shape[n]-1) + + suboffsets = [suboffset0] + [-1 for v in range(ndim-1)] + + # Not correct if slicing has occurred in the first dimension. + stride0 = self.sizeof_void_p + if strides[0] < 0: + stride0 = -stride0 + strides = [stride0] + list(strides[1:]) + + self.assertIs(bufobj.obj, obj) + self.assertEqual(bufobj.len, expected_len) + self.assertEqual(bufobj.itemsize, itemsize) + self.assertEqual(bufobj.format, fmt) + self.assertEqual(bufobj.readonly, readonly) + self.assertEqual(bufobj.ndim, ndim) + self.assertEqual(bufobj.shape, tuple(shape)) + if not (sliced and suboffsets): + self.assertEqual(bufobj.strides, tuple(strides)) + self.assertEqual(bufobj.suboffsets, tuple(suboffsets)) + + if isinstance(bufobj, ndarray) or ndim <= 1: + rep = bufobj.tolist() if fmt else bufobj.tobytes() + self.assertEqual(rep, lst) + + if not fmt: # array has been cast to unsigned bytes, + return # the remaining tests won't work. + + # PyBuffer_GetPointer() is the definition how to access an item. + # If PyBuffer_GetPointer(indices) is correct for all possible + # combinations of indices, the buffer is correct. + # + # Also test tobytes() against the flattened 'lst', with all items + # packed to bytes. + if not cast: # casts chop up 'lst' in different ways + b = bytearray() + buf_err = None + for ind in indices(shape): + try: + item1 = get_pointer(bufobj, ind) + item2 = get_item(lst, ind) + if isinstance(item2, tuple): + x = struct.pack(fmt, *item2) + else: + x = struct.pack(fmt, item2) + b.extend(x) + except BufferError: + buf_err = True # re-exporter does not provide full buffer + break + self.assertEqual(item1, item2) + + if not buf_err and (isinstance(bufobj, ndarray) or + bufobj.c_contiguous): + # Issue #12834: memoryview tobytes() only works for + # C-contiguous arrays. + self.assertEqual(bufobj.tobytes(), b) + + if is_memoryview_format(fmt): + try: + m = memoryview(bufobj) + except BufferError: # re-exporter does not provide full information + return + ex = bufobj.obj if isinstance(bufobj, memoryview) else bufobj + self.assertIs(m.obj, ex) + self.assertEqual(m.len, expected_len) + self.assertEqual(m.itemsize, itemsize) + self.assertEqual(m.format, fmt) + self.assertEqual(m.readonly, readonly) + self.assertEqual(m.ndim, ndim) + self.assertEqual(m.shape, tuple(shape)) + if not (sliced and suboffsets): + self.assertEqual(m.strides, tuple(strides)) + self.assertEqual(m.suboffsets, tuple(suboffsets)) + + n = 1 if ndim == 0 else len(lst) + self.assertEqual(len(m), n) + + if ndim <= 1: + rep = bufobj.tolist() if fmt else bufobj.tobytes() + self.assertEqual(rep, lst) + self.assertEqual(m, bufobj) + + def verify_getbuf(self, orig_ex, ex, req, sliced=False): + def simple_fmt(ex): + return ex.format == '' or ex.format == 'B' + def match(req, flag): + return ((req&flag) == flag) + + if (# writable request to read-only exporter + (ex.readonly and match(req, PyBUF_WRITABLE)) or + # cannot match explicit contiguity request + (match(req, PyBUF_C_CONTIGUOUS) and not ex.c_contiguous) or + (match(req, PyBUF_F_CONTIGUOUS) and not ex.f_contiguous) or + (match(req, PyBUF_ANY_CONTIGUOUS) and not ex.contiguous) or + # buffer needs suboffsets + (not match(req, PyBUF_INDIRECT) and ex.suboffsets) or + # buffer without strides must be C-contiguous + (not match(req, PyBUF_STRIDES) and not ex.c_contiguous) or + # PyBUF_SIMPLE|PyBUF_FORMAT and PyBUF_WRITABLE|PyBUF_FORMAT + (not match(req, PyBUF_ND) and match(req, PyBUF_FORMAT))): + + self.assertRaises(BufferError, ndarray, ex, getbuf=req) + return + + if isinstance(ex, ndarray): + lst = ex.tolist() + else: + nd = ndarray(ex, getbuf=PyBUF_FULL_RO) + lst = nd.tolist() + + # The consumer may have requested default values or a NULL format. + ro = 0 if match(req, PyBUF_WRITABLE) else ex.readonly + fmt = ex.format + itemsize = ex.itemsize + ndim = ex.ndim + if not match(req, PyBUF_FORMAT): + # itemsize refers to the original itemsize before the cast. + # The equality product(shape) * itemsize = len still holds. + # The equality calcsize(format) = itemsize does _not_ hold. + fmt = '' + lst = orig_ex.tobytes() # Issue 12834 + if not match(req, PyBUF_ND): + ndim = 1 + shape = orig_ex.shape if match(req, PyBUF_ND) else () + strides = orig_ex.strides if match(req, PyBUF_STRIDES) else () + + nd = ndarray(ex, getbuf=req) + self.verify(nd, obj=ex, + itemsize=itemsize, fmt=fmt, readonly=ro, + ndim=ndim, shape=shape, strides=strides, + lst=lst, sliced=sliced) + + def test_ndarray_getbuf(self): + requests = ( + # distinct flags + PyBUF_INDIRECT, PyBUF_STRIDES, PyBUF_ND, PyBUF_SIMPLE, + PyBUF_C_CONTIGUOUS, PyBUF_F_CONTIGUOUS, PyBUF_ANY_CONTIGUOUS, + # compound requests + PyBUF_FULL, PyBUF_FULL_RO, + PyBUF_RECORDS, PyBUF_RECORDS_RO, + PyBUF_STRIDED, PyBUF_STRIDED_RO, + PyBUF_CONTIG, PyBUF_CONTIG_RO, + ) + # items and format + items_fmt = ( + ([True if x % 2 else False for x in range(12)], '?'), + ([1,2,3,4,5,6,7,8,9,10,11,12], 'b'), + ([1,2,3,4,5,6,7,8,9,10,11,12], 'B'), + ([(2**31-x) if x % 2 else (-2**31+x) for x in range(12)], 'l') + ) + # shape, strides, offset + structure = ( + ([], [], 0), + ([12], [], 0), + ([12], [-1], 11), + ([6], [2], 0), + ([6], [-2], 11), + ([3, 4], [], 0), + ([3, 4], [-4, -1], 11), + ([2, 2], [4, 1], 4), + ([2, 2], [-4, -1], 8) + ) + # ndarray creation flags + ndflags = ( + 0, ND_WRITABLE, ND_FORTRAN, ND_FORTRAN|ND_WRITABLE, + ND_PIL, ND_PIL|ND_WRITABLE + ) + # flags that can actually be used as flags + real_flags = (0, PyBUF_WRITABLE, PyBUF_FORMAT, + PyBUF_WRITABLE|PyBUF_FORMAT) + + for items, fmt in items_fmt: + itemsize = struct.calcsize(fmt) + for shape, strides, offset in structure: + strides = [v * itemsize for v in strides] + offset *= itemsize + for flags in ndflags: + + if strides and (flags&ND_FORTRAN): + continue + if not shape and (flags&ND_PIL): + continue + + _items = items if shape else items[0] + ex1 = ndarray(_items, format=fmt, flags=flags, + shape=shape, strides=strides, offset=offset) + ex2 = ex1[::-2] if shape else None + + m1 = memoryview(ex1) + if ex2: + m2 = memoryview(ex2) + if ex1.ndim == 0 or (ex1.ndim == 1 and shape and strides): + self.assertEqual(m1, ex1) + if ex2 and ex2.ndim == 1 and shape and strides: + self.assertEqual(m2, ex2) + + for req in requests: + for bits in real_flags: + self.verify_getbuf(ex1, ex1, req|bits) + self.verify_getbuf(ex1, m1, req|bits) + if ex2: + self.verify_getbuf(ex2, ex2, req|bits, + sliced=True) + self.verify_getbuf(ex2, m2, req|bits, + sliced=True) + + items = [1,2,3,4,5,6,7,8,9,10,11,12] + + # ND_GETBUF_FAIL + ex = ndarray(items, shape=[12], flags=ND_GETBUF_FAIL) + self.assertRaises(BufferError, ndarray, ex) + + # Request complex structure from a simple exporter. In this + # particular case the test object is not PEP-3118 compliant. + base = ndarray([9], [1]) + ex = ndarray(base, getbuf=PyBUF_SIMPLE) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_WRITABLE) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_ND) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_STRIDES) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_C_CONTIGUOUS) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_F_CONTIGUOUS) + self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_ANY_CONTIGUOUS) + nd = ndarray(ex, getbuf=PyBUF_SIMPLE) + + def test_ndarray_exceptions(self): + nd = ndarray([9], [1]) + ndm = ndarray([9], [1], flags=ND_VAREXPORT) + + # Initialization of a new ndarray or mutation of an existing array. + for c in (ndarray, nd.push, ndm.push): + # Invalid types. + self.assertRaises(TypeError, c, {1,2,3}) + self.assertRaises(TypeError, c, [1,2,'3']) + self.assertRaises(TypeError, c, [1,2,(3,4)]) + self.assertRaises(TypeError, c, [1,2,3], shape={3}) + self.assertRaises(TypeError, c, [1,2,3], shape=[3], strides={1}) + self.assertRaises(TypeError, c, [1,2,3], shape=[3], offset=[]) + self.assertRaises(TypeError, c, [1], shape=[1], format={}) + self.assertRaises(TypeError, c, [1], shape=[1], flags={}) + self.assertRaises(TypeError, c, [1], shape=[1], getbuf={}) + + # ND_FORTRAN flag is only valid without strides. + self.assertRaises(TypeError, c, [1], shape=[1], strides=[1], + flags=ND_FORTRAN) + + # ND_PIL flag is only valid with ndim > 0. + self.assertRaises(TypeError, c, [1], shape=[], flags=ND_PIL) + + # Invalid items. + self.assertRaises(ValueError, c, [], shape=[1]) + self.assertRaises(ValueError, c, ['XXX'], shape=[1], format="L") + # Invalid combination of items and format. + self.assertRaises(struct.error, c, [1000], shape=[1], format="B") + self.assertRaises(ValueError, c, [1,(2,3)], shape=[2], format="B") + self.assertRaises(ValueError, c, [1,2,3], shape=[3], format="QL") + + # Invalid shape. + self.assertRaises(ValueError, c, [1], shape=[-1]) + self.assertRaises(ValueError, c, [1,2,3], shape=['3']) + self.assertRaises(OverflowError, c, [1], shape=[2**128]) + # prod(shape) * itemsize != len(items) + self.assertRaises(ValueError, c, [1,2,3,4,5], shape=[2,2], offset=3) + + # Invalid strides. + self.assertRaises(ValueError, c, [1,2,3], shape=[3], strides=['1']) + self.assertRaises(OverflowError, c, [1], shape=[1], + strides=[2**128]) + + # Invalid combination of strides and shape. + self.assertRaises(ValueError, c, [1,2], shape=[2,1], strides=[1]) + # Invalid combination of strides and format. + self.assertRaises(ValueError, c, [1,2,3,4], shape=[2], strides=[3], + format="L") + + # Invalid offset. + self.assertRaises(ValueError, c, [1,2,3], shape=[3], offset=4) + self.assertRaises(ValueError, c, [1,2,3], shape=[1], offset=3, + format="L") + + # Invalid format. + self.assertRaises(ValueError, c, [1,2,3], shape=[3], format="") + self.assertRaises(struct.error, c, [(1,2,3)], shape=[1], + format="@#$") + + # Striding out of the memory bounds. + items = [1,2,3,4,5,6,7,8,9,10] + self.assertRaises(ValueError, c, items, shape=[2,3], + strides=[-3, -2], offset=5) + + # Constructing consumer: format argument invalid. + self.assertRaises(TypeError, c, bytearray(), format="Q") + + # Constructing original base object: getbuf argument invalid. + self.assertRaises(TypeError, c, [1], shape=[1], getbuf=PyBUF_FULL) + + # Shape argument is mandatory for original base objects. + self.assertRaises(TypeError, c, [1]) + + + # PyBUF_WRITABLE request to read-only provider. + self.assertRaises(BufferError, ndarray, b'123', getbuf=PyBUF_WRITABLE) + + # ND_VAREXPORT can only be specified during construction. + nd = ndarray([9], [1], flags=ND_VAREXPORT) + self.assertRaises(ValueError, nd.push, [1], [1], flags=ND_VAREXPORT) + + # Invalid operation for consumers: push/pop + nd = ndarray(b'123') + self.assertRaises(BufferError, nd.push, [1], [1]) + self.assertRaises(BufferError, nd.pop) + + # ND_VAREXPORT not set: push/pop fail with exported buffers + nd = ndarray([9], [1]) + nd.push([1], [1]) + m = memoryview(nd) + self.assertRaises(BufferError, nd.push, [1], [1]) + self.assertRaises(BufferError, nd.pop) + m.release() + nd.pop() + + # Single remaining buffer: pop fails + self.assertRaises(BufferError, nd.pop) + del nd + + # get_pointer() + self.assertRaises(TypeError, get_pointer, {}, [1,2,3]) + self.assertRaises(TypeError, get_pointer, b'123', {}) + + nd = ndarray(list(range(100)), shape=[1]*100) + self.assertRaises(ValueError, get_pointer, nd, [5]) + + nd = ndarray(list(range(12)), shape=[3,4]) + self.assertRaises(ValueError, get_pointer, nd, [2,3,4]) + self.assertRaises(ValueError, get_pointer, nd, [3,3]) + self.assertRaises(ValueError, get_pointer, nd, [-3,3]) + self.assertRaises(OverflowError, get_pointer, nd, [1<<64,3]) + + # tolist() needs format + ex = ndarray([1,2,3], shape=[3], format='L') + nd = ndarray(ex, getbuf=PyBUF_SIMPLE) + self.assertRaises(ValueError, nd.tolist) + + # memoryview_from_buffer() + ex1 = ndarray([1,2,3], shape=[3], format='L') + ex2 = ndarray(ex1) + nd = ndarray(ex2) + self.assertRaises(TypeError, nd.memoryview_from_buffer) + + nd = ndarray([(1,)*100], shape=[1], format='L'*100) + self.assertRaises(TypeError, nd.memoryview_from_buffer) + + nd = ndarray(list(range(100)), shape=[1 for _ in range(100)]) + self.assertRaises(TypeError, nd.memoryview_from_buffer) + + def test_ndarray_linked_list(self): + for perm in permutations(range(5)): + m = [0]*5 + nd = ndarray([1,2,3], shape=[3], flags=ND_VAREXPORT) + m[0] = memoryview(nd) + + for i in range(1, 5): + nd.push([1,2,3], shape=[3]) + m[i] = memoryview(nd) + + for i in range(5): + m[perm[i]].release() + + self.assertRaises(BufferError, nd.pop) + del nd + + def test_ndarray_format_scalar(self): + # ndim = 0: scalar + for fmt, scalar, _ in iter_format(0): + itemsize = struct.calcsize(fmt) + nd = ndarray(scalar, shape=(), format=fmt) + self.verify(nd, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=0, shape=(), strides=(), + lst=scalar) + + def test_ndarray_format_shape(self): + # ndim = 1, shape = [n] + nitems = randrange(1, 10) + for fmt, items, _ in iter_format(nitems): + itemsize = struct.calcsize(fmt) + for flags in (0, ND_PIL): + nd = ndarray(items, shape=[nitems], format=fmt, flags=flags) + self.verify(nd, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=1, shape=(nitems,), strides=(itemsize,), + lst=items) + + def test_ndarray_format_strides(self): + # ndim = 1, strides + nitems = randrange(1, 30) + for fmt, items, _ in iter_format(nitems): + itemsize = struct.calcsize(fmt) + for step in range(-5, 5): + if step == 0: + continue + + shape = [len(items[::step])] + strides = [step*itemsize] + offset = itemsize*(nitems-1) if step < 0 else 0 + + for flags in (0, ND_PIL): + nd = ndarray(items, shape=shape, strides=strides, + format=fmt, offset=offset, flags=flags) + self.verify(nd, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=1, shape=shape, strides=strides, + lst=items[::step]) + + def test_ndarray_fortran(self): + items = [1,2,3,4,5,6,7,8,9,10,11,12] + ex = ndarray(items, shape=(3, 4), strides=(1, 3)) + nd = ndarray(ex, getbuf=PyBUF_F_CONTIGUOUS|PyBUF_FORMAT) + self.assertEqual(nd.tolist(), farray(items, (3, 4))) + + def test_ndarray_multidim(self): + for ndim in range(5): + shape_t = [randrange(2, 10) for _ in range(ndim)] + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt, items, _ = randitems(nitems) + itemsize = struct.calcsize(fmt) + + for flags in (0, ND_PIL): + if ndim == 0 and flags == ND_PIL: + continue + + # C array + nd = ndarray(items, shape=shape, format=fmt, flags=flags) + + strides = strides_from_shape(ndim, shape, itemsize, 'C') + lst = carray(items, shape) + self.verify(nd, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + if is_memoryview_format(fmt): + # memoryview: reconstruct strides + ex = ndarray(items, shape=shape, format=fmt) + nd = ndarray(ex, getbuf=PyBUF_CONTIG_RO|PyBUF_FORMAT) + self.assertTrue(nd.strides == ()) + mv = nd.memoryview_from_buffer() + self.verify(mv, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + # Fortran array + nd = ndarray(items, shape=shape, format=fmt, + flags=flags|ND_FORTRAN) + + strides = strides_from_shape(ndim, shape, itemsize, 'F') + lst = farray(items, shape) + self.verify(nd, obj=None, + itemsize=itemsize, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + def test_ndarray_index_invalid(self): + # not writable + nd = ndarray([1], shape=[1]) + self.assertRaises(TypeError, nd.__setitem__, 1, 8) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertRaises(TypeError, mv.__setitem__, 1, 8) + + # cannot be deleted + nd = ndarray([1], shape=[1], flags=ND_WRITABLE) + self.assertRaises(TypeError, nd.__delitem__, 1) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertRaises(TypeError, mv.__delitem__, 1) + + # overflow + nd = ndarray([1], shape=[1], flags=ND_WRITABLE) + self.assertRaises(OverflowError, nd.__getitem__, 1<<64) + self.assertRaises(OverflowError, nd.__setitem__, 1<<64, 8) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertRaises(IndexError, mv.__getitem__, 1<<64) + self.assertRaises(IndexError, mv.__setitem__, 1<<64, 8) + + # format + items = [1,2,3,4,5,6,7,8] + nd = ndarray(items, shape=[len(items)], format="B", flags=ND_WRITABLE) + self.assertRaises(struct.error, nd.__setitem__, 2, 300) + self.assertRaises(ValueError, nd.__setitem__, 1, (100, 200)) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertRaises(ValueError, mv.__setitem__, 2, 300) + self.assertRaises(ValueError, mv.__setitem__, 1, (100, 200)) + + items = [(1,2), (3,4), (5,6)] + nd = ndarray(items, shape=[len(items)], format="LQ", flags=ND_WRITABLE) + self.assertRaises(ValueError, nd.__setitem__, 2, 300) + self.assertRaises(struct.error, nd.__setitem__, 1, (b'\x001', 200)) + + def test_ndarray_index_scalar(self): + # scalar + nd = ndarray(1, shape=(), flags=ND_WRITABLE) + mv = memoryview(nd) + self.assertEqual(mv, nd) + + x = nd[()]; self.assertEqual(x, 1) + x = nd[...]; self.assertEqual(x.tolist(), nd.tolist()) + + x = mv[()]; self.assertEqual(x, 1) + x = mv[...]; self.assertEqual(x.tolist(), nd.tolist()) + + self.assertRaises(TypeError, nd.__getitem__, 0) + self.assertRaises(TypeError, mv.__getitem__, 0) + self.assertRaises(TypeError, nd.__setitem__, 0, 8) + self.assertRaises(TypeError, mv.__setitem__, 0, 8) + + self.assertEqual(nd.tolist(), 1) + self.assertEqual(mv.tolist(), 1) + + nd[()] = 9; self.assertEqual(nd.tolist(), 9) + mv[()] = 9; self.assertEqual(mv.tolist(), 9) + + nd[...] = 5; self.assertEqual(nd.tolist(), 5) + mv[...] = 5; self.assertEqual(mv.tolist(), 5) + + def test_ndarray_index_null_strides(self): + ex = ndarray(list(range(2*4)), shape=[2, 4], flags=ND_WRITABLE) + nd = ndarray(ex, getbuf=PyBUF_CONTIG) + + # Sub-views are only possible for full exporters. + self.assertRaises(BufferError, nd.__getitem__, 1) + # Same for slices. + self.assertRaises(BufferError, nd.__getitem__, slice(3,5,1)) + + def test_ndarray_index_getitem_single(self): + # getitem + for fmt, items, _ in iter_format(5): + nd = ndarray(items, shape=[5], format=fmt) + for i in range(-5, 5): + self.assertEqual(nd[i], items[i]) + + self.assertRaises(IndexError, nd.__getitem__, -6) + self.assertRaises(IndexError, nd.__getitem__, 5) + + if is_memoryview_format(fmt): + mv = memoryview(nd) + self.assertEqual(mv, nd) + for i in range(-5, 5): + self.assertEqual(mv[i], items[i]) + + self.assertRaises(IndexError, mv.__getitem__, -6) + self.assertRaises(IndexError, mv.__getitem__, 5) + + # getitem with null strides + for fmt, items, _ in iter_format(5): + ex = ndarray(items, shape=[5], flags=ND_WRITABLE, format=fmt) + nd = ndarray(ex, getbuf=PyBUF_CONTIG|PyBUF_FORMAT) + + for i in range(-5, 5): + self.assertEqual(nd[i], items[i]) + + if is_memoryview_format(fmt): + mv = nd.memoryview_from_buffer() + self.assertIs(mv.__eq__(nd), NotImplemented) + for i in range(-5, 5): + self.assertEqual(mv[i], items[i]) + + # getitem with null format + items = [1,2,3,4,5] + ex = ndarray(items, shape=[5]) + nd = ndarray(ex, getbuf=PyBUF_CONTIG_RO) + for i in range(-5, 5): + self.assertEqual(nd[i], items[i]) + + # getitem with null shape/strides/format + items = [1,2,3,4,5] + ex = ndarray(items, shape=[5]) + nd = ndarray(ex, getbuf=PyBUF_SIMPLE) + + for i in range(-5, 5): + self.assertEqual(nd[i], items[i]) + + def test_ndarray_index_setitem_single(self): + # assign single value + for fmt, items, single_item in iter_format(5): + nd = ndarray(items, shape=[5], format=fmt, flags=ND_WRITABLE) + for i in range(5): + items[i] = single_item + nd[i] = single_item + self.assertEqual(nd.tolist(), items) + + self.assertRaises(IndexError, nd.__setitem__, -6, single_item) + self.assertRaises(IndexError, nd.__setitem__, 5, single_item) + + if not is_memoryview_format(fmt): + continue + + nd = ndarray(items, shape=[5], format=fmt, flags=ND_WRITABLE) + mv = memoryview(nd) + self.assertEqual(mv, nd) + for i in range(5): + items[i] = single_item + mv[i] = single_item + self.assertEqual(mv.tolist(), items) + + self.assertRaises(IndexError, mv.__setitem__, -6, single_item) + self.assertRaises(IndexError, mv.__setitem__, 5, single_item) + + + # assign single value: lobject = robject + for fmt, items, single_item in iter_format(5): + nd = ndarray(items, shape=[5], format=fmt, flags=ND_WRITABLE) + for i in range(-5, 4): + items[i] = items[i+1] + nd[i] = nd[i+1] + self.assertEqual(nd.tolist(), items) + + if not is_memoryview_format(fmt): + continue + + nd = ndarray(items, shape=[5], format=fmt, flags=ND_WRITABLE) + mv = memoryview(nd) + self.assertEqual(mv, nd) + for i in range(-5, 4): + items[i] = items[i+1] + mv[i] = mv[i+1] + self.assertEqual(mv.tolist(), items) + + def test_ndarray_index_getitem_multidim(self): + shape_t = (2, 3, 5) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt, items, _ = randitems(nitems) + + for flags in (0, ND_PIL): + # C array + nd = ndarray(items, shape=shape, format=fmt, flags=flags) + lst = carray(items, shape) + + for i in range(-shape[0], shape[0]): + self.assertEqual(lst[i], nd[i].tolist()) + for j in range(-shape[1], shape[1]): + self.assertEqual(lst[i][j], nd[i][j].tolist()) + for k in range(-shape[2], shape[2]): + self.assertEqual(lst[i][j][k], nd[i][j][k]) + + # Fortran array + nd = ndarray(items, shape=shape, format=fmt, + flags=flags|ND_FORTRAN) + lst = farray(items, shape) + + for i in range(-shape[0], shape[0]): + self.assertEqual(lst[i], nd[i].tolist()) + for j in range(-shape[1], shape[1]): + self.assertEqual(lst[i][j], nd[i][j].tolist()) + for k in range(shape[2], shape[2]): + self.assertEqual(lst[i][j][k], nd[i][j][k]) + + def test_ndarray_sequence(self): + nd = ndarray(1, shape=()) + self.assertRaises(TypeError, eval, "1 in nd", locals()) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertRaises(TypeError, eval, "1 in mv", locals()) + + for fmt, items, _ in iter_format(5): + nd = ndarray(items, shape=[5], format=fmt) + for i, v in enumerate(nd): + self.assertEqual(v, items[i]) + self.assertTrue(v in nd) + + if is_memoryview_format(fmt): + mv = memoryview(nd) + for i, v in enumerate(mv): + self.assertEqual(v, items[i]) + self.assertTrue(v in mv) + + def test_ndarray_slice_invalid(self): + items = [1,2,3,4,5,6,7,8] + + # rvalue is not an exporter + xl = ndarray(items, shape=[8], flags=ND_WRITABLE) + ml = memoryview(xl) + self.assertRaises(TypeError, xl.__setitem__, slice(0,8,1), items) + self.assertRaises(TypeError, ml.__setitem__, slice(0,8,1), items) + + # rvalue is not a full exporter + xl = ndarray(items, shape=[8], flags=ND_WRITABLE) + ex = ndarray(items, shape=[8], flags=ND_WRITABLE) + xr = ndarray(ex, getbuf=PyBUF_ND) + self.assertRaises(BufferError, xl.__setitem__, slice(0,8,1), xr) + + # zero step + nd = ndarray(items, shape=[8], format="L", flags=ND_WRITABLE) + mv = memoryview(nd) + self.assertRaises(ValueError, nd.__getitem__, slice(0,1,0)) + self.assertRaises(ValueError, mv.__getitem__, slice(0,1,0)) + + nd = ndarray(items, shape=[2,4], format="L", flags=ND_WRITABLE) + mv = memoryview(nd) + + self.assertRaises(ValueError, nd.__getitem__, + (slice(0,1,1), slice(0,1,0))) + self.assertRaises(ValueError, nd.__getitem__, + (slice(0,1,0), slice(0,1,1))) + self.assertRaises(TypeError, nd.__getitem__, "@%$") + self.assertRaises(TypeError, nd.__getitem__, ("@%$", slice(0,1,1))) + self.assertRaises(TypeError, nd.__getitem__, (slice(0,1,1), {})) + + # memoryview: not implemented + self.assertRaises(NotImplementedError, mv.__getitem__, + (slice(0,1,1), slice(0,1,0))) + self.assertRaises(TypeError, mv.__getitem__, "@%$") + + # differing format + xl = ndarray(items, shape=[8], format="B", flags=ND_WRITABLE) + xr = ndarray(items, shape=[8], format="b") + ml = memoryview(xl) + mr = memoryview(xr) + self.assertRaises(ValueError, xl.__setitem__, slice(0,1,1), xr[7:8]) + self.assertEqual(xl.tolist(), items) + self.assertRaises(ValueError, ml.__setitem__, slice(0,1,1), mr[7:8]) + self.assertEqual(ml.tolist(), items) + + # differing itemsize + xl = ndarray(items, shape=[8], format="B", flags=ND_WRITABLE) + yr = ndarray(items, shape=[8], format="L") + ml = memoryview(xl) + mr = memoryview(xr) + self.assertRaises(ValueError, xl.__setitem__, slice(0,1,1), xr[7:8]) + self.assertEqual(xl.tolist(), items) + self.assertRaises(ValueError, ml.__setitem__, slice(0,1,1), mr[7:8]) + self.assertEqual(ml.tolist(), items) + + # differing ndim + xl = ndarray(items, shape=[2, 4], format="b", flags=ND_WRITABLE) + xr = ndarray(items, shape=[8], format="b") + ml = memoryview(xl) + mr = memoryview(xr) + self.assertRaises(ValueError, xl.__setitem__, slice(0,1,1), xr[7:8]) + self.assertEqual(xl.tolist(), [[1,2,3,4], [5,6,7,8]]) + self.assertRaises(NotImplementedError, ml.__setitem__, slice(0,1,1), + mr[7:8]) + self.assertRaises(NotImplementedError, ml.tolist) + + # differing shape + xl = ndarray(items, shape=[8], format="b", flags=ND_WRITABLE) + xr = ndarray(items, shape=[8], format="b") + ml = memoryview(xl) + mr = memoryview(xr) + self.assertRaises(ValueError, xl.__setitem__, slice(0,2,1), xr[7:8]) + self.assertEqual(xl.tolist(), items) + self.assertRaises(ValueError, ml.__setitem__, slice(0,2,1), mr[7:8]) + self.assertEqual(ml.tolist(), items) + + # _testbuffer.c module functions + self.assertRaises(TypeError, slice_indices, slice(0,1,2), {}) + self.assertRaises(ValueError, slice_indices, slice(0,1,0), 4) + + x = ndarray(items, shape=[8], format="b", flags=ND_PIL) + self.assertRaises(TypeError, x.add_suboffsets) + + ex = ndarray(items, shape=[8], format="B") + x = ndarray(ex, getbuf=PyBUF_SIMPLE) + self.assertRaises(TypeError, x.add_suboffsets) + + def test_ndarray_slice_zero_shape(self): + items = [1,2,3,4,5,6,7,8,9,10,11,12] + + x = ndarray(items, shape=[12], format="L", flags=ND_WRITABLE) + y = ndarray(items, shape=[12], format="L") + x[4:4] = y[9:9] + self.assertEqual(x.tolist(), items) + + ml = memoryview(x) + mr = memoryview(y) + self.assertEqual(ml, x) + self.assertEqual(ml, y) + ml[4:4] = mr[9:9] + self.assertEqual(ml.tolist(), items) + + x = ndarray(items, shape=[3, 4], format="L", flags=ND_WRITABLE) + y = ndarray(items, shape=[4, 3], format="L") + x[1:2, 2:2] = y[1:2, 3:3] + self.assertEqual(x.tolist(), carray(items, [3, 4])) + + def test_ndarray_slice_multidim(self): + shape_t = (2, 3, 5) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt, items, _ = randitems(nitems) + itemsize = struct.calcsize(fmt) + + for flags in (0, ND_PIL): + nd = ndarray(items, shape=shape, format=fmt, flags=flags) + lst = carray(items, shape) + + for slices in rslices_ndim(ndim, shape): + + listerr = None + try: + sliced = multislice(lst, slices) + except Exception as e: + listerr = e.__class__ + + nderr = None + try: + ndsliced = nd[slices] + except Exception as e: + nderr = e.__class__ + + if nderr or listerr: + self.assertIs(nderr, listerr) + else: + self.assertEqual(ndsliced.tolist(), sliced) + + def test_ndarray_slice_redundant_suboffsets(self): + shape_t = (2, 3, 5, 2) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt, items, _ = randitems(nitems) + itemsize = struct.calcsize(fmt) + + nd = ndarray(items, shape=shape, format=fmt) + nd.add_suboffsets() + ex = ndarray(items, shape=shape, format=fmt) + ex.add_suboffsets() + mv = memoryview(ex) + lst = carray(items, shape) + + for slices in rslices_ndim(ndim, shape): + + listerr = None + try: + sliced = multislice(lst, slices) + except Exception as e: + listerr = e.__class__ + + nderr = None + try: + ndsliced = nd[slices] + except Exception as e: + nderr = e.__class__ + + if nderr or listerr: + self.assertIs(nderr, listerr) + else: + self.assertEqual(ndsliced.tolist(), sliced) + + def test_ndarray_slice_assign_single(self): + for fmt, items, _ in iter_format(5): + for lslice in genslices(5): + for rslice in genslices(5): + for flags in (0, ND_PIL): + + f = flags|ND_WRITABLE + nd = ndarray(items, shape=[5], format=fmt, flags=f) + ex = ndarray(items, shape=[5], format=fmt, flags=f) + mv = memoryview(ex) + + lsterr = None + diff_structure = None + lst = items[:] + try: + lval = lst[lslice] + rval = lst[rslice] + lst[lslice] = lst[rslice] + diff_structure = len(lval) != len(rval) + except Exception as e: + lsterr = e.__class__ + + nderr = None + try: + nd[lslice] = nd[rslice] + except Exception as e: + nderr = e.__class__ + + if diff_structure: # ndarray cannot change shape + self.assertIs(nderr, ValueError) + else: + self.assertEqual(nd.tolist(), lst) + self.assertIs(nderr, lsterr) + + if not is_memoryview_format(fmt): + continue + + mverr = None + try: + mv[lslice] = mv[rslice] + except Exception as e: + mverr = e.__class__ + + if diff_structure: # memoryview cannot change shape + self.assertIs(mverr, ValueError) + else: + self.assertEqual(mv.tolist(), lst) + self.assertEqual(mv, nd) + self.assertIs(mverr, lsterr) + self.verify(mv, obj=ex, + itemsize=nd.itemsize, fmt=fmt, readonly=0, + ndim=nd.ndim, shape=nd.shape, strides=nd.strides, + lst=nd.tolist()) + + def test_ndarray_slice_assign_multidim(self): + shape_t = (2, 3, 5) + ndim = len(shape_t) + nitems = prod(shape_t) + for shape in permutations(shape_t): + + fmt, items, _ = randitems(nitems) + + for flags in (0, ND_PIL): + for _ in range(ITERATIONS): + lslices, rslices = randslice_from_shape(ndim, shape) + + nd = ndarray(items, shape=shape, format=fmt, + flags=flags|ND_WRITABLE) + lst = carray(items, shape) + + listerr = None + try: + result = multislice_assign(lst, lst, lslices, rslices) + except Exception as e: + listerr = e.__class__ + + nderr = None + try: + nd[lslices] = nd[rslices] + except Exception as e: + nderr = e.__class__ + + if nderr or listerr: + self.assertIs(nderr, listerr) + else: + self.assertEqual(nd.tolist(), result) + + def test_ndarray_random(self): + # construction of valid arrays + for _ in range(ITERATIONS): + for fmt in fmtdict['@']: + itemsize = struct.calcsize(fmt) + + t = rand_structure(itemsize, True, maxdim=MAXDIM, + maxshape=MAXSHAPE) + self.assertTrue(verify_structure(*t)) + items = randitems_from_structure(fmt, t) + + x = ndarray_from_structure(items, fmt, t) + xlist = x.tolist() + + mv = memoryview(x) + if mv.ndim <= 1: + mvlist = mv.tolist() + self.assertEqual(mvlist, xlist) + + if t[2] > 0: + # ndim > 0: test against suboffsets representation. + y = ndarray_from_structure(items, fmt, t, flags=ND_PIL) + ylist = y.tolist() + self.assertEqual(xlist, ylist) + + mv = memoryview(y) + if mv.ndim <= 1: + self.assertEqual(mv, y) + mvlist = mv.tolist() + self.assertEqual(mvlist, ylist) + + if numpy_array: + shape = t[3] + if 0 in shape: + continue # http://projects.scipy.org/numpy/ticket/1910 + z = numpy_array_from_structure(items, fmt, t) + self.verify(x, obj=None, + itemsize=z.itemsize, fmt=fmt, readonly=0, + ndim=z.ndim, shape=z.shape, strides=z.strides, + lst=z.tolist()) + + def test_ndarray_random_invalid(self): + # exceptions during construction of invalid arrays + for _ in range(ITERATIONS): + for fmt in fmtdict['@']: + itemsize = struct.calcsize(fmt) + + t = rand_structure(itemsize, False, maxdim=MAXDIM, + maxshape=MAXSHAPE) + self.assertFalse(verify_structure(*t)) + items = randitems_from_structure(fmt, t) + + nderr = False + try: + x = ndarray_from_structure(items, fmt, t) + except Exception as e: + nderr = e.__class__ + self.assertTrue(nderr) + + if numpy_array: + numpy_err = False + try: + y = numpy_array_from_structure(items, fmt, t) + except Exception as e: + numpy_err = e.__class__ + + if 0: # http://projects.scipy.org/numpy/ticket/1910 + self.assertTrue(numpy_err) + + def test_ndarray_random_slice_assign(self): + # valid slice assignments + for _ in range(ITERATIONS): + for fmt in fmtdict['@']: + itemsize = struct.calcsize(fmt) + + lshape, rshape, lslices, rslices = \ + rand_aligned_slices(maxdim=MAXDIM, maxshape=MAXSHAPE) + tl = rand_structure(itemsize, True, shape=lshape) + tr = rand_structure(itemsize, True, shape=rshape) + self.assertTrue(verify_structure(*tl)) + self.assertTrue(verify_structure(*tr)) + litems = randitems_from_structure(fmt, tl) + ritems = randitems_from_structure(fmt, tr) + + xl = ndarray_from_structure(litems, fmt, tl) + xr = ndarray_from_structure(ritems, fmt, tr) + xl[lslices] = xr[rslices] + xllist = xl.tolist() + xrlist = xr.tolist() + + # just test construction + ml = memoryview(xl) + mr = memoryview(xr) + + if tl[2] > 0 and tr[2] > 0: + # ndim > 0: test against suboffsets representation. + yl = ndarray_from_structure(litems, fmt, tl, flags=ND_PIL) + yr = ndarray_from_structure(ritems, fmt, tr, flags=ND_PIL) + yl[lslices] = yr[rslices] + yllist = yl.tolist() + yrlist = yr.tolist() + self.assertEqual(xllist, yllist) + self.assertEqual(xrlist, yrlist) + + if numpy_array: + if 0 in lshape or 0 in rshape: + continue # http://projects.scipy.org/numpy/ticket/1910 + + zl = numpy_array_from_structure(litems, fmt, tl) + zr = numpy_array_from_structure(ritems, fmt, tr) + zl[lslices] = zr[rslices] + + if not is_overlapping(tl) and not is_overlapping(tr): + # Slice assignment of overlapping structures + # is undefined in NumPy. + self.verify(xl, obj=None, + itemsize=zl.itemsize, fmt=fmt, readonly=0, + ndim=zl.ndim, shape=zl.shape, + strides=zl.strides, lst=zl.tolist()) + + self.verify(xr, obj=None, + itemsize=zr.itemsize, fmt=fmt, readonly=0, + ndim=zr.ndim, shape=zr.shape, + strides=zr.strides, lst=zr.tolist()) + + def test_ndarray_re_export(self): + items = [1,2,3,4,5,6,7,8,9,10,11,12] + + nd = ndarray(items, shape=[3,4], flags=ND_PIL) + ex = ndarray(nd) + + self.assertTrue(ex.flags & ND_PIL) + self.assertIs(ex.obj, nd) + self.assertEqual(ex.suboffsets, (0, -1)) + self.assertFalse(ex.c_contiguous) + self.assertFalse(ex.f_contiguous) + self.assertFalse(ex.contiguous) + + def test_ndarray_zero_shape(self): + # zeros in shape + for flags in (0, ND_PIL): + nd = ndarray([1,2,3], shape=[0], flags=flags) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertEqual(nd.tolist(), []) + self.assertEqual(mv.tolist(), []) + + nd = ndarray([1,2,3], shape=[0,3,3], flags=flags) + self.assertEqual(nd.tolist(), []) + + nd = ndarray([1,2,3], shape=[3,0,3], flags=flags) + self.assertEqual(nd.tolist(), [[], [], []]) + + nd = ndarray([1,2,3], shape=[3,3,0], flags=flags) + self.assertEqual(nd.tolist(), + [[[], [], []], [[], [], []], [[], [], []]]) + + def test_ndarray_zero_strides(self): + # zero strides + for flags in (0, ND_PIL): + nd = ndarray([1], shape=[5], strides=[0], flags=flags) + mv = memoryview(nd) + self.assertEqual(mv, nd) + self.assertEqual(nd.tolist(), [1, 1, 1, 1, 1]) + self.assertEqual(mv.tolist(), [1, 1, 1, 1, 1]) + + def test_ndarray_offset(self): + nd = ndarray(list(range(20)), shape=[3], offset=7) + self.assertEqual(nd.offset, 7) + self.assertEqual(nd.tolist(), [7,8,9]) + + def test_ndarray_memoryview_from_buffer(self): + for flags in (0, ND_PIL): + nd = ndarray(list(range(3)), shape=[3], flags=flags) + m = nd.memoryview_from_buffer() + self.assertEqual(m, nd) + + def test_ndarray_get_pointer(self): + for flags in (0, ND_PIL): + nd = ndarray(list(range(3)), shape=[3], flags=flags) + for i in range(3): + self.assertEqual(nd[i], get_pointer(nd, [i])) + + def test_ndarray_tolist_null_strides(self): + ex = ndarray(list(range(20)), shape=[2,2,5]) + nd = ndarray(ex, getbuf=PyBUF_ND|PyBUF_FORMAT) + self.assertEqual(ex.tolist(), nd.tolist()) + + def test_memoryview_construction(self): + + items_shape = [(9, []), ([1,2,3], [3]), (list(range(2*3*5)), [2,3,5])] + + # NumPy style, C-contiguous: + for items, shape in items_shape: + + # From PEP-3118 compliant exporter: + ex = ndarray(items, shape=shape) + m = memoryview(ex) + self.assertTrue(m.c_contiguous) + self.assertTrue(m.contiguous) + + ndim = len(shape) + strides = strides_from_shape(ndim, shape, 1, 'C') + lst = carray(items, shape) + + self.verify(m, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + # From memoryview: + m2 = memoryview(m) + self.verify(m2, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + # PyMemoryView_FromBuffer(): no strides + nd = ndarray(ex, getbuf=PyBUF_CONTIG_RO|PyBUF_FORMAT) + self.assertEqual(nd.strides, ()) + m = nd.memoryview_from_buffer() + self.verify(m, obj=None, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + # PyMemoryView_FromBuffer(): no format, shape, strides + nd = ndarray(ex, getbuf=PyBUF_SIMPLE) + self.assertEqual(nd.format, '') + self.assertEqual(nd.shape, ()) + self.assertEqual(nd.strides, ()) + m = nd.memoryview_from_buffer() + + lst = [items] if ndim == 0 else items + self.verify(m, obj=None, + itemsize=1, fmt='B', readonly=1, + ndim=1, shape=[ex.len], strides=(1,), + lst=lst) + + # NumPy style, Fortran contiguous: + for items, shape in items_shape: + + # From PEP-3118 compliant exporter: + ex = ndarray(items, shape=shape, flags=ND_FORTRAN) + m = memoryview(ex) + self.assertTrue(m.f_contiguous) + self.assertTrue(m.contiguous) + + ndim = len(shape) + strides = strides_from_shape(ndim, shape, 1, 'F') + lst = farray(items, shape) + + self.verify(m, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + # From memoryview: + m2 = memoryview(m) + self.verify(m2, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst) + + # PIL style: + for items, shape in items_shape[1:]: + + # From PEP-3118 compliant exporter: + ex = ndarray(items, shape=shape, flags=ND_PIL) + m = memoryview(ex) + + ndim = len(shape) + lst = carray(items, shape) + + self.verify(m, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=ex.strides, + lst=lst) + + # From memoryview: + m2 = memoryview(m) + self.verify(m2, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=ndim, shape=shape, strides=ex.strides, + lst=lst) + + # Invalid number of arguments: + self.assertRaises(TypeError, memoryview, b'9', 'x') + # Not a buffer provider: + self.assertRaises(TypeError, memoryview, {}) + # Non-compliant buffer provider: + ex = ndarray([1,2,3], shape=[3]) + nd = ndarray(ex, getbuf=PyBUF_SIMPLE) + self.assertRaises(BufferError, memoryview, nd) + nd = ndarray(ex, getbuf=PyBUF_CONTIG_RO|PyBUF_FORMAT) + self.assertRaises(BufferError, memoryview, nd) + + def test_memoryview_cast_zero_shape(self): + # Casts are undefined if shape contains zeros. These arrays are + # regarded as C-contiguous by Numpy and PyBuffer_GetContiguous(), + # so they are not caught by the test for C-contiguity in memory_cast(). + items = [1,2,3] + for shape in ([0,3,3], [3,0,3], [0,3,3]): + ex = ndarray(items, shape=shape) + self.assertTrue(ex.c_contiguous) + msrc = memoryview(ex) + self.assertRaises(TypeError, msrc.cast, 'c') + + def test_memoryview_cast_zero_strides(self): + # Casts are undefined if strides contains zeros. These arrays are + # (sometimes!) regarded as C-contiguous by Numpy, but not by + # PyBuffer_GetContiguous(). + ex = ndarray([1,2,3], shape=[3], strides=[0]) + self.assertFalse(ex.c_contiguous) + msrc = memoryview(ex) + self.assertRaises(TypeError, msrc.cast, 'c') + + def test_memoryview_cast_invalid(self): + # invalid format + for sfmt in NON_BYTE_FORMAT: + sformat = '@' + sfmt if randrange(2) else sfmt + ssize = struct.calcsize(sformat) + for dfmt in NON_BYTE_FORMAT: + dformat = '@' + dfmt if randrange(2) else dfmt + dsize = struct.calcsize(dformat) + ex = ndarray(list(range(32)), shape=[32//ssize], format=sformat) + msrc = memoryview(ex) + self.assertRaises(TypeError, msrc.cast, dfmt, [32//dsize]) + + for sfmt, sitems, _ in iter_format(1): + ex = ndarray(sitems, shape=[1], format=sfmt) + msrc = memoryview(ex) + for dfmt, _, _ in iter_format(1): + if (not is_memoryview_format(sfmt) or + not is_memoryview_format(dfmt)): + self.assertRaises(NotImplementedError, msrc.cast, dfmt, + [32//dsize]) + else: + if not is_byte_format(sfmt) and not is_byte_format(dfmt): + self.assertRaises(TypeError, msrc.cast, dfmt, + [32//dsize]) + + # invalid shape + size_h = struct.calcsize('h') + size_d = struct.calcsize('d') + ex = ndarray(list(range(2*2*size_d)), shape=[2,2,size_d], format='h') + msrc = memoryview(ex) + self.assertRaises(TypeError, msrc.cast, shape=[2,2,size_h], format='d') + + ex = ndarray(list(range(120)), shape=[1,2,3,4,5]) + m = memoryview(ex) + + # incorrect number of args + self.assertRaises(TypeError, m.cast) + self.assertRaises(TypeError, m.cast, 1, 2, 3) + + # incorrect dest format type + self.assertRaises(TypeError, m.cast, {}) + + # incorrect dest format + self.assertRaises(ValueError, m.cast, "X") + self.assertRaises(ValueError, m.cast, "@X") + self.assertRaises(ValueError, m.cast, "@XY") + + # dest format not implemented + self.assertRaises(NotImplementedError, m.cast, "=B") + self.assertRaises(NotImplementedError, m.cast, "!L") + self.assertRaises(NotImplementedError, m.cast, "l") + self.assertRaises(NotImplementedError, m.cast, "BI") + self.assertRaises(NotImplementedError, m.cast, "xBI") + + # src format not implemented + ex = ndarray([(1,2), (3,4)], shape=[2], format="II") + m = memoryview(ex) + self.assertRaises(NotImplementedError, m.__getitem__, 0) + self.assertRaises(NotImplementedError, m.__setitem__, 0, 8) + self.assertRaises(NotImplementedError, m.tolist) + + # incorrect shape type + ex = ndarray(list(range(120)), shape=[1,2,3,4,5]) + m = memoryview(ex) + self.assertRaises(TypeError, m.cast, "B", shape={}) + + # incorrect shape elements + ex = ndarray(list(range(120)), shape=[2*3*4*5]) + m = memoryview(ex) + self.assertRaises(OverflowError, m.cast, "B", shape=[2**64]) + self.assertRaises(ValueError, m.cast, "B", shape=[-1]) + self.assertRaises(ValueError, m.cast, "B", shape=[2,3,4,5,6,7,-1]) + self.assertRaises(ValueError, m.cast, "B", shape=[2,3,4,5,6,7,0]) + self.assertRaises(TypeError, m.cast, "B", shape=[2,3,4,5,6,7,'x']) + + # N-D -> N-D cast + ex = ndarray(list([9 for _ in range(3*5*7*11)]), shape=[3,5,7,11]) + m = memoryview(ex) + self.assertRaises(TypeError, m.cast, "I", shape=[2,3,4,5]) + + # view->len not a multiple of itemsize + ex = ndarray(list([9 for _ in range(3*5*7*11)]), shape=[3*5*7*11]) + m = memoryview(ex) + self.assertRaises(TypeError, m.cast, "I", shape=[2,3,4,5]) + + # product(shape) * itemsize != buffer size + ex = ndarray(list([9 for _ in range(3*5*7*11)]), shape=[3*5*7*11]) + m = memoryview(ex) + self.assertRaises(TypeError, m.cast, "B", shape=[2,3,4,5]) + + def test_memoryview_cast(self): + bytespec = ( + ('B', lambda ex: list(ex.tobytes())), + ('b', lambda ex: [x-256 if x > 127 else x for x in list(ex.tobytes())]), + ('c', lambda ex: [bytes(chr(x), 'latin-1') for x in list(ex.tobytes())]), + ) + + def iter_roundtrip(ex, m, items, fmt): + srcsize = struct.calcsize(fmt) + for bytefmt, to_bytelist in bytespec: + + m2 = m.cast(bytefmt) + lst = to_bytelist(ex) + self.verify(m2, obj=ex, + itemsize=1, fmt=bytefmt, readonly=0, + ndim=1, shape=[31*srcsize], strides=(1,), + lst=lst, cast=True) + + m3 = m2.cast(fmt) + self.assertEqual(m3, ex) + lst = ex.tolist() + self.verify(m3, obj=ex, + itemsize=srcsize, fmt=fmt, readonly=0, + ndim=1, shape=[31], strides=(srcsize,), + lst=lst, cast=True) + + # cast from ndim = 0 to ndim = 1 + srcsize = struct.calcsize('I') + ex = ndarray(9, shape=[], format='I') + destitems, destshape = cast_items(ex, 'B', 1) + m = memoryview(ex) + m2 = m.cast('B') + self.verify(m2, obj=ex, + itemsize=1, fmt='B', readonly=1, + ndim=1, shape=destshape, strides=(1,), + lst=destitems, cast=True) + + # cast from ndim = 1 to ndim = 0 + destsize = struct.calcsize('I') + ex = ndarray([9]*destsize, shape=[destsize], format='B') + destitems, destshape = cast_items(ex, 'I', destsize, shape=[]) + m = memoryview(ex) + m2 = m.cast('I', shape=[]) + self.verify(m2, obj=ex, + itemsize=destsize, fmt='I', readonly=1, + ndim=0, shape=(), strides=(), + lst=destitems, cast=True) + + # array.array: roundtrip to/from bytes + for fmt, items, _ in iter_format(31, 'array'): + ex = array.array(fmt, items) + m = memoryview(ex) + iter_roundtrip(ex, m, items, fmt) + + # ndarray: roundtrip to/from bytes + for fmt, items, _ in iter_format(31, 'memoryview'): + ex = ndarray(items, shape=[31], format=fmt, flags=ND_WRITABLE) + m = memoryview(ex) + iter_roundtrip(ex, m, items, fmt) + + def test_memoryview_cast_1D_ND(self): + # Cast between C-contiguous buffers. At least one buffer must + # be 1D, at least one format must be 'c', 'b' or 'B'. + for _tshape in gencastshapes(): + for char in fmtdict['@']: + tfmt = ('', '@')[randrange(2)] + char + tsize = struct.calcsize(tfmt) + n = prod(_tshape) * tsize + obj = 'memoryview' if is_byte_format(tfmt) else 'bytefmt' + for fmt, items, _ in iter_format(n, obj): + size = struct.calcsize(fmt) + shape = [n] if n > 0 else [] + tshape = _tshape + [size] + + ex = ndarray(items, shape=shape, format=fmt) + m = memoryview(ex) + + titems, tshape = cast_items(ex, tfmt, tsize, shape=tshape) + + if titems is None: + self.assertRaises(TypeError, m.cast, tfmt, tshape) + continue + if titems == 'nan': + continue # NaNs in lists are a recipe for trouble. + + # 1D -> ND + nd = ndarray(titems, shape=tshape, format=tfmt) + + m2 = m.cast(tfmt, shape=tshape) + ndim = len(tshape) + strides = nd.strides + lst = nd.tolist() + self.verify(m2, obj=ex, + itemsize=tsize, fmt=tfmt, readonly=1, + ndim=ndim, shape=tshape, strides=strides, + lst=lst, cast=True) + + # ND -> 1D + m3 = m2.cast(fmt) + m4 = m2.cast(fmt, shape=shape) + ndim = len(shape) + strides = ex.strides + lst = ex.tolist() + + self.verify(m3, obj=ex, + itemsize=size, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst, cast=True) + + self.verify(m4, obj=ex, + itemsize=size, fmt=fmt, readonly=1, + ndim=ndim, shape=shape, strides=strides, + lst=lst, cast=True) + + def test_memoryview_tolist(self): + + # Most tolist() tests are in self.verify() etc. + + a = array.array('h', list(range(-6, 6))) + m = memoryview(a) + self.assertEqual(m, a) + self.assertEqual(m.tolist(), a.tolist()) + + a = a[2::3] + m = m[2::3] + self.assertEqual(m, a) + self.assertEqual(m.tolist(), a.tolist()) + + ex = array.array('u', 'xyz') + m = memoryview(ex) + self.assertRaises(NotImplementedError, m.tolist) + + ex = ndarray([b'12345'], shape=[1], format="s") + m = memoryview(ex) + self.assertRaises(NotImplementedError, m.tolist) + + ex = ndarray([1,2,3,4,5,6,7,8], shape=[2,4], format="L") + m = memoryview(ex) + self.assertRaises(NotImplementedError, m.tolist) + + def test_memoryview_repr(self): + m = memoryview(bytearray(9)) + r = m.__repr__() + self.assertTrue(r.startswith(" 1. # NOTE: support for multi-dimensional objects is unimplemented. diff -r 92842e347d98 -r 51cedae5acfc Lib/test/test_sys.py --- a/Lib/test/test_sys.py Thu Sep 08 00:56:17 2011 +0200 +++ b/Lib/test/test_sys.py Thu Sep 08 15:02:41 2011 +0200 @@ -770,7 +770,7 @@ check(int(PyLong_BASE**2-1), size(vh) + 2*self.longdigit) check(int(PyLong_BASE**2), size(vh) + 3*self.longdigit) # memory - check(memoryview(b''), size(h + 'PP2P2i7P')) + check(memoryview(b''), size(h + 'PiP4P2i5P3cP')) # module check(unittest, size(h + '3P')) # None diff -r 92842e347d98 -r 51cedae5acfc Modules/_testbuffer.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/Modules/_testbuffer.c Thu Sep 08 15:02:41 2011 +0200 @@ -0,0 +1,2457 @@ +/* C Extension module to test all aspects of PEP-3118. + Written by Stefan Krah. */ + + +#define PY_SSIZE_T_CLEAN + +#include "Python.h" + + +/* struct module */ +PyObject *structmodule = NULL; +PyObject *Struct = NULL; +PyObject *calcsize = NULL; + +/* cache simple format string */ +static const char *simple_fmt = "B"; +PyObject *simple_format = NULL; +#define SIMPLE_FORMAT(fmt) (fmt == NULL || strcmp(fmt, "B") == 0) + + +/**************************************************************************/ +/* NDArray Object */ +/**************************************************************************/ + +static PyTypeObject NDArray_Type; +#define NDArray_Check(v) (Py_TYPE(v) == &NDArray_Type) + +#define CHECK_LIST_OR_TUPLE(v) \ + if (!PyList_Check(v) && !PyTuple_Check(v)) { \ + PyErr_SetString(PyExc_TypeError, \ + #v " must be a list or a tuple"); \ + return NULL; \ + } \ + +#define PyMem_XFree(v) \ + do { if (v) PyMem_Free(v); } while (0) + +/* Check for the presence of suboffsets in the first dimension. */ +#define HAVE_PTR(suboffsets) (suboffsets && suboffsets[0] >= 0) +/* Adjust ptr if suboffsets are present. */ +#define ADJUST_PTR(ptr, suboffsets) \ + (HAVE_PTR(suboffsets) ? *((char**)ptr) + suboffsets[0] : ptr) + +/* User configurable flags for the ndarray */ +#define ND_VAREXPORT 0x001 /* change layout while buffers are exported */ + +/* User configurable flags for each base buffer */ +#define ND_WRITABLE 0x002 /* mark base buffer as writable */ +#define ND_FORTRAN 0x004 /* Fortran contiguous layout */ +#define ND_SCALAR 0x008 /* scalar: ndim = 0 */ +#define ND_PIL 0x010 /* convert to PIL-style array (suboffsets) */ +#define ND_GETBUF_FAIL 0x020 /* test issue 7385 */ + +/* Default: NumPy style (strides), read-only, no var-export, C-style layout */ +#define ND_DEFAULT 0x0 + +/* Internal flags for the base buffer */ +#define ND_C 0x040 /* C contiguous layout (default) */ +#define ND_OWN_ARRAYS 0x080 /* consumer owns arrays */ +#define ND_UNUSED 0x100 /* initializer */ + +/* ndarray properties */ +#define ND_IS_CONSUMER(nd) \ + (((NDArrayObject *)nd)->head == &((NDArrayObject *)nd)->staticbuf) + +/* ndbuf->flags properties */ +#define ND_C_CONTIGUOUS(flags) (!!(flags&(ND_SCALAR|ND_C))) +#define ND_FORTRAN_CONTIGUOUS(flags) (!!(flags&(ND_SCALAR|ND_FORTRAN))) +#define ND_ANY_CONTIGUOUS(flags) (!!(flags&(ND_SCALAR|ND_C|ND_FORTRAN))) + +/* getbuffer() requests */ +#define REQ_INDIRECT(flags) ((flags&PyBUF_INDIRECT) == PyBUF_INDIRECT) +#define REQ_C_CONTIGUOUS(flags) ((flags&PyBUF_C_CONTIGUOUS) == PyBUF_C_CONTIGUOUS) +#define REQ_F_CONTIGUOUS(flags) ((flags&PyBUF_F_CONTIGUOUS) == PyBUF_F_CONTIGUOUS) +#define REQ_ANY_CONTIGUOUS(flags) ((flags&PyBUF_ANY_CONTIGUOUS) == PyBUF_ANY_CONTIGUOUS) +#define REQ_STRIDES(flags) ((flags&PyBUF_STRIDES) == PyBUF_STRIDES) +#define REQ_SHAPE(flags) ((flags&PyBUF_ND) == PyBUF_ND) +#define REQ_WRITABLE(flags) (flags&PyBUF_WRITABLE) +#define REQ_FORMAT(flags) (flags&PyBUF_FORMAT) + + +/* Single node of a list of base buffers. The list is needed to implement + changes in memory layout while exported buffers are active. */ +static PyTypeObject NDArray_Type; + +struct ndbuf; +typedef struct ndbuf { + struct ndbuf *next; + struct ndbuf *prev; + Py_ssize_t len; /* length of data */ + Py_ssize_t offset; /* start of the array relative to data */ + char *data; /* raw data */ + int flags; /* capabilities of the base buffer */ + Py_ssize_t exports; /* number of exports */ + Py_buffer base; /* base buffer */ +} ndbuf_t; + +typedef struct { + PyObject_HEAD + int flags; /* ndarray flags */ + ndbuf_t staticbuf; /* static buffer for re-exporting mode */ + ndbuf_t *head; /* currently active base buffer */ +} NDArrayObject; + + +static ndbuf_t * +ndbuf_new(Py_ssize_t nitems, Py_ssize_t itemsize, Py_ssize_t offset, int flags) +{ + ndbuf_t *ndbuf; + Py_buffer *base; + Py_ssize_t len; + + len = nitems * itemsize; + if (offset % itemsize) { + PyErr_SetString(PyExc_ValueError, + "offset must be a multiple of itemsize"); + return NULL; + } + if (offset < 0 || offset+itemsize > len) { + PyErr_SetString(PyExc_ValueError, "offset out of bounds"); + return NULL; + } + + ndbuf = PyMem_Malloc(sizeof *ndbuf); + if (ndbuf == NULL) { + PyErr_NoMemory(); + return NULL; + } + + ndbuf->next = NULL; + ndbuf->prev = NULL; + ndbuf->len = len; + ndbuf->offset= offset; + + ndbuf->data = PyMem_Malloc(len); + if (ndbuf->data == NULL) { + PyErr_NoMemory(); + PyMem_Free(ndbuf); + return NULL; + } + + ndbuf->flags = flags; + ndbuf->exports = 0; + + base = &ndbuf->base; + base->obj = NULL; + base->buf = ndbuf->data; + base->len = len; + base->itemsize = 1; + base->readonly = 0; + base->format = NULL; + base->ndim = 1; + base->shape = NULL; + base->strides = NULL; + base->suboffsets = NULL; + base->internal = ndbuf; + + return ndbuf; +} + +static void +ndbuf_free(ndbuf_t *ndbuf) +{ + Py_buffer *base = &ndbuf->base; + + PyMem_XFree(ndbuf->data); + PyMem_XFree(base->format); + PyMem_XFree(base->shape); + PyMem_XFree(base->strides); + PyMem_XFree(base->suboffsets); + + PyMem_Free(ndbuf); +} + +static void +ndbuf_push(NDArrayObject *nd, ndbuf_t *elt) +{ + elt->next = nd->head; + if (nd->head) nd->head->prev = elt; + nd->head = elt; + elt->prev = NULL; +} + +static void +ndbuf_delete(NDArrayObject *nd, ndbuf_t *elt) +{ + if (elt->prev) + elt->prev->next = elt->next; + else + nd->head = elt->next; + + if (elt->next) + elt->next->prev = elt->prev; + + ndbuf_free(elt); +} + +static void +ndbuf_pop(NDArrayObject *nd) +{ + ndbuf_delete(nd, nd->head); +} + + +static PyObject * +ndarray_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +{ + NDArrayObject *nd; + + nd = PyObject_New(NDArrayObject, &NDArray_Type); + if (nd == NULL) + return NULL; + + nd->flags = 0; + nd->head = NULL; + return (PyObject *)nd; +} + +static void +ndarray_dealloc(NDArrayObject *self) +{ + if (self->head) { + if (ND_IS_CONSUMER(self)) { + Py_buffer *base = &self->head->base; + if (self->head->flags & ND_OWN_ARRAYS) { + PyMem_XFree(base->shape); + PyMem_XFree(base->strides); + PyMem_XFree(base->suboffsets); + } + PyBuffer_Release(base); + } + else { + while (self->head) + ndbuf_pop(self); + } + } + PyObject_Del(self); +} + +static int +ndarray_init_staticbuf(PyObject *exporter, NDArrayObject *nd, int flags) +{ + Py_buffer *base = &nd->staticbuf.base; + + if (PyObject_GetBuffer(exporter, base, flags) < 0) + return -1; + + nd->head = &nd->staticbuf; + + nd->head->next = NULL; + nd->head->prev = NULL; + nd->head->len = -1; + nd->head->offset = -1; + nd->head->data = NULL; + + nd->head->flags = base->readonly ? 0 : ND_WRITABLE; + nd->head->exports = 0; + + return 0; +} + +static void +init_flags(ndbuf_t *ndbuf) +{ + if (ndbuf->base.ndim == 0) + ndbuf->flags |= ND_SCALAR; + if (ndbuf->base.suboffsets) + ndbuf->flags |= ND_PIL; + if (PyBuffer_IsContiguous(&ndbuf->base, 'C')) + ndbuf->flags |= ND_C; + if (PyBuffer_IsContiguous(&ndbuf->base, 'F')) + ndbuf->flags |= ND_FORTRAN; +} + + +/****************************************************************************/ +/* Buffer/List conversions */ +/****************************************************************************/ + +static Py_ssize_t *strides_from_shape(const ndbuf_t *, int flags); + +/* Get number of members in a struct: see issue #12740 */ +typedef struct { + PyObject_HEAD + Py_ssize_t s_size; + Py_ssize_t s_len; +} PyPartialStructObject; + +static Py_ssize_t +get_nmemb(PyObject *s) +{ + return ((PyPartialStructObject *)s)->s_len; +} + +/* Pack all items into the buffer of 'obj'. The 'format' parameter must be + in struct module syntax. For standard C types, a single item is an integer. + For compound types, a single item is a tuple of integers. */ +static int +pack_from_list(PyObject *obj, PyObject *items, PyObject *format, + Py_ssize_t itemsize) +{ + PyObject *structobj, *pack_into; + PyObject *args, *offset; + PyObject *item, *tmp; + Py_ssize_t nitems; /* number of items */ + Py_ssize_t nmemb; /* number of members in a single item */ + Py_ssize_t i, j; + int ret = 0; + + assert(PyObject_CheckBuffer(obj)); + assert(PyList_Check(items) || PyTuple_Check(items)); + + structobj = PyObject_CallFunctionObjArgs(Struct, format, NULL); + if (structobj == NULL) + return -1; + + nitems = PySequence_Fast_GET_SIZE(items); + nmemb = get_nmemb(structobj); + assert(nmemb >= 1); + + pack_into = PyObject_GetAttrString(structobj, "pack_into"); + if (pack_into == NULL) { + Py_DECREF(structobj); + return -1; + } + + /* nmemb >= 1 */ + args = PyTuple_New(2 + nmemb); + if (args == NULL) { + Py_DECREF(pack_into); + Py_DECREF(structobj); + return -1; + } + + offset = NULL; + for (i = 0; i < nitems; i++) { + /* Loop invariant: args[j] are borrowed references or NULL. */ + PyTuple_SET_ITEM(args, 0, obj); + for (j = 1; j < 2+nmemb; j++) + PyTuple_SET_ITEM(args, j, NULL); + + Py_XDECREF(offset); + offset = PyLong_FromSsize_t(i*itemsize); + if (offset == NULL) { + ret = -1; + break; + } + PyTuple_SET_ITEM(args, 1, offset); + + item = PySequence_Fast_GET_ITEM(items, i); + if ((PyBytes_Check(item) || PyLong_Check(item) || + PyFloat_Check(item)) && nmemb == 1) { + PyTuple_SET_ITEM(args, 2, item); + } + else if ((PyList_Check(item) || PyTuple_Check(item)) && + PySequence_Length(item) == nmemb) { + for (j = 0; j < nmemb; j++) { + tmp = PySequence_Fast_GET_ITEM(item, j); + PyTuple_SET_ITEM(args, 2+j, tmp); + } + } + else { + PyErr_SetString(PyExc_ValueError, + "mismatch between initializer element and format string"); + ret = -1; + break; + } + + tmp = PyObject_CallObject(pack_into, args); + if (tmp == NULL) { + ret = -1; + break; + } + Py_DECREF(tmp); + } + + Py_INCREF(obj); /* args[0] */ + /* args[1]: offset is either NULL or should be dealloc'd */ + for (i = 2; i < 2+nmemb; i++) { + tmp = PyTuple_GET_ITEM(args, i); + Py_XINCREF(tmp); + } + Py_DECREF(args); + + Py_DECREF(pack_into); + Py_DECREF(structobj); + return ret; + +} + +/* Pack single element */ +static int +pack_single(char *ptr, PyObject *item, const char *fmt, Py_ssize_t itemsize) +{ + PyObject *structobj = NULL, *pack_into = NULL, *args = NULL; + PyObject *format = NULL, *mview = NULL, *zero = NULL; + Py_ssize_t i, nmemb; + int ret = -1; + PyObject *x; + + if (fmt == NULL) fmt = "B"; + + format = PyUnicode_FromString(fmt); + if (format == NULL) + goto out; + + structobj = PyObject_CallFunctionObjArgs(Struct, format, NULL); + if (structobj == NULL) + goto out; + + nmemb = get_nmemb(structobj); + assert(nmemb >= 1); + + mview = PyMemoryView_FromMemory(ptr, itemsize, PyBUF_WRITE); + if (mview == NULL) + goto out; + + zero = PyLong_FromLong(0); + if (zero == NULL) + goto out; + + pack_into = PyObject_GetAttrString(structobj, "pack_into"); + if (pack_into == NULL) + goto out; + + args = PyTuple_New(2+nmemb); + if (args == NULL) + goto out; + + PyTuple_SET_ITEM(args, 0, mview); + PyTuple_SET_ITEM(args, 1, zero); + + if ((PyBytes_Check(item) || PyLong_Check(item) || + PyFloat_Check(item)) && nmemb == 1) { + PyTuple_SET_ITEM(args, 2, item); + } + else if ((PyList_Check(item) || PyTuple_Check(item)) && + PySequence_Length(item) == nmemb) { + for (i = 0; i < nmemb; i++) { + x = PySequence_Fast_GET_ITEM(item, i); + PyTuple_SET_ITEM(args, 2+i, x); + } + } + else { + PyErr_SetString(PyExc_ValueError, + "mismatch between initializer element and format string"); + goto args_out; + } + + x = PyObject_CallObject(pack_into, args); + if (x != NULL) { + Py_DECREF(x); + ret = 0; + } + + +args_out: + for (i = 0; i < 2+nmemb; i++) + Py_XINCREF(PyTuple_GET_ITEM(args, i)); + Py_XDECREF(args); +out: + Py_XDECREF(pack_into); + Py_XDECREF(zero); + Py_XDECREF(mview); + Py_XDECREF(structobj); + Py_XDECREF(format); + return ret; +} + +static void +copy_rec(const Py_ssize_t *shape, Py_ssize_t ndim, Py_ssize_t itemsize, + char *dptr, const Py_ssize_t *dstrides, const Py_ssize_t *dsuboffsets, + char *sptr, const Py_ssize_t *sstrides, const Py_ssize_t *ssuboffsets, + char *mem) +{ + Py_ssize_t i; + + assert(ndim >= 1); + + if (ndim == 1) { + if (!HAVE_PTR(dsuboffsets) && !HAVE_PTR(ssuboffsets) && + dstrides[0] == itemsize && sstrides[0] == itemsize) { + memmove(dptr, sptr, shape[0] * itemsize); + } + else { + char *p; + assert(mem != NULL); + for (i=0, p=mem; iformat == NULL && src->format == NULL) || \ + (strcmp(dest->format, src->format) == 0)); + + if (!same_fmt || + dest->itemsize != src->itemsize || + dest->ndim != src->ndim) + return -1; + + for (i = 0; i < dest->ndim; i++) { + if (dest->shape[i] != src->shape[i]) + return -1; + if (dest->shape[i] == 0) + break; + } + + return 0; +} + +/* Copy src to dest. Both buffers must have the same format, itemsize, + ndim and shape. Copying is atomic, the function never fails with + a partial copy. */ +static int +copy_buffer(Py_buffer *dest, Py_buffer *src) +{ + char *mem = NULL; + + assert(dest->ndim > 0); + + if (cmp_structure(dest, src) < 0) { + PyErr_SetString(PyExc_ValueError, + "ndarray assignment: lvalue and rvalue have different structures"); + return -1; + } + + if ((dest->suboffsets && dest->suboffsets[dest->ndim-1] >= 0) || + (src->suboffsets && src->suboffsets[src->ndim-1] >= 0) || + dest->strides[dest->ndim-1] != dest->itemsize || + src->strides[src->ndim-1] != src->itemsize) { + mem = PyMem_Malloc(dest->shape[dest->ndim-1] * dest->itemsize); + if (mem == NULL) { + PyErr_NoMemory(); + return -1; + } + } + + copy_rec(dest->shape, dest->ndim, dest->itemsize, + dest->buf, dest->strides, dest->suboffsets, + src->buf, src->strides, src->suboffsets, + mem); + + PyMem_XFree(mem); + return 0; +} + + +/* Unpack single element */ +static PyObject * +unpack_single(char *ptr, const char *fmt, Py_ssize_t itemsize) +{ + PyObject *x, *unpack_from, *mview; + + if (fmt == NULL) { + fmt = "B"; + itemsize = 1; + } + + unpack_from = PyObject_GetAttrString(structmodule, "unpack_from"); + if (unpack_from == NULL) + return NULL; + + mview = PyMemoryView_FromMemory(ptr, itemsize, PyBUF_READ); + if (mview == NULL) { + Py_DECREF(unpack_from); + return NULL; + } + + x = PyObject_CallFunction(unpack_from, "sO", fmt, mview); + Py_DECREF(unpack_from); + Py_DECREF(mview); + if (x == NULL) + return NULL; + + if (PyTuple_GET_SIZE(x) == 1) { + PyObject *tmp = PyTuple_GET_ITEM(x, 0); + Py_INCREF(tmp); + Py_DECREF(x); + return tmp; + } + + return x; +} + +/* Unpack a multi-dimensional matrix into a nested list. Return a scalar + for ndim = 0. */ +static PyObject * +unpack_rec(PyObject *unpack_from, char *ptr, PyObject *mview, char *item, + const Py_ssize_t *shape, const Py_ssize_t *strides, + const Py_ssize_t *suboffsets, Py_ssize_t ndim, Py_ssize_t itemsize) +{ + PyObject *lst, *x; + Py_ssize_t i; + + assert(ndim >= 0); + assert(shape != NULL); + assert(strides != NULL); + + if (ndim == 0) { + memcpy(item, ptr, itemsize); + x = PyObject_CallFunctionObjArgs(unpack_from, mview, NULL); + if (x == NULL) + return NULL; + if (PyTuple_GET_SIZE(x) == 1) { + PyObject *tmp = PyTuple_GET_ITEM(x, 0); + Py_INCREF(tmp); + Py_DECREF(x); + return tmp; + } + return x; + } + + lst = PyList_New(shape[0]); + if (lst == NULL) + return NULL; + + for (i = 0; i < shape[0]; ptr+=strides[0], i++) { + char *nextptr = ADJUST_PTR(ptr, suboffsets); + + x = unpack_rec(unpack_from, nextptr, mview, item, + shape+1, strides+1, suboffsets ? suboffsets+1 : NULL, + ndim-1, itemsize); + if (x == NULL) { + Py_DECREF(lst); + return NULL; + } + + PyList_SET_ITEM(lst, i, x); + } + + return lst; +} + + +static PyObject * +ndarray_as_list(NDArrayObject *nd) +{ + PyObject *structobj = NULL, *unpack_from = NULL; + PyObject *lst = NULL, *mview = NULL; + Py_buffer *base = &nd->head->base; + Py_ssize_t *shape = base->shape; + Py_ssize_t *strides = base->strides; + Py_ssize_t simple_shape[1]; + Py_ssize_t simple_strides[1]; + char *item = NULL; + PyObject *format; + char *fmt = base->format; + + base = &nd->head->base; + + if (fmt == NULL) { + PyErr_SetString(PyExc_ValueError, + "ndarray: tolist() does not support format=NULL, use " + "tobytes()"); + return NULL; + } + if (shape == NULL) { + assert(ND_C_CONTIGUOUS(nd->head->flags)); + assert(base->strides == NULL); + assert(base->ndim <= 1); + shape = simple_shape; + shape[0] = base->len; + strides = simple_strides; + strides[0] = base->itemsize; + } + else if (strides == NULL) { + assert(ND_C_CONTIGUOUS(nd->head->flags)); + strides = strides_from_shape(nd->head, 0); + if (strides == NULL) + return NULL; + } + + format = PyUnicode_FromString(fmt); + if (format == NULL) + goto out; + + structobj = PyObject_CallFunctionObjArgs(Struct, format, NULL); + Py_DECREF(format); + if (structobj == NULL) + goto out; + + unpack_from = PyObject_GetAttrString(structobj, "unpack_from"); + if (unpack_from == NULL) + goto out; + + item = PyMem_Malloc(base->itemsize); + if (item == NULL) { + PyErr_NoMemory(); + goto out; + } + + mview = PyMemoryView_FromMemory(item, base->itemsize, PyBUF_WRITE); + if (mview == NULL) + goto out; + + lst = unpack_rec(unpack_from, base->buf, mview, item, + shape, strides, base->suboffsets, + base->ndim, base->itemsize); + +out: + Py_XDECREF(mview); + PyMem_XFree(item); + Py_XDECREF(unpack_from); + Py_XDECREF(structobj); + if (strides != base->strides && strides != simple_strides) + PyMem_XFree(strides); + + return lst; +} + + +/****************************************************************************/ +/* Initialize ndbuf */ +/****************************************************************************/ + +/* + State of a new ndbuf during initialization. 'OK' means that initialization + is complete. 'PTR' means that a pointer has been initialized, but the + state of the memory is still undefined and ndbuf->offset is disregarded. + + +-----------------+-----------+-------------+----------------+ + | | ndbuf_new | init_simple | init_structure | + +-----------------+-----------+-------------+----------------+ + | next | OK (NULL) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | prev | OK (NULL) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | len | OK | OK | OK | + +-----------------+-----------+-------------+----------------+ + | offset | OK | OK | OK | + +-----------------+-----------+-------------+----------------+ + | data | PTR | OK | OK | + +-----------------+-----------+-------------+----------------+ + | flags | user | user | OK | + +-----------------+-----------+-------------+----------------+ + | exports | OK (0) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.obj | OK (NULL) | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.buf | PTR | PTR | OK | + +-----------------+-----------+-------------+----------------+ + | base.len | len(data) | len(data) | OK | + +-----------------+-----------+-------------+----------------+ + | base.itemsize | 1 | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.readonly | 0 | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.format | NULL | OK | OK | + +-----------------+-----------+-------------+----------------+ + | base.ndim | 1 | 1 | OK | + +-----------------+-----------+-------------+----------------+ + | base.shape | NULL | NULL | OK | + +-----------------+-----------+-------------+----------------+ + | base.strides | NULL | NULL | OK | + +-----------------+-----------+-------------+----------------+ + | base.suboffsets | NULL | NULL | OK | + +-----------------+-----------+-------------+----------------+ + | base.internal | OK | OK | OK | + +-----------------+-----------+-------------+----------------+ + +*/ + +static Py_ssize_t +get_itemsize(PyObject *format) +{ + PyObject *tmp; + Py_ssize_t itemsize; + + tmp = PyObject_CallFunctionObjArgs(calcsize, format, NULL); + if (tmp == NULL) + return -1; + itemsize = PyLong_AsSsize_t(tmp); + Py_DECREF(tmp); + + return itemsize; +} + +static char * +get_format(PyObject *format) +{ + PyObject *tmp; + char *fmt; + + tmp = PyUnicode_AsASCIIString(format); + if (tmp == NULL) + return NULL; + fmt = PyMem_Malloc(PyBytes_GET_SIZE(tmp)+1); + if (fmt == NULL) { + PyErr_NoMemory(); + Py_DECREF(tmp); + return NULL; + } + strcpy(fmt, PyBytes_AS_STRING(tmp)); + Py_DECREF(tmp); + + return fmt; +} + +static int +init_simple(ndbuf_t *ndbuf, PyObject *items, PyObject *format, + Py_ssize_t itemsize) +{ + PyObject *mview; + Py_buffer *base = &ndbuf->base; + int ret; + + mview = PyMemoryView_FromBuffer(base); + if (mview == NULL) + return -1; + + ret = pack_from_list(mview, items, format, itemsize); + Py_DECREF(mview); + if (ret < 0) + return -1; + + base->readonly = !(ndbuf->flags & ND_WRITABLE); + base->itemsize = itemsize; + base->format = get_format(format); + if (base->format == NULL) + return -1; + + return 0; +} + +static Py_ssize_t * +seq_as_ssize_array(PyObject *seq, Py_ssize_t len, int is_shape) +{ + Py_ssize_t *dest; + Py_ssize_t x, i; + + dest = PyMem_Malloc(len * (sizeof *dest)); + if (dest == NULL) { + PyErr_NoMemory(); + return NULL; + } + + for (i = 0; i < len; i++) { + PyObject *tmp = PySequence_Fast_GET_ITEM(seq, i); + if (!PyLong_Check(tmp)) { + PyErr_Format(PyExc_ValueError, + "elements of %s must be integers", + is_shape ? "shape" : "strides"); + PyMem_Free(dest); + return NULL; + } + x = PyLong_AsSsize_t(tmp); + if (PyErr_Occurred()) { + PyMem_Free(dest); + return NULL; + } + if (is_shape && x < 0) { + PyErr_Format(PyExc_ValueError, + "elements of shape must be integers >= 0"); + PyMem_Free(dest); + return NULL; + } + dest[i] = x; + } + + return dest; +} + +static Py_ssize_t * +strides_from_shape(const ndbuf_t *ndbuf, int flags) +{ + const Py_buffer *base = &ndbuf->base; + Py_ssize_t *s, i; + char layout; + + s = PyMem_Malloc(base->ndim * (sizeof *s)); + if (s == NULL) { + PyErr_NoMemory(); + return NULL; + } + + if (flags & ND_FORTRAN) { + s[0] = base->itemsize; + for (i = 1; i < base->ndim; i++) + s[i] = s[i-1] * base->shape[i-1]; + layout = 'F'; + } + else { + s[base->ndim-1] = base->itemsize; + for (i = base->ndim-2; i >= 0; i--) + s[i] = s[i+1] * base->shape[i+1]; + layout = 'C'; + } + + return s; +} + +/* Bounds check: + + len := complete length of allocated memory + offset := start of the array + + A single array element is indexed by: + + i = indices[0] * strides[0] + indices[1] * strides[1] + ... + + imin is reached when all indices[n] combined with positive strides are 0 + and all indices combined with negative strides are shape[n]-1, which is + the maximum index for the nth dimension. + + imax is reached when all indices[n] combined with negative strides are 0 + and all indices combined with positive strides are shape[n]-1. +*/ +static int +verify_structure(Py_ssize_t len, Py_ssize_t itemsize, Py_ssize_t offset, + const Py_ssize_t *shape, const Py_ssize_t *strides, + Py_ssize_t ndim) +{ + Py_ssize_t imin, imax; + Py_ssize_t n; + + assert(ndim >= 0); + + if (ndim == 0 && (offset < 0 || offset+itemsize > len)) + goto invalid_combination; + + for (n = 0; n < ndim; n++) + if (strides[n] % itemsize) { + PyErr_SetString(PyExc_ValueError, + "strides must be a multiple of itemsize"); + return -1; + } + + for (n = 0; n < ndim; n++) + if (shape[n] == 0) + return 0; + + imin = imax = 0; + for (n = 0; n < ndim; n++) + if (strides[n] <= 0) + imin += (shape[n]-1) * strides[n]; + else + imax += (shape[n]-1) * strides[n]; + + if (imin + offset < 0 || imax + offset + itemsize > len) + goto invalid_combination; + + return 0; + + +invalid_combination: + PyErr_SetString(PyExc_ValueError, + "invalid combination of buffer, shape and strides"); + return -1; +} + +/* + Convert a NumPy-style array to an array using suboffsets to stride in + the first dimension. Requirements: ndim > 0. + + Contiguous example + ================== + + Input: + ------ + shape = {2, 2, 3}; + strides = {6, 3, 1}; + suboffsets = NULL; + data = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}; + buf = &data[0] + + Output: + ------- + shape = {2, 2, 3}; + strides = {sizeof(char *), 3, 1}; + suboffsets = {0, -1, -1}; + data = {p1, p2, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}; + | | ^ ^ + `---'---' | + | | + `---------------------' + buf = &data[0] + + So, in the example the input resembles the three-dimensional array + char v[2][2][3], while the output resembles an array of two pointers + to two-dimensional arrays: char (*v[2])[2][3]. + + + Non-contiguous example: + ======================= + + Input (with offset and negative strides): + ----------------------------------------- + shape = {2, 2, 3}; + strides = {-6, 3, -1}; + offset = 8 + suboffsets = NULL; + data = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}; + + Output: + ------- + shape = {2, 2, 3}; + strides = {-sizeof(char *), 3, -1}; + suboffsets = {2, -1, -1}; + newdata = {p1, p2, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}; + | | ^ ^ ^ ^ + `---'---' | | `- p2+suboffsets[0] + | `-----------|--- p1+suboffsets[0] + `---------------------' + buf = &newdata[1] # striding backwards over the pointers. + + suboffsets[0] is the same as the offset that one would specify if + the two {2, 3} subarrays were created directly, hence the name. +*/ +static int +init_suboffsets(ndbuf_t *ndbuf) +{ + Py_buffer *base = &ndbuf->base; + Py_ssize_t start, step; + Py_ssize_t imin, suboffset0; + Py_ssize_t addsize; + Py_ssize_t n; + char *data; + + assert(base->ndim > 0); + assert(base->suboffsets == NULL); + + /* Allocate new data with additional space for shape[0] pointers. */ + addsize = base->shape[0] * (sizeof (char *)); + + /* Align array start to a multiple of 8. */ + addsize = 8 * ((addsize + 7) / 8); + + data = PyMem_Malloc(ndbuf->len + addsize); + if (data == NULL) { + PyErr_NoMemory(); + return -1; + } + + memcpy(data + addsize, ndbuf->data, ndbuf->len); + + PyMem_Free(ndbuf->data); + ndbuf->data = data; + ndbuf->len += addsize; + base->buf = ndbuf->data; + + /* imin: minimum index of the input array relative to ndbuf->offset. + suboffset0: offset for each sub-array of the output. This is the + same as calculating -imin' for a sub-array of ndim-1. */ + imin = suboffset0 = 0; + for (n = 0; n < base->ndim; n++) { + if (base->shape[n] == 0) + break; + if (base->strides[n] <= 0) { + Py_ssize_t x = (base->shape[n]-1) * base->strides[n]; + imin += x; + suboffset0 += (n >= 1) ? -x : 0; + } + } + + /* Initialize the array of pointers to the sub-arrays. */ + start = addsize + ndbuf->offset + imin; + step = base->strides[0] < 0 ? -base->strides[0] : base->strides[0]; + + for (n = 0; n < base->shape[0]; n++) + ((char **)base->buf)[n] = (char *)base->buf + start + n*step; + + /* Initialize suboffsets. */ + base->suboffsets = PyMem_Malloc(base->ndim * (sizeof *base->suboffsets)); + if (base->suboffsets == NULL) { + PyErr_NoMemory(); + return -1; + } + base->suboffsets[0] = suboffset0; + for (n = 1; n < base->ndim; n++) + base->suboffsets[n] = -1; + + /* Adjust strides for the first (zeroth) dimension. */ + if (base->strides[0] >= 0) { + base->strides[0] = sizeof(char *); + } + else { + /* Striding backwards. */ + base->strides[0] = -(Py_ssize_t)sizeof(char *); + if (base->shape[0] > 0) + base->buf = (char *)base->buf + (base->shape[0]-1) * sizeof(char *); + } + + ndbuf->flags &= ~(ND_C|ND_FORTRAN); + ndbuf->offset = 0; + return 0; +} + +static void +init_len(Py_buffer *base) +{ + Py_ssize_t i; + + base->len = 1; + for (i = 0; i < base->ndim; i++) + base->len *= base->shape[i]; + base->len *= base->itemsize; +} + +static int +init_structure(ndbuf_t *ndbuf, PyObject *shape, PyObject *strides, + Py_ssize_t ndim) +{ + Py_buffer *base = &ndbuf->base; + + base->ndim = (int)ndim; + if (ndim == 0) { + if (ndbuf->flags & ND_PIL) { + PyErr_SetString(PyExc_TypeError, + "ndim = 0 cannot be used in conjunction with ND_PIL"); + return -1; + } + ndbuf->flags |= (ND_SCALAR|ND_C|ND_FORTRAN); + return 0; + } + + /* shape */ + base->shape = seq_as_ssize_array(shape, ndim, 1); + if (base->shape == NULL) + return -1; + + /* strides */ + if (strides) { + base->strides = seq_as_ssize_array(strides, ndim, 0); + } + else { + base->strides = strides_from_shape(ndbuf, ndbuf->flags); + } + if (base->strides == NULL) + return -1; + if (verify_structure(base->len, base->itemsize, ndbuf->offset, + base->shape, base->strides, ndim) < 0) + return -1; + + /* buf */ + base->buf = ndbuf->data + ndbuf->offset; + + /* len */ + init_len(base); + + /* ndbuf->flags */ + if (PyBuffer_IsContiguous(base, 'C')) + ndbuf->flags |= ND_C; + if (PyBuffer_IsContiguous(base, 'F')) + ndbuf->flags |= ND_FORTRAN; + + + /* convert numpy array to suboffset representation */ + if (ndbuf->flags & ND_PIL) { + /* modifies base->buf, base->strides and base->suboffsets **/ + return init_suboffsets(ndbuf); + } + + return 0; +} + +static ndbuf_t * +init_ndbuf(PyObject *items, PyObject *shape, PyObject *strides, + Py_ssize_t offset, PyObject *format, int flags) +{ + ndbuf_t *ndbuf; + Py_ssize_t ndim; + Py_ssize_t nitems; + Py_ssize_t itemsize; + + /* ndim = len(shape) */ + CHECK_LIST_OR_TUPLE(shape) + ndim = PySequence_Fast_GET_SIZE(shape); + if (ndim > INT_MAX) { + PyErr_SetString(PyExc_ValueError, "ndim must not exceed INT_MAX"); + return NULL; + } + + /* len(strides) = len(shape) */ + if (strides) { + CHECK_LIST_OR_TUPLE(strides) + if (PySequence_Fast_GET_SIZE(strides) == 0) + strides = NULL; + else if (flags & ND_FORTRAN) { + PyErr_SetString(PyExc_TypeError, + "ND_FORTRAN cannot be used together with strides"); + return NULL; + } + else if (PySequence_Fast_GET_SIZE(strides) != ndim) { + PyErr_SetString(PyExc_ValueError, + "len(shape) != len(strides)"); + return NULL; + } + } + + /* itemsize */ + itemsize = get_itemsize(format); + if (itemsize <= 0) { + if (itemsize == 0) { + PyErr_SetString(PyExc_ValueError, + "itemsize must not be zero"); + } + return NULL; + } + + /* convert scalar to list */ + if (ndim == 0) { + items = Py_BuildValue("(O)", items); + if (items == NULL) + return NULL; + } + else { + CHECK_LIST_OR_TUPLE(items) + Py_INCREF(items); + } + + /* number of items */ + nitems = PySequence_Fast_GET_SIZE(items); + if (nitems == 0) { + PyErr_SetString(PyExc_ValueError, + "initializer list or tuple must not be empty"); + Py_DECREF(items); + return NULL; + } + + ndbuf = ndbuf_new(nitems, itemsize, offset, flags); + if (ndbuf == NULL) { + Py_DECREF(items); + return NULL; + } + + + if (init_simple(ndbuf, items, format, itemsize) < 0) + goto error; + if (init_structure(ndbuf, shape, strides, ndim) < 0) + goto error; + + Py_DECREF(items); + return ndbuf; + +error: + Py_DECREF(items); + ndbuf_free(ndbuf); + return NULL; +} + +/* initialize and push a new base onto the linked list */ +static int +ndarray_push_base(NDArrayObject *nd, PyObject *items, + PyObject *shape, PyObject *strides, + Py_ssize_t offset, PyObject *format, int flags) +{ + ndbuf_t *ndbuf; + + ndbuf = init_ndbuf(items, shape, strides, offset, format, flags); + if (ndbuf == NULL) + return -1; + + ndbuf_push(nd, ndbuf); + return 0; +} + +#define PyBUF_UNUSED 0x10000 +static int +ndarray_init(PyObject *self, PyObject *args, PyObject *kwds) +{ + NDArrayObject *nd = (NDArrayObject *)self; + static char *kwlist[] = { + "obj", "shape", "strides", "offset", "format", "flags", "getbuf", NULL + }; + PyObject *v = NULL; /* initializer: scalar, list, tuple or base object */ + PyObject *shape = NULL; /* size of each dimension */ + PyObject *strides = NULL; /* number of bytes to the next elt in each dim */ + Py_ssize_t offset = 0; /* buffer offset */ + PyObject *format = simple_format; /* struct module specifier: "B" */ + int flags = ND_UNUSED; /* base buffer and ndarray flags */ + + int getbuf = PyBUF_UNUSED; /* re-exporter: getbuffer request flags */ + + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|OOnOii", kwlist, + &v, &shape, &strides, &offset, &format, &flags, &getbuf)) + return -1; + + /* NDArrayObject is re-exporter */ + if (PyObject_CheckBuffer(v) && shape == NULL) { + if (strides || offset || format != simple_format || + flags != ND_UNUSED) { + PyErr_SetString(PyExc_TypeError, + "construction from exporter object only takes a single " + "additional getbuf argument"); + return -1; + } + + getbuf = (getbuf == PyBUF_UNUSED) ? PyBUF_FULL_RO : getbuf; + + if (ndarray_init_staticbuf(v, nd, getbuf) < 0) + return -1; + + init_flags(nd->head); + + return 0; + } + + /* NDArrayObject is the original base object. */ + if (getbuf != PyBUF_UNUSED) { + PyErr_SetString(PyExc_TypeError, + "getbuf argument only valid for construction from exporter " + "object"); + return -1; + } + if (shape == NULL) { + PyErr_SetString(PyExc_TypeError, + "shape is a required argument when constructing from " + "list, tuple or scalar"); + return -1; + } + + if (flags == ND_UNUSED) + flags = ND_DEFAULT; + if (flags & ND_VAREXPORT) { + nd->flags |= ND_VAREXPORT; + flags &= ~ND_VAREXPORT; + } + + /* Initialize and push the first base buffer onto the linked list. */ + return ndarray_push_base(nd, v, shape, strides, offset, format, flags); +} + +/* Push an additional base onto the linked list. */ +static PyObject * +ndarray_push(PyObject *self, PyObject *args, PyObject *kwds) +{ + NDArrayObject *nd = (NDArrayObject *)self; + static char *kwlist[] = { + "items", "shape", "strides", "offset", "format", "flags", NULL + }; + PyObject *items = NULL; /* initializer: scalar, list or tuple */ + PyObject *shape = NULL; /* size of each dimension */ + PyObject *strides = NULL; /* number of bytes to the next elt in each dim */ + PyObject *format = simple_format; /* struct module specifier: "B" */ + Py_ssize_t offset = 0; /* buffer offset */ + int flags = ND_UNUSED; /* base buffer flags */ + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO|OnOi", kwlist, + &items, &shape, &strides, &offset, &format, &flags)) + return NULL; + + if (flags & ND_VAREXPORT) { + PyErr_SetString(PyExc_ValueError, + "ND_VAREXPORT flag can only be used during object creation"); + return NULL; + } + if (ND_IS_CONSUMER(nd)) { + PyErr_SetString(PyExc_BufferError, + "structure of re-exporting object is immutable"); + return NULL; + } + if (!(nd->flags&ND_VAREXPORT) && nd->head->exports > 0) { + PyErr_Format(PyExc_BufferError, + "cannot change structure: %zd exported buffer%s", + nd->head->exports, nd->head->exports==1 ? "" : "s"); + return NULL; + } + + if (ndarray_push_base(nd, items, shape, strides, + offset, format, flags) < 0) + return NULL; + Py_RETURN_NONE; +} + +/* Pop a base from the linked list (if possible). */ +static PyObject * +ndarray_pop(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + if (ND_IS_CONSUMER(nd)) { + PyErr_SetString(PyExc_BufferError, + "structure of re-exporting object is immutable"); + return NULL; + } + if (nd->head->exports > 0) { + PyErr_Format(PyExc_BufferError, + "cannot change structure: %zd exported buffer%s", + nd->head->exports, nd->head->exports==1 ? "" : "s"); + return NULL; + } + if (nd->head->next == NULL) { + PyErr_SetString(PyExc_BufferError, + "list only has a single base"); + return NULL; + } + + ndbuf_pop(nd); + Py_RETURN_NONE; +} + +/**************************************************************************/ +/* getbuffer */ +/**************************************************************************/ + +static int +ndarray_getbuf(NDArrayObject *self, Py_buffer *view, int flags) +{ + ndbuf_t *ndbuf = self->head; + Py_buffer *base = &ndbuf->base; + int baseflags = ndbuf->flags; + + /* start with complete information */ + *view = *base; + view->obj = NULL; + + /* reconstruct format */ + if (view->format == NULL) + view->format = "B"; + + if (base->ndim != 0 && + ((REQ_SHAPE(flags) && base->shape == NULL) || + (REQ_STRIDES(flags) && base->strides == NULL))) { + /* The ndarray is a re-exporter that has been created without full + information for testing purposes. In this particular case the + ndarray is not a PEP-3118 compliant buffer provider. */ + PyErr_SetString(PyExc_BufferError, + "re-exporter does not provide format, shape or strides"); + return -1; + } + + if (baseflags & ND_GETBUF_FAIL) { + PyErr_SetString(PyExc_BufferError, + "ND_GETBUF_FAIL: forced test exception"); + return -1; + } + + if (REQ_WRITABLE(flags) && base->readonly) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not writable"); + return -1; + } + if (!REQ_FORMAT(flags)) { + /* NULL indicates that the buffer's data type has been cast to 'B'. + view->itemsize is the _previous_ itemsize. If shape is present, + the equality product(shape) * itemsize = len still holds at this + point. The equality calcsize(format) = itemsize does _not_ hold + from here on! */ + view->format = NULL; + } + + if (REQ_C_CONTIGUOUS(flags) && !ND_C_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not C-contiguous"); + return -1; + } + if (REQ_F_CONTIGUOUS(flags) && !ND_FORTRAN_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not Fortran contiguous"); + return -1; + } + if (REQ_ANY_CONTIGUOUS(flags) && !ND_ANY_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not contiguous"); + return -1; + } + if (!REQ_INDIRECT(flags) && (baseflags & ND_PIL)) { + PyErr_SetString(PyExc_BufferError, + "ndarray cannot be represented without suboffsets"); + return -1; + } + if (!REQ_STRIDES(flags)) { + if (!ND_C_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "ndarray is not C-contiguous"); + return -1; + } + view->strides = NULL; + } + if (!REQ_SHAPE(flags)) { + /* PyBUF_SIMPLE or PyBUF_WRITABLE: at this point buf is C-contiguous, + so base->buf = ndbuf->data. */ + if (view->format != NULL) { + /* PyBUF_SIMPLE|PyBUF_FORMAT and PyBUF_WRITABLE|PyBUF_FORMAT do + not make sense. */ + PyErr_Format(PyExc_BufferError, + "ndarray: cannot cast to unsigned bytes if the format flag " + "is present"); + return -1; + } + /* product(shape) * itemsize = len and calcsize(format) = itemsize + do _not_ hold from here on! */ + view->ndim = 1; + view->shape = NULL; + } + + view->obj = (PyObject *)self; + Py_INCREF(view->obj); + self->head->exports++; + + return 0; +} + +static int +ndarray_releasebuf(NDArrayObject *self, Py_buffer *view) +{ + if (!ND_IS_CONSUMER(self)) { + ndbuf_t *ndbuf = view->internal; + if (--ndbuf->exports == 0 && ndbuf != self->head) + ndbuf_delete(self, ndbuf); + } + + return 0; +} + +static PyBufferProcs ndarray_as_buffer = { + (getbufferproc)ndarray_getbuf, /* bf_getbuffer */ + (releasebufferproc)ndarray_releasebuf /* bf_releasebuffer */ +}; + + +/**************************************************************************/ +/* indexing/slicing */ +/**************************************************************************/ + +static char * +ptr_from_index(Py_buffer *base, Py_ssize_t index) +{ + char *ptr; + Py_ssize_t nitems; /* items in the first dimension */ + + if (base->shape) + nitems = base->shape[0]; + else { + assert(base->ndim == 1 && SIMPLE_FORMAT(base->format)); + nitems = base->len; + } + + if (index < 0) { + index += nitems; + } + if (index < 0 || index >= nitems) { + PyErr_SetString(PyExc_IndexError, "index out of bounds"); + return NULL; + } + + ptr = (char *)base->buf; + + if (base->strides == NULL) + ptr += base->itemsize * index; + else + ptr += base->strides[0] * index; + + ptr = ADJUST_PTR(ptr, base->suboffsets); + + return ptr; +} + +static PyObject * +ndarray_item(NDArrayObject *self, Py_ssize_t index) +{ + ndbuf_t *ndbuf = self->head; + Py_buffer *base = &ndbuf->base; + char *ptr; + + if (base->ndim == 0) { + PyErr_SetString(PyExc_TypeError, "invalid indexing of scalar"); + return NULL; + } + + ptr = ptr_from_index(base, index); + if (ptr == NULL) + return NULL; + + if (base->ndim == 1) { + return unpack_single(ptr, base->format, base->itemsize); + } + else { + NDArrayObject *nd; + Py_buffer *subview; + + nd = (NDArrayObject *)ndarray_new(&NDArray_Type, NULL, NULL); + if (nd == NULL) + return NULL; + + if (ndarray_init_staticbuf((PyObject *)self, nd, PyBUF_FULL_RO) < 0) { + Py_DECREF(nd); + return NULL; + } + + subview = &nd->staticbuf.base; + + subview->buf = ptr; + subview->len /= subview->shape[0]; + + subview->ndim--; + subview->shape++; + if (subview->strides) subview->strides++; + if (subview->suboffsets) subview->suboffsets++; + + init_flags(&nd->staticbuf); + + return (PyObject *)nd; + } +} + +/* + For each dimension, we get valid (start, stop, step, slicelength) quadruples + from PySlice_GetIndicesEx(). + + Slicing NumPy arrays + ==================== + + A pointer to an element in a NumPy array is defined by: + + ptr = (char *)buf + indices[0] * strides[0] + + ... + + indices[ndim-1] * strides[ndim-1] + + Adjust buf: + ----------- + Adding start[n] for each dimension effectively adds the constant: + + c = start[0] * strides[0] + ... + start[ndim-1] * strides[ndim-1] + + Therefore init_slice() adds all start[n] directly to buf. + + Adjust shape: + ------------- + Obviously shape[n] = slicelength[n] + + Adjust strides: + --------------- + In the original array, the next element in a dimension is reached + by adding strides[n] to the pointer. In the sliced array, elements + may be skipped, so the next element is reached by adding: + + strides[n] * step[n] + + Slicing PIL arrays + ================== + + Layout: + ------- + In the first (zeroth) dimension, PIL arrays have an array of pointers + to sub-arrays of ndim-1. Striding in the first dimension is done by + getting the index of the nth pointer, dereference it and then add a + suboffset to it. The arrays pointed to can best be seen a regular + NumPy arrays. + + Adjust buf: + ----------- + In the original array, buf points to a location (usually the start) + in the array of pointers. For the sliced array, start[0] can be + added to buf in the same manner as for NumPy arrays. + + Adjust suboffsets: + ------------------ + Due to the dereferencing step in the addressing scheme, it is not + possible to adjust buf for higher dimensions. Recall that the + sub-arrays pointed to are regular NumPy arrays, so for each of + those arrays adding start[n] effectively adds the constant: + + c = start[1] * strides[1] + ... + start[ndim-1] * strides[ndim-1] + + This constant is added to suboffsets[0]. suboffsets[0] in turn is + added to each pointer right after dereferencing. + + Adjust shape and strides: + ------------------------- + Shape and strides are not influenced by the dereferencing step, so + they are adjusted in the same manner as for NumPy arrays. + + Multiple levels of suboffsets + ============================= + + For a construct like an array of pointers to array of pointers to + sub-arrays of ndim-2: + + suboffsets[0] = start[1] * strides[1] + suboffsets[1] = start[2] * strides[2] + ... +*/ +static int +init_slice(Py_buffer *base, PyObject *key, int dim) +{ + Py_ssize_t start, stop, step, slicelength; + + if (PySlice_GetIndicesEx(key, base->shape[dim], + &start, &stop, &step, &slicelength) < 0) { + return -1; + } + + + if (base->suboffsets == NULL || dim == 0) { + adjust_buf: + base->buf = (char *)base->buf + base->strides[dim] * start; + } + else { + Py_ssize_t n = dim-1; + while (n >= 0 && base->suboffsets[n] < 0) + n--; + if (n < 0) + goto adjust_buf; /* all suboffsets are negative */ + base->suboffsets[n] = base->suboffsets[n] + base->strides[dim] * start; + } + base->shape[dim] = slicelength; + base->strides[dim] = base->strides[dim] * step; + + return 0; +} + +static int +copy_structure(Py_buffer *base) +{ + Py_ssize_t *shape = NULL, *strides = NULL, *suboffsets = NULL; + Py_ssize_t i; + + shape = PyMem_Malloc(base->ndim * (sizeof *shape)); + strides = PyMem_Malloc(base->ndim * (sizeof *strides)); + if (shape == NULL || strides == NULL) + goto err_nomem; + + suboffsets = NULL; + if (base->suboffsets) { + suboffsets = PyMem_Malloc(base->ndim * (sizeof *suboffsets)); + if (suboffsets == NULL) + goto err_nomem; + } + + for (i = 0; i < base->ndim; i++) { + shape[i] = base->shape[i]; + strides[i] = base->strides[i]; + if (suboffsets) + suboffsets[i] = base->suboffsets[i]; + } + + base->shape = shape; + base->strides = strides; + base->suboffsets = suboffsets; + + return 0; + +err_nomem: + PyErr_NoMemory(); + PyMem_XFree(shape); + PyMem_XFree(strides); + PyMem_XFree(suboffsets); + return -1; +} + +static PyObject * +ndarray_subscript(NDArrayObject *self, PyObject *key) +{ + NDArrayObject *nd; + ndbuf_t *ndbuf; + Py_buffer *base = &self->head->base; + + if (base->ndim == 0) { + if (PyTuple_Check(key) && PyTuple_GET_SIZE(key) == 0) { + return unpack_single(base->buf, base->format, base->itemsize); + } + else if (key == Py_Ellipsis) { + Py_INCREF(self); + return (PyObject *)self; + } + else { + PyErr_SetString(PyExc_TypeError, "invalid indexing of scalar"); + return NULL; + } + } + if (PyIndex_Check(key)) { + Py_ssize_t index = PyLong_AsSsize_t(key); + if (index == -1 && PyErr_Occurred()) + return NULL; + return ndarray_item(self, index); + } + + nd = (NDArrayObject *)ndarray_new(&NDArray_Type, NULL, NULL); + if (nd == NULL) + return NULL; + + /* new ndarray is a consumer */ + if (ndarray_init_staticbuf((PyObject *)self, nd, PyBUF_FULL_RO) < 0) { + Py_DECREF(nd); + return NULL; + } + + /* copy shape, strides and suboffsets */ + ndbuf = nd->head; + base = &ndbuf->base; + if (copy_structure(base) < 0) { + Py_DECREF(nd); + return NULL; + } + ndbuf->flags |= ND_OWN_ARRAYS; + + if (PySlice_Check(key)) { + /* one-dimensional slice */ + if (init_slice(base, key, 0) < 0) + goto err_occurred; + } + else if PyTuple_Check(key) { + /* multi-dimensional slice */ + PyObject *tuple = key; + Py_ssize_t i, n; + + n = PyTuple_GET_SIZE(tuple); + for (i = 0; i < n; i++) { + key = PyTuple_GET_ITEM(tuple, i); + if (!PySlice_Check(key)) + goto type_error; + if (init_slice(base, key, (int)i) < 0) + goto err_occurred; + } + } + else { + goto type_error; + } + + init_len(base); + init_flags(ndbuf); + + return (PyObject *)nd; + + +type_error: + PyErr_Format(PyExc_TypeError, + "cannot index memory using \"%.200s\"", + key->ob_type->tp_name); +err_occurred: + Py_DECREF(nd); + return NULL; +} + + +static int +ndarray_ass_subscript(NDArrayObject *self, PyObject *key, PyObject *value) +{ + NDArrayObject *nd; + Py_buffer *dest = &self->head->base; + Py_buffer src; + char *ptr; + Py_ssize_t index; + int ret = -1; + + if (dest->readonly) { + PyErr_SetString(PyExc_TypeError, "ndarray is not writable"); + return -1; + } + if (value == NULL) { + PyErr_SetString(PyExc_TypeError, "ndarray data cannot be deleted"); + return -1; + } + if (dest->ndim == 0) { + if (key == Py_Ellipsis || + (PyTuple_Check(key) && PyTuple_GET_SIZE(key) == 0)) { + ptr = (char *)dest->buf; + return pack_single(ptr, value, dest->format, dest->itemsize); + } + else { + PyErr_SetString(PyExc_TypeError, "invalid indexing of scalar"); + return -1; + } + } + if (dest->ndim == 1 && PyIndex_Check(key)) { + /* rvalue must be a single item */ + index = PyLong_AsSsize_t(key); + if (index == -1 && PyErr_Occurred()) + return -1; + else { + ptr = ptr_from_index(dest, index); + if (ptr == NULL) + return -1; + } + return pack_single(ptr, value, dest->format, dest->itemsize); + } + + /* rvalue must be an exporter */ + if (PyObject_GetBuffer(value, &src, PyBUF_FULL_RO) == -1) + return -1; + + nd = (NDArrayObject *)ndarray_subscript(self, key); + if (nd != NULL) { + dest = &nd->head->base; + ret = copy_buffer(dest, &src); + Py_DECREF(nd); + } + + PyBuffer_Release(&src); + return ret; +} + +static PyObject * +slice_indices(PyObject *self, PyObject *args) +{ + PyObject *ret, *key, *tmp; + Py_ssize_t s[4]; /* start, stop, step, slicelength */ + Py_ssize_t i, len; + + if (!PyArg_ParseTuple(args, "On", &key, &len)) { + return NULL; + } + if (PySlice_GetIndicesEx(key, len, &s[0], &s[1], &s[2], &s[3]) < 0) { + return NULL; + } + + ret = PyTuple_New(4); + if (ret == NULL) + return NULL; + + for (i = 0; i < 4; i++) { + tmp = PyLong_FromSsize_t(s[i]); + if (tmp == NULL) + goto error; + PyTuple_SET_ITEM(ret, i, tmp); + } + + return ret; + +error: + Py_DECREF(ret); + return NULL; +} + + +static PyMappingMethods ndarray_as_mapping = { + NULL, /* mp_length */ + (binaryfunc)ndarray_subscript, /* mp_subscript */ + (objobjargproc)ndarray_ass_subscript /* mp_ass_subscript */ +}; + +static PySequenceMethods ndarray_as_sequence = { + 0, /* sq_length */ + 0, /* sq_concat */ + 0, /* sq_repeat */ + (ssizeargfunc)ndarray_item, /* sq_item */ +}; + + +/**************************************************************************/ +/* getters */ +/**************************************************************************/ + +static PyObject * +ssize_array_as_tuple(Py_ssize_t *array, Py_ssize_t len) +{ + PyObject *tuple, *x; + Py_ssize_t i; + + if (array == NULL) + return PyTuple_New(0); + + tuple = PyTuple_New(len); + if (tuple == NULL) + return NULL; + + for (i = 0; i < len; i++) { + x = PyLong_FromSsize_t(array[i]); + if (x == NULL) { + Py_DECREF(tuple); + return NULL; + } + PyTuple_SET_ITEM(tuple, i, x); + } + + return tuple; +} + +static PyObject * +ndarray_getflags(NDArrayObject *self, void *closure) +{ + return PyLong_FromLong(self->head->flags); +} + +static PyObject * +ndarray_getoffset(NDArrayObject *self, void *closure) +{ + ndbuf_t *ndbuf = self->head; + return PyLong_FromSsize_t(ndbuf->offset); +} + +static PyObject * +ndarray_getobj(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + + if (base->obj == NULL) { + Py_RETURN_NONE; + } + Py_INCREF(base->obj); + return base->obj; +} + +static PyObject * +ndarray_getlen(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromSsize_t(base->len); +} + +static PyObject * +ndarray_getreadonly(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromLong(base->readonly); +} + +static PyObject * +ndarray_getitemsize(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromSsize_t(base->itemsize); +} + +static PyObject * +ndarray_getformat(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + char *fmt = base->format ? base->format : ""; + return PyUnicode_FromString(fmt); +} + +static PyObject * +ndarray_getndim(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return PyLong_FromSsize_t(base->ndim); +} + +static PyObject * +ndarray_getshape(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return ssize_array_as_tuple(base->shape, base->ndim); +} + +static PyObject * +ndarray_getstrides(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return ssize_array_as_tuple(base->strides, base->ndim); +} + +static PyObject * +ndarray_getsuboffsets(NDArrayObject *self, void *closure) +{ + Py_buffer *base = &self->head->base; + return ssize_array_as_tuple(base->suboffsets, base->ndim); +} + +static PyObject * +ndarray_c_contig(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + int ret = PyBuffer_IsContiguous(&nd->head->base, 'C'); + + if (ret != ND_C_CONTIGUOUS(nd->head->flags)) { + PyErr_SetString(PyExc_RuntimeError, + "results from PyBuffer_IsContiguous() and flags differ"); + return NULL; + } + return PyBool_FromLong(ret); +} + +static PyObject * +ndarray_fortran_contig(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + int ret = PyBuffer_IsContiguous(&nd->head->base, 'F'); + + if (ret != ND_FORTRAN_CONTIGUOUS(nd->head->flags)) { + PyErr_SetString(PyExc_RuntimeError, + "results from PyBuffer_IsContiguous() and flags differ"); + return NULL; + } + return PyBool_FromLong(ret); +} + +static PyObject * +ndarray_contig(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + int ret = PyBuffer_IsContiguous(&nd->head->base, 'A'); + + if (ret != ND_ANY_CONTIGUOUS(nd->head->flags)) { + PyErr_SetString(PyExc_RuntimeError, + "results from PyBuffer_IsContiguous() and flags differ"); + return NULL; + } + return PyBool_FromLong(ret); +} + + +static PyGetSetDef ndarray_getset [] = +{ + /* ndbuf */ + { "flags", (getter)ndarray_getflags, NULL, NULL, NULL}, + { "offset", (getter)ndarray_getoffset, NULL, NULL, NULL}, + /* ndbuf.base */ + { "obj", (getter)ndarray_getobj, NULL, NULL, NULL}, + { "len", (getter)ndarray_getlen, NULL, NULL, NULL}, + { "readonly", (getter)ndarray_getreadonly, NULL, NULL, NULL}, + { "itemsize", (getter)ndarray_getitemsize, NULL, NULL, NULL}, + { "format", (getter)ndarray_getformat, NULL, NULL, NULL}, + { "ndim", (getter)ndarray_getndim, NULL, NULL, NULL}, + { "shape", (getter)ndarray_getshape, NULL, NULL, NULL}, + { "strides", (getter)ndarray_getstrides, NULL, NULL, NULL}, + { "suboffsets", (getter)ndarray_getsuboffsets, NULL, NULL, NULL}, + { "c_contiguous", (getter)ndarray_c_contig, NULL, NULL, NULL }, + { "f_contiguous", (getter)ndarray_fortran_contig, NULL, NULL, NULL }, + { "contiguous", (getter)ndarray_contig, NULL, NULL, NULL }, + {NULL} +}; + +static PyObject * +ndarray_tolist(PyObject *self, PyObject *dummy) +{ + return ndarray_as_list((NDArrayObject *)self); +} + +static PyObject * +ndarray_tobytes(PyObject *self, PyObject *dummy) +{ + ndbuf_t *ndbuf = ((NDArrayObject *)self)->head; + Py_buffer *src = &ndbuf->base; + Py_buffer dest; + PyObject *ret = NULL; + char *mem; + + if (ND_C_CONTIGUOUS(ndbuf->flags)) + return PyBytes_FromStringAndSize(src->buf, src->len); + + assert(src->shape != NULL); + assert(src->strides != NULL); + assert(src->ndim > 0); + + mem = PyMem_Malloc(src->len); + if (mem == NULL) { + PyErr_NoMemory(); + return NULL; + } + + dest = *src; + dest.buf = mem; + dest.suboffsets = NULL; + dest.strides = strides_from_shape(ndbuf, 0); + if (dest.strides == NULL) + goto out; + if (copy_buffer(&dest, src) < 0) + goto out; + + ret = PyBytes_FromStringAndSize(mem, src->len); + +out: + PyMem_XFree(dest.strides); + PyMem_Free(mem); + return ret; +} + +/* add redundant (negative) suboffsets for testing */ +static PyObject * +ndarray_add_suboffsets(PyObject *self, PyObject *dummy) +{ + NDArrayObject *nd = (NDArrayObject *)self; + Py_buffer *base = &nd->head->base; + Py_ssize_t i; + + if (base->suboffsets != NULL) { + PyErr_SetString(PyExc_TypeError, + "cannot add suboffsets to PIL-style array"); + return NULL; + } + if (base->strides == NULL) { + PyErr_SetString(PyExc_TypeError, + "cannot add suboffsets to array without strides"); + return NULL; + } + + base->suboffsets = PyMem_Malloc(base->ndim * (sizeof *base->suboffsets)); + if (base->suboffsets == NULL) { + PyErr_NoMemory(); + return NULL; + } + + for (i = 0; i < base->ndim; i++) + base->suboffsets[i] = -1; + + Py_RETURN_NONE; +} + +/* Test PyMemoryView_FromBuffer(): return a memoryview from a static buffer. + Obviously this is fragile and only one such view may be active at any + time. Never use anything like this in real code! */ +static char *infobuf = NULL; +static PyObject * +ndarray_memoryview_from_buffer(PyObject *self, PyObject *dummy) +{ + const NDArrayObject *nd = (NDArrayObject *)self; + const Py_buffer *view = &nd->head->base; + const ndbuf_t *ndbuf; + static char format[64+1]; + static Py_ssize_t shape[64]; + static Py_ssize_t strides[64]; + static Py_ssize_t suboffsets[64]; + static Py_buffer info; + char *p; + + if (!ND_IS_CONSUMER(nd)) + ndbuf = nd->head; /* self is ndarray/original exporter */ + else if (NDArray_Check(view->obj) && !ND_IS_CONSUMER(view->obj)) + /* self is ndarray and consumer from ndarray/original exporter */ + ndbuf = ((NDArrayObject *)view->obj)->head; + else { + PyErr_SetString(PyExc_TypeError, + "memoryview_from_buffer(): ndarray must be original exporter or " + "consumer from ndarray/original exporter"); + return NULL; + } + + info = *view; + p = PyMem_Realloc(infobuf, ndbuf->len); + if (p == NULL) { + PyMem_Free(infobuf); + PyErr_NoMemory(); + infobuf = NULL; + return NULL; + } + else { + infobuf = p; + } + /* copy the complete raw data */ + memcpy(infobuf, ndbuf->data, ndbuf->len); + info.buf = infobuf + ((char *)view->buf - ndbuf->data); + + if (view->format) { + if (strlen(view->format) > 64) { + PyErr_SetString(PyExc_TypeError, + "memoryview_from_buffer: format is limited to 64 characters"); + return NULL; + } + strcpy(format, view->format); + info.format = format; + } + if (view->ndim > 64) { + PyErr_SetString(PyExc_TypeError, + "memoryview_from_buffer: ndim is limited to 64"); + return NULL; + } + if (view->shape) { + memcpy(shape, view->shape, view->ndim * sizeof(Py_ssize_t)); + info.shape = shape; + } + if (view->strides) { + memcpy(strides, view->strides, view->ndim * sizeof(Py_ssize_t)); + info.strides = strides; + } + if (view->suboffsets) { + memcpy(suboffsets, view->suboffsets, view->ndim * sizeof(Py_ssize_t)); + info.suboffsets = suboffsets; + } + + return PyMemoryView_FromBuffer(&info); +} + +/* Get a single item from bufobj at the location specified by seq. + seq is a list or tuple of indices. The purpose of this function + is to check other functions against PyBuffer_GetPointer(). */ +static PyObject * +get_pointer(PyObject *self, PyObject *args) +{ + PyObject *ret = NULL, *bufobj, *seq; + Py_buffer view; + Py_ssize_t indices[64]; + Py_ssize_t i; + void *ptr; + + if (!PyArg_ParseTuple(args, "OO", &bufobj, &seq)) { + return NULL; + } + + CHECK_LIST_OR_TUPLE(seq); + if (PyObject_GetBuffer(bufobj, &view, PyBUF_FULL_RO) < 0) + return NULL; + + if (view.ndim > 64) { + PyErr_SetString(PyExc_ValueError, + "get_pointer(): ndim > 64"); + goto out; + } + if (PySequence_Fast_GET_SIZE(seq) != view.ndim) { + PyErr_SetString(PyExc_ValueError, + "get_pointer(): len(indices) != ndim"); + goto out; + } + + for (i = 0; i < view.ndim; i++) { + PyObject *x = PySequence_Fast_GET_ITEM(seq, i); + indices[i] = PyLong_AsSsize_t(x); + if (PyErr_Occurred()) + goto out; + if (indices[i] < 0 || indices[i] >= view.shape[i]) { + PyErr_Format(PyExc_ValueError, + "get_pointer(): invalid index %zd at position %zd", + indices[i], i); + goto out; + } + } + + ptr = PyBuffer_GetPointer(&view, indices); + ret = unpack_single(ptr, view.format, view.itemsize); + +out: + PyBuffer_Release(&view); + return ret; +} + + +static PyMethodDef ndarray_methods [] = +{ + { "tolist", ndarray_tolist, METH_NOARGS, NULL }, + { "tobytes", ndarray_tobytes, METH_NOARGS, NULL }, + { "push", (PyCFunction)ndarray_push, METH_VARARGS|METH_KEYWORDS, NULL }, + { "pop", ndarray_pop, METH_NOARGS, NULL }, + { "add_suboffsets", ndarray_add_suboffsets, METH_NOARGS, NULL }, + { "memoryview_from_buffer", ndarray_memoryview_from_buffer, METH_NOARGS, NULL }, + {NULL} +}; + +static PyTypeObject NDArray_Type = { + PyVarObject_HEAD_INIT(NULL, 0) + "ndarray", /* Name of this type */ + sizeof(NDArrayObject), /* Basic object size */ + 0, /* Item size for varobject */ + (destructor)ndarray_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + &ndarray_as_sequence, /* tp_as_sequence */ + &ndarray_as_mapping, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + &ndarray_as_buffer, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT, /* tp_flags */ + 0, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + ndarray_methods, /* tp_methods */ + 0, /* tp_members */ + ndarray_getset, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + ndarray_init, /* tp_init */ + 0, /* tp_alloc */ + ndarray_new, /* tp_new */ +}; + + +static struct PyMethodDef _testbuffer_functions[] = { + {"slice_indices", slice_indices, METH_VARARGS, NULL}, + {"get_pointer", get_pointer, METH_VARARGS, NULL}, + {NULL, NULL} +}; + +static struct PyModuleDef _testbuffermodule = { + PyModuleDef_HEAD_INIT, + "_testbuffer", + NULL, + -1, + _testbuffer_functions, + NULL, + NULL, + NULL, + NULL +}; + + +PyMODINIT_FUNC +PyInit__testbuffer(void) +{ + PyObject *m; + + m = PyModule_Create(&_testbuffermodule); + if (m == NULL) + return NULL; + + Py_TYPE(&NDArray_Type)=&PyType_Type; + Py_INCREF(&NDArray_Type); + PyModule_AddObject(m, "ndarray", (PyObject *)&NDArray_Type); + + structmodule = PyImport_ImportModule("struct"); + if (structmodule == NULL) + return NULL; + + Struct = PyObject_GetAttrString(structmodule, "Struct"); + calcsize = PyObject_GetAttrString(structmodule, "calcsize"); + if (Struct == NULL || calcsize == NULL) + return NULL; + + simple_format = PyUnicode_FromString(simple_fmt); + if (simple_format == NULL) + return NULL; + + PyModule_AddIntConstant(m, "ND_VAREXPORT", ND_VAREXPORT); + PyModule_AddIntConstant(m, "ND_WRITABLE", ND_WRITABLE); + PyModule_AddIntConstant(m, "ND_FORTRAN", ND_FORTRAN); + PyModule_AddIntConstant(m, "ND_SCALAR", ND_SCALAR); + PyModule_AddIntConstant(m, "ND_PIL", ND_PIL); + PyModule_AddIntConstant(m, "ND_GETBUF_FAIL", ND_GETBUF_FAIL); + + PyModule_AddIntConstant(m, "PyBUF_SIMPLE", PyBUF_SIMPLE); + PyModule_AddIntConstant(m, "PyBUF_WRITABLE", PyBUF_WRITABLE); + PyModule_AddIntConstant(m, "PyBUF_FORMAT", PyBUF_FORMAT); + PyModule_AddIntConstant(m, "PyBUF_ND", PyBUF_ND); + PyModule_AddIntConstant(m, "PyBUF_STRIDES", PyBUF_STRIDES); + PyModule_AddIntConstant(m, "PyBUF_INDIRECT", PyBUF_INDIRECT); + PyModule_AddIntConstant(m, "PyBUF_C_CONTIGUOUS", PyBUF_C_CONTIGUOUS); + PyModule_AddIntConstant(m, "PyBUF_F_CONTIGUOUS", PyBUF_F_CONTIGUOUS); + PyModule_AddIntConstant(m, "PyBUF_ANY_CONTIGUOUS", PyBUF_ANY_CONTIGUOUS); + PyModule_AddIntConstant(m, "PyBUF_FULL", PyBUF_FULL); + PyModule_AddIntConstant(m, "PyBUF_FULL_RO", PyBUF_FULL_RO); + PyModule_AddIntConstant(m, "PyBUF_RECORDS", PyBUF_RECORDS); + PyModule_AddIntConstant(m, "PyBUF_RECORDS_RO", PyBUF_RECORDS_RO); + PyModule_AddIntConstant(m, "PyBUF_STRIDED", PyBUF_STRIDED); + PyModule_AddIntConstant(m, "PyBUF_STRIDED_RO", PyBUF_STRIDED_RO); + PyModule_AddIntConstant(m, "PyBUF_CONTIG", PyBUF_CONTIG); + PyModule_AddIntConstant(m, "PyBUF_CONTIG_RO", PyBUF_CONTIG_RO); + + return m; +} + + + diff -r 92842e347d98 -r 51cedae5acfc Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c Thu Sep 08 00:56:17 2011 +0200 +++ b/Modules/_testcapimodule.c Thu Sep 08 15:02:41 2011 +0200 @@ -275,95 +275,6 @@ } -/* Issue #7385: Check that memoryview() does not crash - * when bf_getbuffer returns an error - */ - -static int -broken_buffer_getbuffer(PyObject *self, Py_buffer *view, int flags) -{ - PyErr_SetString( - TestError, - "test_broken_memoryview: expected error in bf_getbuffer"); - return -1; -} - -static PyBufferProcs memoryviewtester_as_buffer = { - (getbufferproc)broken_buffer_getbuffer, /* bf_getbuffer */ - 0, /* bf_releasebuffer */ -}; - -static PyTypeObject _MemoryViewTester_Type = { - PyVarObject_HEAD_INIT(NULL, 0) - "memoryviewtester", /* Name of this type */ - sizeof(PyObject), /* Basic object size */ - 0, /* Item size for varobject */ - (destructor)PyObject_Del, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_compare */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - PyObject_GenericGetAttr, /* tp_getattro */ - 0, /* tp_setattro */ - &memoryviewtester_as_buffer, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - 0, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - 0, /* tp_members */ - 0, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - 0, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - PyType_GenericNew, /* tp_new */ -}; - -static PyObject* -test_broken_memoryview(PyObject* self) -{ - PyObject *obj = PyObject_New(PyObject, &_MemoryViewTester_Type); - PyObject *res; - - if (obj == NULL) { - PyErr_Clear(); - PyErr_SetString( - TestError, - "test_broken_memoryview: failed to create object"); - return NULL; - } - - res = PyMemoryView_FromObject(obj); - if (res || !PyErr_Occurred()){ - PyErr_SetString( - TestError, - "test_broken_memoryview: memoryview() didn't raise an Exception"); - Py_XDECREF(res); - Py_DECREF(obj); - return NULL; - } - - PyErr_Clear(); - Py_DECREF(obj); - Py_RETURN_NONE; -} - - /* Tests of PyLong_{As, From}{Unsigned,}Long(), and (#ifdef HAVE_LONG_LONG) PyLong_{As, From}{Unsigned,}LongLong(). @@ -2325,7 +2236,6 @@ {"test_list_api", (PyCFunction)test_list_api, METH_NOARGS}, {"test_dict_iteration", (PyCFunction)test_dict_iteration,METH_NOARGS}, {"test_lazy_hash_inheritance", (PyCFunction)test_lazy_hash_inheritance,METH_NOARGS}, - {"test_broken_memoryview", (PyCFunction)test_broken_memoryview,METH_NOARGS}, {"test_long_api", (PyCFunction)test_long_api, METH_NOARGS}, {"test_long_and_overflow", (PyCFunction)test_long_and_overflow, METH_NOARGS}, @@ -2585,7 +2495,6 @@ return NULL; Py_TYPE(&_HashInheritanceTester_Type)=&PyType_Type; - Py_TYPE(&_MemoryViewTester_Type)=&PyType_Type; Py_TYPE(&test_structmembersType)=&PyType_Type; Py_INCREF(&test_structmembersType); diff -r 92842e347d98 -r 51cedae5acfc Objects/abstract.c --- a/Objects/abstract.c Thu Sep 08 00:56:17 2011 +0200 +++ b/Objects/abstract.c Thu Sep 08 15:02:41 2011 +0200 @@ -340,7 +340,7 @@ } static int -_IsFortranContiguous(Py_buffer *view) +_IsFortranContiguous(const Py_buffer *view) { Py_ssize_t sd, dim; int i; @@ -361,7 +361,7 @@ } static int -_IsCContiguous(Py_buffer *view) +_IsCContiguous(const Py_buffer *view) { Py_ssize_t sd, dim; int i; @@ -382,7 +382,7 @@ } int -PyBuffer_IsContiguous(Py_buffer *view, char fort) +PyBuffer_IsContiguous(const Py_buffer *view, char fort) { if (view->suboffsets != NULL) return 0; @@ -651,7 +651,7 @@ PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int flags) { - if (view == NULL) return 0; + if (view == NULL) return 0; /* XXX why not -1? */ if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) && (readonly == 1)) { PyErr_SetString(PyExc_BufferError, diff -r 92842e347d98 -r 51cedae5acfc Objects/memoryobject.c --- a/Objects/memoryobject.c Thu Sep 08 00:56:17 2011 +0200 +++ b/Objects/memoryobject.c Thu Sep 08 15:02:41 2011 +0200 @@ -1,126 +1,519 @@ - /* Memoryview object implementation */ #include "Python.h" +#include -#define IS_RELEASED(memobj) \ - (((PyMemoryViewObject *) memobj)->view.buf == NULL) + +/****************************************************************************/ +/* ManagedBuffer Object */ +/****************************************************************************/ + +/* + ManagedBuffer Object: + --------------------- + + The purpose of this object is to facilitate the handling of chained + memoryviews that have the same underlying exporting object. PEP-3118 + allows the underlying object to change while a view is exported. This + could lead to unexpected results when constructing a new memoryview + from an existing memoryview. + + Rather than repeatedly redirecting buffer requests to the original base + object, all chained memoryviews use a single buffer snapshot. This + snapshot is generated by the constructor PyManagedBuffer_FromObject(). + + Ownership rules: + ---------------- + + The master buffer inside a managed buffer is filled in by the original + base object. shape, strides, suboffsets and format are read-only for + all consumers. + + A memoryview's buffer is a private copy of the exporter's buffer. shape, + strides and suboffsets belong to the memoryview and are thus writable. + + If a memoryview itself exports several buffers via memory_getbuf(), all + buffer copies share shape, strides and suboffsets. In this case, the + arrays are NOT writable. + + Reference count assumptions: + ---------------------------- + + The 'obj' member of a Py_buffer must either be NULL or refer to the + exporting base object. In the Python codebase, all getbufferprocs + return a new reference to view.obj (example: bytes_buffer_getbuffer()). + + PyBuffer_Release() decrements view.obj (if non-NULL), so the + releasebufferprocs must NOT decrement view.obj. +*/ + + +#define CHECK_MBUF_RELEASED(mbuf) \ + if (((PyManagedBufferObject *)mbuf)->released) { \ + PyErr_SetString(PyExc_ValueError, \ + "operation forbidden on released managedbuffer object"); \ + return NULL; \ + } + + +Py_LOCAL_INLINE(PyManagedBufferObject *) +mbuf_alloc(void) +{ + PyManagedBufferObject *mbuf; + + mbuf = (PyManagedBufferObject *) + PyObject_GC_New(PyManagedBufferObject, &PyManagedBuffer_Type); + if (mbuf == NULL) + return NULL; + mbuf->released = 0; + mbuf->exports = 0; + mbuf->master.obj = NULL; + _PyObject_GC_TRACK(mbuf); + + return mbuf; +} + +static PyObject * +PyManagedBuffer_FromObject(PyObject *base) +{ + PyManagedBufferObject *mbuf; + + mbuf = mbuf_alloc(); + if (mbuf == NULL) + return NULL; + + if (PyObject_GetBuffer(base, &mbuf->master, PyBUF_FULL_RO) < 0) { + /* mbuf->master.obj must be NULL. */ + Py_DECREF(mbuf); + return NULL; + } + + /* Assume that master.obj is a new reference to base. */ + assert(mbuf->master.obj == base); + + return (PyObject *)mbuf; +} + +static void +mbuf_release(PyManagedBufferObject *self) +{ + if (self->released) + return; + + /* NOTE: at this point self->exports still can be > 0 if this function + is called from mbuf_clear() to break up a reference cycle. */ + self->released = 1; + + /* PyBuffer_Release() decrements master->obj and sets it to NULL. */ + _PyObject_GC_UNTRACK(self); + PyBuffer_Release(&self->master); +} + +static void +mbuf_dealloc(PyManagedBufferObject *self) +{ + assert(self->exports == 0); + mbuf_release(self); + PyObject_GC_Del(self); +} + +static int +mbuf_traverse(PyManagedBufferObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->master.obj); + return 0; +} + +static int +mbuf_clear(PyManagedBufferObject *self) +{ + assert(self->exports >= 0); + mbuf_release(self); + return 0; +} + +PyTypeObject PyManagedBuffer_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "managedbuffer", + sizeof(PyManagedBufferObject), + 0, + (destructor)mbuf_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ + 0, /* tp_doc */ + (traverseproc)mbuf_traverse, /* tp_traverse */ + (inquiry)mbuf_clear /* tp_clear */ +}; + + +/****************************************************************************/ +/* MemoryView Object */ +/****************************************************************************/ + +/* In the process of breaking reference cycles mbuf_release() can be + called before memory_release(). */ +#define BASE_INACCESSIBLE(memobj) \ + ((((PyMemoryViewObject *) memobj)->flags & Py_MEMORYVIEW_RELEASED) || \ + ((PyMemoryViewObject *) memobj)->mbuf->released) #define CHECK_RELEASED(memobj) \ - if (IS_RELEASED(memobj)) { \ - PyErr_SetString(PyExc_ValueError, \ - "operation forbidden on released memoryview object"); \ - return NULL; \ + if (BASE_INACCESSIBLE(memobj)) { \ + PyErr_SetString(PyExc_ValueError, \ + "operation forbidden on released memoryview object"); \ + return NULL; \ } #define CHECK_RELEASED_INT(memobj) \ - if (IS_RELEASED(memobj)) { \ - PyErr_SetString(PyExc_ValueError, \ - "operation forbidden on released memoryview object"); \ - return -1; \ + if (BASE_INACCESSIBLE(memobj)) { \ + PyErr_SetString(PyExc_ValueError, \ + "operation forbidden on released memoryview object"); \ + return -1; \ } -static Py_ssize_t -get_shape0(Py_buffer *buf) -{ - if (buf->shape != NULL) - return buf->shape[0]; - if (buf->ndim == 0) - return 1; - PyErr_SetString(PyExc_TypeError, - "exported buffer does not have any shape information associated " - "to it"); - return -1; -} +#define CHECK_LIST_OR_TUPLE(v) \ + if (!PyList_Check(v) && !PyTuple_Check(v)) { \ + PyErr_SetString(PyExc_TypeError, \ + #v " must be a list or a tuple"); \ + return NULL; \ + } -static void -dup_buffer(Py_buffer *dest, Py_buffer *src) -{ - *dest = *src; - if (src->ndim == 1 && src->shape != NULL) { - dest->shape = &(dest->smalltable[0]); - dest->shape[0] = get_shape0(src); - } - if (src->ndim == 1 && src->strides != NULL) { - dest->strides = &(dest->smalltable[1]); - dest->strides[0] = src->strides[0]; - } -} +#define VIEW_ADDR(mv) (&((PyMemoryViewObject *)mv)->view) -static int -memory_getbuf(PyMemoryViewObject *self, Py_buffer *view, int flags) -{ - int res = 0; - CHECK_RELEASED_INT(self); - if (self->view.obj != NULL) - res = PyObject_GetBuffer(self->view.obj, view, flags); - if (view) - dup_buffer(view, &self->view); - return res; -} +/* Check for the presence of suboffsets in the first dimension. */ +#define HAVE_PTR(suboffsets) (suboffsets && suboffsets[0] >= 0) +/* Adjust ptr if suboffsets are present. */ +#define ADJUST_PTR(ptr, suboffsets) \ + (HAVE_PTR(suboffsets) ? *((char**)ptr) + suboffsets[0] : ptr) -static void -memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view) -{ - PyBuffer_Release(view); -} +/* Memoryview buffer properties */ +#define MV_C_CONTIGUOUS(flags) (flags&(Py_MEMORYVIEW_SCALAR|Py_MEMORYVIEW_C)) +#define MV_F_CONTIGUOUS(flags) \ + (flags&(Py_MEMORYVIEW_SCALAR|Py_MEMORYVIEW_FORTRAN)) +#define MV_ANY_CONTIGUOUS(flags) \ + (flags&(Py_MEMORYVIEW_SCALAR|Py_MEMORYVIEW_C|Py_MEMORYVIEW_FORTRAN)) + +/* Fast contiguity test. Caller must ensure suboffsets==NULL and ndim==1. */ +#define MV_CONTIGUOUS_NDIM1(view) \ + ((view)->shape[0] == 1 || (view)->strides[0] == (view)->itemsize) + +/* getbuffer() requests */ +#define REQ_INDIRECT(flags) ((flags&PyBUF_INDIRECT) == PyBUF_INDIRECT) +#define REQ_C_CONTIGUOUS(flags) ((flags&PyBUF_C_CONTIGUOUS) == PyBUF_C_CONTIGUOUS) +#define REQ_F_CONTIGUOUS(flags) ((flags&PyBUF_F_CONTIGUOUS) == PyBUF_F_CONTIGUOUS) +#define REQ_ANY_CONTIGUOUS(flags) ((flags&PyBUF_ANY_CONTIGUOUS) == PyBUF_ANY_CONTIGUOUS) +#define REQ_STRIDES(flags) ((flags&PyBUF_STRIDES) == PyBUF_STRIDES) +#define REQ_SHAPE(flags) ((flags&PyBUF_ND) == PyBUF_ND) +#define REQ_WRITABLE(flags) (flags&PyBUF_WRITABLE) +#define REQ_FORMAT(flags) (flags&PyBUF_FORMAT) + PyDoc_STRVAR(memory_doc, "memoryview(object)\n\ \n\ Create a new memoryview object which references the given object."); + +/****************************************************************************/ +/* Constructors */ +/****************************************************************************/ + +/* Initialize values that are shared with the managed buffer. */ +Py_LOCAL_INLINE(void) +init_shared_values(Py_buffer *dest, const Py_buffer *src) +{ + dest->obj = src->obj; + dest->buf = src->buf; + dest->len = src->len; + dest->itemsize = src->itemsize; + dest->readonly = src->readonly; + dest->format = src->format ? src->format : "B"; + dest->internal = src->internal; +} + +/* Initialize strides for a C-contiguous array. */ +Py_LOCAL_INLINE(void) +init_strides_from_shape(Py_buffer *view) +{ + Py_ssize_t i; + + assert(view->ndim > 0); + + view->strides[view->ndim-1] = view->itemsize; + for (i = view->ndim-2; i >= 0; i--) + view->strides[i] = view->strides[i+1] * view->shape[i+1]; +} + +/* Copy shape and strides. Reconstruct missing values. */ +static void +init_shape_strides(Py_buffer *dest, const Py_buffer *src) +{ + Py_ssize_t i; + + if (src->ndim == 0) { + dest->shape = NULL; + dest->strides = NULL; + return; + } + if (src->ndim == 1) { + dest->shape[0] = src->shape ? src->shape[0] : src->len / src->itemsize; + dest->strides[0] = src->strides ? src->strides[0] : src->itemsize; + return; + } + + for (i = 0; i < src->ndim; i++) + dest->shape[i] = src->shape[i]; + if (src->strides) { + for (i = 0; i < src->ndim; i++) + dest->strides[i] = src->strides[i]; + } + else { + init_strides_from_shape(dest); + } +} + +Py_LOCAL_INLINE(void) +init_suboffsets(Py_buffer *dest, const Py_buffer *src) +{ + Py_ssize_t i; + + if (src->suboffsets == NULL) { + dest->suboffsets = NULL; + return; + } + for (i = 0; i < src->ndim; i++) + dest->suboffsets[i] = src->suboffsets[i]; +} + +/* len = product(shape) * itemsize */ +Py_LOCAL_INLINE(void) +init_len(Py_buffer *view) +{ + Py_ssize_t i, len; + + len = 1; + for (i = 0; i < view->ndim; i++) + len *= view->shape[i]; + len *= view->itemsize; + + view->len = len; +} + +/* Initialize memoryview buffer properties. */ +static void +init_flags(PyMemoryViewObject *mv) +{ + const Py_buffer *view = &mv->view; + int flags = 0; + + switch (view->ndim) { + case 0: + flags |= (Py_MEMORYVIEW_SCALAR|Py_MEMORYVIEW_C|Py_MEMORYVIEW_FORTRAN); + break; + case 1: + if (MV_CONTIGUOUS_NDIM1(view)) + flags |= (Py_MEMORYVIEW_C|Py_MEMORYVIEW_FORTRAN); + break; + default: + if (PyBuffer_IsContiguous(view, 'C')) + flags |= Py_MEMORYVIEW_C; + if (PyBuffer_IsContiguous(view, 'F')) + flags |= Py_MEMORYVIEW_FORTRAN; + break; + } + + if (view->suboffsets) { + flags |= Py_MEMORYVIEW_PIL; + flags &= ~(Py_MEMORYVIEW_C|Py_MEMORYVIEW_FORTRAN); + } + + mv->flags = flags; +} + +Py_LOCAL_INLINE(PyMemoryViewObject *) +mv_alloc(int ndim) +{ + PyMemoryViewObject *mv; + + mv = (PyMemoryViewObject *) + PyObject_GC_NewVar(PyMemoryViewObject, &PyMemoryView_Type, 3*ndim); + if (mv == NULL) + return NULL; + + mv->mbuf = NULL; + mv->flags = 0; + mv->exports = 0; + mv->view.ndim = ndim; + mv->view.shape = mv->ob_array; + mv->view.strides = mv->ob_array + ndim; + mv->view.suboffsets = mv->ob_array + 2 * ndim; + + _PyObject_GC_TRACK(mv); + return mv; +} + +/* + Return a new memoryview that is registered with mbuf. If src is NULL, + use mbuf->master as the underlying buffer. Otherwise, use src. + + The new memoryview has full buffer information: shape and strides + are always present, suboffsets as needed. Arrays are copied to + the memoryview's ob_array field. + */ +static PyObject * +mbuf_add_view(PyManagedBufferObject *mbuf, const Py_buffer *src) +{ + PyMemoryViewObject *mv; + Py_buffer *dest; + + if (src == NULL) + src = &mbuf->master; + + mv = mv_alloc(src->ndim); + if (mv == NULL) + return NULL; + + dest = &mv->view; + init_shared_values(dest, src); + init_shape_strides(dest, src); + init_suboffsets(dest, src); + init_flags(mv); + + mv->mbuf = mbuf; + Py_INCREF(mbuf); + mbuf->exports++; + + return (PyObject *)mv; +} + +/* Register an incomplete view without copying shape, strides and suboffsets. + Use 'ndim' instead of src->ndim to determine the size of the memoryview's + ob_array. */ +static PyObject * +mbuf_add_incomplete_view(PyManagedBufferObject *mbuf, const Py_buffer *src, + int ndim) +{ + PyMemoryViewObject *mv; + Py_buffer *dest; + + mv = mv_alloc(ndim); + if (mv == NULL) + return NULL; + + dest = &mv->view; + init_shared_values(dest, src); + + mv->mbuf = mbuf; + Py_INCREF(mbuf); + mbuf->exports++; + + return (PyObject *)mv; +} + +/* Expose a raw memory area as a view of contiguous bytes. flags can be + PyBUF_READ or PyBUF_WRITE. view->format is set to "B" (unsigned bytes). + The memoryview has complete buffer information. */ +PyObject * +PyMemoryView_FromMemory(char *mem, Py_ssize_t size, int flags) +{ + PyManagedBufferObject *mbuf; + PyObject *mv; + int readonly; + + assert(mem != NULL); + assert(flags == PyBUF_READ || flags == PyBUF_WRITE); + + mbuf = mbuf_alloc(); + if (mbuf == NULL) + return NULL; + + readonly = (flags == PyBUF_WRITE) ? 0 : 1; + (void)PyBuffer_FillInfo(&mbuf->master, NULL, mem, size, readonly, + PyBUF_FULL_RO); + + mv = mbuf_add_view(mbuf, NULL); + Py_DECREF(mbuf); + + return mv; +} + +/* Create a memoryview from a given Py_buffer. For simple byte views, + PyMemoryView_FromMemory() should be used instead. + This function is the only entry point that can create a master buffer + without full information. Because of this fact init_shape_strides() + must be able to reconstruct missing values. */ PyObject * PyMemoryView_FromBuffer(Py_buffer *info) { - PyMemoryViewObject *mview; + PyManagedBufferObject *mbuf; + PyObject *mv; if (info->buf == NULL) { PyErr_SetString(PyExc_ValueError, - "cannot make memory view from a buffer with a NULL data pointer"); - return NULL; - } - mview = (PyMemoryViewObject *) - PyObject_GC_New(PyMemoryViewObject, &PyMemoryView_Type); - if (mview == NULL) - return NULL; - dup_buffer(&mview->view, info); - /* NOTE: mview->view.obj should already have been incref'ed as - part of PyBuffer_FillInfo(). */ - _PyObject_GC_TRACK(mview); - return (PyObject *)mview; -} - -PyObject * -PyMemoryView_FromObject(PyObject *base) -{ - PyMemoryViewObject *mview; - Py_buffer view; - - if (!PyObject_CheckBuffer(base)) { - PyErr_SetString(PyExc_TypeError, - "cannot make memory view because object does " - "not have the buffer interface"); + "PyMemoryView_FromBuffer(): info->buf must not be NULL"); return NULL; } - if (PyObject_GetBuffer(base, &view, PyBUF_FULL_RO) < 0) + mbuf = mbuf_alloc(); + if (mbuf == NULL) return NULL; - mview = (PyMemoryViewObject *)PyMemoryView_FromBuffer(&view); - if (mview == NULL) { - PyBuffer_Release(&view); - return NULL; + /* info->obj is either NULL or a borrowed reference. This reference + should not be decremented in PyBuffer_Release(). */ + mbuf->master = *info; + mbuf->master.obj = NULL; + + mv = mbuf_add_view(mbuf, NULL); + Py_DECREF(mbuf); + + return mv; +} + +/* Create a memoryview from an object that implements the buffer protocol. + If the object is a memoryview, the new memoryview must be registered + with the same managed buffer. Otherwise, a new managed buffer is created. */ +PyObject * +PyMemoryView_FromObject(PyObject *v) +{ + PyManagedBufferObject *mbuf; + + if (PyMemoryView_Check(v)) { + PyMemoryViewObject *mv = (PyMemoryViewObject *)v; + CHECK_RELEASED(mv); + return mbuf_add_view(mv->mbuf, &mv->view); + } + else if (PyObject_CheckBuffer(v)) { + PyObject *ret; + mbuf = (PyManagedBufferObject *)PyManagedBuffer_FromObject(v); + if (mbuf == NULL) + return NULL; + ret = mbuf_add_view(mbuf, NULL); + Py_DECREF(mbuf); + return ret; } - return (PyObject *)mview; + PyErr_SetString(PyExc_TypeError, + "memoryview: object does not have the buffer interface"); + return NULL; } static PyObject * memory_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds) { PyObject *obj; - static char *kwlist[] = {"object", 0}; + static char *kwlist[] = {"object", NULL}; if (!PyArg_ParseTupleAndKeywords(args, kwds, "O:memoryview", kwlist, &obj)) { @@ -131,6 +524,1252 @@ } +/****************************************************************************/ +/* Release/GC management */ +/****************************************************************************/ + +/* Inform the managed buffer that this particular memoryview will not access + the underlying buffer again. If no other memoryviews are registered with + the managed buffer, the underlying buffer is released instantly and + marked as inaccessible for both the memoryview and the managed buffer. + + This function fails if the memoryview itself has exported buffers. */ +static int +_memory_release(PyMemoryViewObject *self) +{ + if (self->flags & Py_MEMORYVIEW_RELEASED) + return 0; + + if (self->exports == 0) { + self->flags |= Py_MEMORYVIEW_RELEASED; + assert(self->mbuf->exports > 0); + if (--self->mbuf->exports == 0) + mbuf_release(self->mbuf); + return 0; + } + if (self->exports > 0) { + (void)PyErr_Format(PyExc_BufferError, + "memoryview has %zd exported buffer%s", self->exports, + self->exports==1 ? "" : "s"); + return -1; + } + + Py_FatalError("_memory_release(): negative export count"); + return -1; +} + +static PyObject * +memory_release(PyMemoryViewObject *self) +{ + if (_memory_release(self) < 0) + return NULL; + Py_RETURN_NONE; +} + +static void +memory_dealloc(PyMemoryViewObject *self) +{ + assert(self->exports == 0); + _PyObject_GC_UNTRACK(self); + (void)_memory_release(self); + Py_CLEAR(self->mbuf); + PyObject_GC_Del(self); +} + +static int +memory_traverse(PyMemoryViewObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->mbuf); + return 0; +} + +static int +memory_clear(PyMemoryViewObject *self) +{ + (void)_memory_release(self); + Py_CLEAR(self->mbuf); + return 0; +} + +static PyObject * +memory_enter(PyObject *self, PyObject *args) +{ + CHECK_RELEASED(self); + Py_INCREF(self); + return self; +} + +static PyObject * +memory_exit(PyObject *self, PyObject *args) +{ + return memory_release((PyMemoryViewObject *)self); +} + + +/****************************************************************************/ +/* Casting format and shape */ +/****************************************************************************/ + +#define IS_BYTE_FORMAT(f) (f == 'b' || f == 'B' || f == 'c') +const char *format_chars = "0123456789xcbB?hHiIlLqQfdspP"; + +Py_LOCAL_INLINE(Py_ssize_t) +get_native_fmtchar(char *result, const char *fmt) +{ + Py_ssize_t size = -1; + + if (fmt[0] == '@') fmt++; + + switch (fmt[0]) { + case 'c': case 'b': case 'B': size = sizeof(char); break; + case 'h': case 'H': size = sizeof(short); break; + case 'i': case 'I': size = sizeof(int); break; + case 'l': case 'L': size = sizeof(long); break; + #ifdef HAVE_LONG_LONG + case 'q': case 'Q': size = sizeof(PY_LONG_LONG); break; + #endif + case 'f': size = sizeof(float); break; + case 'd': size = sizeof(double); break; + #ifdef HAVE_C99_BOOL + case '?': size = sizeof(_Bool); break; + #else + case '?': size = sizeof(char); break; + #endif + case 'P': size = sizeof(void *); break; + } + + if (size > 0 && fmt[1] == '\0') { + *result = fmt[0]; + return size; + } + + switch (fmt[0]) { + case '<': case '>': case '=': case '!': fmt++; + } + + for (; *fmt != '\0'; fmt++) { + if (!strchr(format_chars, (unsigned char)*fmt)) { + PyErr_Format(PyExc_ValueError, + "memoryview: invalid format: %s", fmt); + return -1; + } + } + + PyErr_Format(PyExc_NotImplementedError, + "memoryview: unsupported format: %s", fmt); + return -1; +} + +/* Cast a memoryview's data type to 'format'. The input array must be + C-contiguous. At least one of input-format, output-format must have + byte size. The output array is 1-D, with the same byte length as the + input array. Thus, view->len must be a multiple of the new itemsize. */ +static int +cast_to_1D(PyMemoryViewObject *mv, PyObject *format) +{ + Py_buffer *view = &mv->view; + PyObject *asciifmt; + char srcchar, destchar; + Py_ssize_t itemsize; + int ret = -1; + + assert(view->ndim >= 1); + assert(Py_SIZE(mv) == 3*view->ndim); + assert(view->shape == mv->ob_array); + assert(view->strides == mv->ob_array + view->ndim); + assert(view->suboffsets == mv->ob_array + 2*view->ndim); + + if (get_native_fmtchar(&srcchar, view->format) < 0) + return ret; + + asciifmt = PyUnicode_AsASCIIString(format); + if (asciifmt == NULL) + return ret; + + itemsize = get_native_fmtchar(&destchar, PyBytes_AS_STRING(asciifmt)); + if (itemsize < 0) + goto out; + + if (!IS_BYTE_FORMAT(srcchar) && !IS_BYTE_FORMAT(destchar)) { + PyErr_SetString(PyExc_TypeError, + "memoryview: cannot cast between two non-byte formats"); + goto out; + } + if (view->len % itemsize) { + PyErr_SetString(PyExc_TypeError, + "memoryview: length is not a multiple of itemsize"); + goto out; + } + + strncpy(mv->format, PyBytes_AS_STRING(asciifmt), + Py_MEMORYVIEW_MAX_FORMAT); + mv->format[Py_MEMORYVIEW_MAX_FORMAT-1] = '\0'; + view->format = mv->format; + view->itemsize = itemsize; + + view->ndim = 1; + view->shape[0] = view->len / view->itemsize; + view->strides[0] = view->itemsize; + view->suboffsets = NULL; + + init_flags(mv); + + ret = 0; + +out: + Py_DECREF(asciifmt); + return ret; +} + +/* The memoryview must have space for 3*len(seq) elements. */ +static int +copy_shape(Py_ssize_t *shape, const PyObject *seq, Py_ssize_t ndim) +{ + Py_ssize_t x, i; + + for (i = 0; i < ndim; i++) { + PyObject *tmp = PySequence_Fast_GET_ITEM(seq, i); + if (!PyLong_Check(tmp)) { + PyErr_SetString(PyExc_TypeError, + "memoryview.cast(): elements of shape must be integers"); + return -1; + } + x = PyLong_AsSsize_t(tmp); + if (x == -1 && PyErr_Occurred()) { + return -1; + } + if (x <= 0) { + /* In general elements of shape may be 0, but not for casting. */ + PyErr_Format(PyExc_ValueError, + "memoryview.cast(): elements of shape must be integers > 0"); + return -1; + } + shape[i] = x; + } + + return 0; +} + +/* Cast a 1-D array to a new shape. The result array will be C-contiguous. + If the result array does not have exactly the same byte length as the + input array, raise ValueError. */ +static int +cast_to_ND(PyMemoryViewObject *mv, const PyObject *shape, int ndim) +{ + Py_buffer *view = &mv->view; + Py_ssize_t len; + + assert(view->ndim == 1); /* ndim from cast_to_1D() */ + assert(Py_SIZE(mv) == 3*(ndim==0?1:ndim)); /* ndim of result array */ + assert(view->shape == mv->ob_array); + assert(view->strides == mv->ob_array + (ndim==0?1:ndim)); + assert(view->suboffsets == NULL); + + view->ndim = ndim; + if (view->ndim == 0) { + view->shape = NULL; + view->strides = NULL; + } + else { + if (copy_shape(view->shape, shape, ndim) < 0) + return -1; + init_strides_from_shape(view); + } + + len = view->len; + init_len(view); + if (view->len != len) { + PyErr_SetString(PyExc_TypeError, + "memoryview: product(shape) * itemsize != buffer size"); + return -1; + } + + init_flags(mv); + + return 0; +} + +static int +zero_in_shape(PyMemoryViewObject *mv) +{ + Py_buffer *view = &mv->view; + Py_ssize_t i; + + for (i = 0; i < view->ndim; i++) + if (view->shape[i] == 0) + return 1; + + return 0; +} + +/* + Cast a copy of 'self' to a different view. The input view must + be C-contiguous. The function always casts the input view to a + 1-D output according to 'format'. At least one of input-format, + output-format must have byte size. + + If 'shape' is given, the 1-D view from the previous step will + be cast to a C-contiguous view with new shape and strides. + + All casts must result in views that will have the exact byte + size of the original input. Otherwise, an error is raised. +*/ +static PyObject * +memory_cast(PyMemoryViewObject *self, PyObject *args, PyObject *kwds) +{ + static char *kwlist[] = {"format", "shape", NULL}; + PyMemoryViewObject *mv = NULL; + PyObject *shape = NULL; + PyObject *format; + Py_ssize_t ndim = 1; + + CHECK_RELEASED(self); + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O", kwlist, + &format, &shape)) { + return NULL; + } + if (!PyUnicode_Check(format)) { + PyErr_SetString(PyExc_TypeError, + "memoryview: format argument must be a string"); + return NULL; + } + if (!MV_C_CONTIGUOUS(self->flags)) { + PyErr_SetString(PyExc_TypeError, + "memoryview: cannot cast non-contiguous view"); + return NULL; + } + if (zero_in_shape(self)) { + PyErr_SetString(PyExc_TypeError, + "memoryview: cannot cast view with zeros in shape or strides"); + return NULL; + } + if (shape) { + CHECK_LIST_OR_TUPLE(shape) + ndim = PySequence_Fast_GET_SIZE(shape); + if (ndim > INT_MAX) { + PyErr_SetString(PyExc_ValueError, + "memoryview: ndim must not exceed INT_MAX"); + return NULL; + } + if (self->view.ndim != 1 && ndim != 1) { + PyErr_SetString(PyExc_TypeError, + "memoryview: cast must be 1D -> ND or ND -> 1D"); + return NULL; + } + } + + mv = (PyMemoryViewObject *) + mbuf_add_incomplete_view(self->mbuf, &self->view, ndim==0 ? 1 : (int)ndim); + if (mv == NULL) + return NULL; + + if (cast_to_1D(mv, format) < 0) + goto error; + if (shape && cast_to_ND(mv, shape, (int)ndim) < 0) + goto error; + + return (PyObject *)mv; + +error: + Py_DECREF(mv); + return NULL; +} + + +/**************************************************************************/ +/* getbuffer */ +/**************************************************************************/ + +static int +memory_getbuf(PyMemoryViewObject *self, Py_buffer *view, int flags) +{ + Py_buffer *base = &self->view; + int baseflags = self->flags; + + CHECK_RELEASED_INT(self); + + /* start with complete information */ + *view = *base; + view->obj = NULL; + + if (REQ_WRITABLE(flags) && base->readonly) { + PyErr_SetString(PyExc_BufferError, + "memoryview: underlying buffer is not writable"); + return -1; + } + if (!REQ_FORMAT(flags)) { + /* NULL indicates that the buffer's data type has been cast to 'B'. + view->itemsize is the _previous_ itemsize. If shape is present, + the equality product(shape) * itemsize = len still holds at this + point. The equality calcsize(format) = itemsize does _not_ hold + from here on! */ + view->format = NULL; + } + + if (REQ_C_CONTIGUOUS(flags) && !MV_C_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "memoryview: underlying buffer is not C-contiguous"); + return -1; + } + if (REQ_F_CONTIGUOUS(flags) && !MV_F_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "memoryview: underlying buffer is not Fortran contiguous"); + return -1; + } + if (REQ_ANY_CONTIGUOUS(flags) && !MV_ANY_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "memoryview: underlying buffer is not contiguous"); + return -1; + } + if (!REQ_INDIRECT(flags) && (baseflags & Py_MEMORYVIEW_PIL)) { + PyErr_SetString(PyExc_BufferError, + "memoryview: underlying buffer requires suboffsets"); + return -1; + } + if (!REQ_STRIDES(flags)) { + if (!MV_C_CONTIGUOUS(baseflags)) { + PyErr_SetString(PyExc_BufferError, + "memoryview: underlying buffer is not C-contiguous"); + return -1; + } + view->strides = NULL; + } + if (!REQ_SHAPE(flags)) { + /* PyBUF_SIMPLE or PyBUF_WRITABLE: at this point buf is C-contiguous, + so base->buf = ndbuf->data. */ + if (view->format != NULL) { + /* PyBUF_SIMPLE|PyBUF_FORMAT and PyBUF_WRITABLE|PyBUF_FORMAT do + not make sense. */ + PyErr_Format(PyExc_BufferError, + "ndarray: cannot cast to unsigned bytes if the format flag " + "is present"); + return -1; + } + /* product(shape) * itemsize = len and calcsize(format) = itemsize + do _not_ hold from here on! */ + view->ndim = 1; + view->shape = NULL; + } + + + view->obj = (PyObject *)self; + Py_INCREF(view->obj); + self->exports++; + + return 0; +} + +static void +memory_releasebuf(PyMemoryViewObject *self, Py_buffer *view) +{ + self->exports--; + return; + /* PyBuffer_Release() decrements view->obj after this function returns. */ +} + +/* Buffer methods */ +static PyBufferProcs memory_as_buffer = { + (getbufferproc)memory_getbuf, /* bf_getbuffer */ + (releasebufferproc)memory_releasebuf, /* bf_releasebuffer */ +}; + + +/****************************************************************************/ +/* Optimized pack/unpack for all native format specifiers */ +/****************************************************************************/ + +/* Timings with the ndarray from _testbuffer.c indicate that using the + struct module is orders of magnitude slower than the two functions + below. */ + +/* Unpack a single item. 'fmt' can be any native format character in struct + module syntax. This function is very sensitive to small changes. With this + layout gcc automatically generates a fast jump table. */ +Py_LOCAL_INLINE(PyObject *) +unpack_single(const char *ptr, const char *fmt) +{ + unsigned PY_LONG_LONG llu; + unsigned long lu; + PY_LONG_LONG lld; + long ld; + double d; + unsigned char uc; + + switch (fmt[0]) { + + /* signed integers and fast path for 'B' */ + case 'B': uc = *((unsigned char *)ptr); goto convert_uc; + case 'b': ld = *((signed char *)ptr); goto convert_ld; + case 'h': ld = *((short *)ptr); goto convert_ld; + case 'i': ld = *((int *)ptr); goto convert_ld; + case 'l': ld = *((long *)ptr); goto convert_ld; + + /* boolean */ + #ifdef HAVE_C99_BOOL + case '?': ld = (long)*((_Bool *)ptr); goto convert_bool; + #else + case '?': ld = *((char *)ptr); goto convert_bool; + #endif + + /* unsigned integers */ + case 'H': lu = *((unsigned short *)ptr); goto convert_lu; + case 'I': lu = *((unsigned int *)ptr); goto convert_lu; + case 'L': lu = *((unsigned long *)ptr); goto convert_lu; + + /* native 64-bit */ + #ifdef HAVE_LONG_LONG + case 'q': lld = *((PY_LONG_LONG *)ptr); goto convert_lld; + case 'Q': llu = *((unsigned PY_LONG_LONG *)ptr); goto convert_llu; + #endif + + /* floats */ + #ifndef PY_NO_SHORT_FLOAT_REPR + case 'f': d = *((float *)ptr); goto convert_double; + case 'd': d = *((double *)ptr); goto convert_double; + #endif + + /* bytes object */ + case 'c': goto convert_bytes; + /* pointer */ + case 'P': goto convert_pointer; + + default: goto err_format; + } + +convert_uc: + /* PyLong_FromUnsignedLong() is slower */ + return PyLong_FromLong(uc); +convert_ld: + return PyLong_FromLong(ld); +convert_lu: + return PyLong_FromUnsignedLong(lu); +convert_lld: + return PyLong_FromLongLong(lld); +convert_llu: + return PyLong_FromUnsignedLongLong(llu); +convert_double: + return PyFloat_FromDouble(d); +convert_bool: + return PyBool_FromLong(ld); +convert_bytes: + return PyBytes_FromStringAndSize(ptr, 1); +convert_pointer: + return PyLong_FromVoidPtr(*((void **)ptr)); +err_format: + PyErr_Format(PyExc_NotImplementedError, + "memoryview: format %s not supported", fmt); + return NULL; +} + +/* Pack a single item. 'fmt' can be any native format character in + struct module syntax. */ +static int +pack_single(char *ptr, PyObject *item, const char *fmt) +{ + PyObject *bytes; + unsigned PY_LONG_LONG llu; + unsigned long lu; + PY_LONG_LONG lld; + long ld; + double d; + void *p; + + switch (fmt[0]) { + /* signed integers */ + case 'b': case 'h': case 'i': case 'l': + ld = PyLong_AsLong(item); + if (ld == -1 && PyErr_Occurred()) + goto err_occurred; + switch (fmt[0]) { + case 'b': + if (ld < SCHAR_MIN || ld > SCHAR_MAX) goto err_range; + *((signed char *)ptr) = (signed char)ld; break; + case 'h': + if (ld < SHRT_MIN || ld > SHRT_MAX) goto err_range; + *((short *)ptr) = (short)ld; break; + case 'i': + if (ld < INT_MIN || ld > INT_MAX) goto err_range; + *((int *)ptr) = (int)ld; break; + default: /* 'l' */ + *((long *)ptr) = (long)ld; break; + } + break; + + /* unsigned integers */ + case 'B': case 'H': case 'I': case 'L': + lu = PyLong_AsUnsignedLong(item); + if (lu == (unsigned long)-1 && PyErr_Occurred()) + goto err_occurred; + switch (fmt[0]) { + case 'B': + if (lu > UCHAR_MAX) goto err_range; + *((unsigned char *)ptr) = (unsigned char)lu; break; + case 'H': + if (lu > USHRT_MAX) goto err_range; + *((unsigned short *)ptr) = (unsigned short)lu; break; + case 'I': + if (lu > UINT_MAX) goto err_range; + *((unsigned int *)ptr) = (unsigned int)lu; break; + default: /* 'L' */ + *((unsigned long *)ptr) = (unsigned long)lu; break; + } + break; + + /* native 64-bit */ + #ifdef HAVE_LONG_LONG + case 'q': + lld = PyLong_AsLongLong(item); + if (lld == -1 && PyErr_Occurred()) + goto err_occurred; + *((PY_LONG_LONG *)ptr) = lld; break; + break; + case 'Q': + llu = PyLong_AsUnsignedLongLong(item); + if (llu == (unsigned PY_LONG_LONG)-1 && PyErr_Occurred()) + goto err_occurred; + *((unsigned PY_LONG_LONG *)ptr) = llu; break; + break; + #endif + + /* floats */ + #ifndef PY_NO_SHORT_FLOAT_REPR + case 'f': case 'd': + d = PyFloat_AsDouble(item); + if (d == -1.0 && PyErr_Occurred()) + goto err_occurred; + if (fmt[0] == 'f') { + float y = (float)d; + if (Py_IS_INFINITY(y) && !Py_IS_INFINITY(d)) + goto err_range; + *((float *)ptr) = y; + } + else { + *((double *)ptr) = d; + } + break; + #endif + + /* bool */ + case '?': + ld = PyObject_IsTrue(item); + if (PyErr_Occurred()) + return -1; + #ifdef HAVE_C99_BOOL + *((_Bool *)ptr) = (_Bool)ld; + #else + *ptr = (char)ld; + #endif + break; + + /* bytes object */ + case 'c': + bytes = PyObject_Bytes(item); + if (bytes == NULL) + goto err_occurred; + if (PyBytes_GET_SIZE(bytes) != 1) { + Py_DECREF(bytes); + goto err_range; + } + *ptr = PyBytes_AS_STRING(bytes)[0]; + Py_DECREF(bytes); + break; + + /* pointer */ + case 'P': + p = PyLong_AsVoidPtr(item); + if (PyErr_Occurred()) + goto err_occurred; + *(void **)ptr = p; + break; + default: goto err_format; + } + + return 0; + +err_occurred: + PyErr_Clear(); /* The individual error messages are incomprehensible. */ +err_range: + PyErr_Format(PyExc_ValueError, + "memoryview: invalid value for format \"%s\"", fmt); + return -1; +err_format: + PyErr_Format(PyExc_NotImplementedError, + "memoryview: format %s not supported", fmt); + return -1; +} + + +/****************************************************************************/ +/* Representations */ +/****************************************************************************/ + +/* allow explicit form of native format */ +Py_LOCAL_INLINE(const char *) +adjust_fmt(const Py_buffer *view) +{ + const char *fmt; + + fmt = (view->format[0] == '@') ? view->format+1 : view->format; + if (!(fmt[0] && fmt[1] == '\0')) { + PyErr_Format(PyExc_NotImplementedError, + "memoryview: unsupported format %s", view->format); + return NULL; + } + + return fmt; +} + + +/* Return a list representation of the memoryview. Currently only + one-dimensional buffers with native format strings are supported. */ +static PyObject * +memory_tolist(PyMemoryViewObject *mem, PyObject *noargs) +{ + const Py_buffer *view = &(mem->view); + PyObject *res, *item; + Py_ssize_t i; + const char *fmt; + const char *ptr; + + CHECK_RELEASED(mem); + + fmt = adjust_fmt(view); + if (fmt == NULL) + return NULL; + if (view->ndim == 0) { + ptr = view->buf; + return unpack_single(ptr, fmt); + } + if (view->ndim != 1) { + PyErr_SetString(PyExc_NotImplementedError, + "memoryview: tolist() only supports one-dimensional objects"); + return NULL; + } + + res = PyList_New(view->shape[0]); + if (res == NULL) + return NULL; + for (i=0, ptr=view->buf; i < view->shape[0]; ptr+=view->strides[0], i++) { + const char *xptr = ADJUST_PTR(ptr, view->suboffsets); + item = unpack_single(xptr, fmt); + if (item == NULL) { + Py_DECREF(res); + return NULL; + } + PyList_SET_ITEM(res, i, item); + } + return res; +} + +static PyObject * +memory_tobytes(PyMemoryViewObject *self, PyObject *noargs) +{ + int baseflags = self->flags; + + CHECK_RELEASED(self); + + if (!MV_ANY_CONTIGUOUS(baseflags)) { + /* Issue #12834 */ + PyErr_SetString(PyExc_NotImplementedError, + "memoryview: tobytes() currently requires a contiguous buffer"); + return NULL; + } + return PyObject_CallFunctionObjArgs((PyObject *)&PyBytes_Type, self, NULL); +} + +static PyObject * +memory_repr(PyMemoryViewObject *self) +{ + if (self->flags & Py_MEMORYVIEW_RELEASED) + return PyUnicode_FromFormat("", self); + else + return PyUnicode_FromFormat("", self); +} + + +/**************************************************************************/ +/* Indexing and slicing */ +/**************************************************************************/ + +/* Get the pointer to the item at index. */ +static char * +ptr_from_index(Py_buffer *view, Py_ssize_t index) +{ + char *ptr; + Py_ssize_t nitems; /* items in the first dimension */ + + assert(view->shape); + assert(view->strides); + + nitems = view->shape[0]; + if (index < 0) { + index += nitems; + } + if (index < 0 || index >= nitems) { + PyErr_SetString(PyExc_IndexError, "index out of bounds"); + return NULL; + } + + ptr = (char *)view->buf; + ptr += view->strides[0] * index; + + ptr = ADJUST_PTR(ptr, view->suboffsets); + + return ptr; +} + +static PyObject * +memory_item(PyMemoryViewObject *self, Py_ssize_t index) +{ + Py_buffer *view = &(self->view); + const char *fmt; + + CHECK_RELEASED(self); + + fmt = adjust_fmt(view); + if (fmt == NULL) + return NULL; + + if (view->ndim == 0) { + PyErr_SetString(PyExc_TypeError, "invalid indexing of 0-dim memory"); + return NULL; + } + if (view->ndim == 1) { + char *ptr = ptr_from_index(view, index); + if (ptr == NULL) + return NULL; + return unpack_single(ptr, fmt); + } + + PyErr_SetString(PyExc_NotImplementedError, + "multi-dimensional sub-views are not implemented"); + return NULL; +} + +Py_LOCAL_INLINE(int) +init_slice(Py_buffer *base, PyObject *key, int dim) +{ + Py_ssize_t start, stop, step, slicelength; + + if (PySlice_GetIndicesEx(key, base->shape[dim], + &start, &stop, &step, &slicelength) < 0) { + return -1; + } + + + if (base->suboffsets == NULL || dim == 0) { + adjust_buf: + base->buf = (char *)base->buf + base->strides[dim] * start; + } + else { + Py_ssize_t n = dim-1; + while (n >= 0 && base->suboffsets[n] < 0) + n--; + if (n < 0) + goto adjust_buf; /* all suboffsets are negative */ + base->suboffsets[n] = base->suboffsets[n] + base->strides[dim] * start; + } + base->shape[dim] = slicelength; + base->strides[dim] = base->strides[dim] * step; + + return 0; +} + +static int +is_multislice(PyObject *key) +{ + Py_ssize_t size, i; + + if (!PyTuple_Check(key)) + return 0; + size = PyTuple_GET_SIZE(key); + if (size == 0) + return 0; + + for (i = 0; i < PyTuple_GET_SIZE(key); i++) { + PyObject *x = PyTuple_GET_ITEM(key, i); + if (!PySlice_Check(x)) + return 0; + } + return 1; +} + +/* + mem[obj] returns an object holding the data for one element if + obj fully indexes the memory view or another memory-view object + if it does not. + + 0-d memory-view objects can be referenced using ... or () but + not with anything else. + */ +static PyObject * +memory_subscript(PyMemoryViewObject *self, PyObject *key) +{ + Py_buffer *view; + view = &(self->view); + + CHECK_RELEASED(self); + + if (view->ndim == 0) { + if (PyTuple_Check(key) && PyTuple_GET_SIZE(key) == 0) { + const char *fmt = adjust_fmt(view); + if (fmt == NULL) + return NULL; + return unpack_single(view->buf, fmt); + } + else if (key == Py_Ellipsis) { + Py_INCREF(self); + return (PyObject *)self; + } + else { + PyErr_SetString(PyExc_TypeError, + "invalid indexing of 0-dim memory"); + return NULL; + } + } + if (PyIndex_Check(key)) { + Py_ssize_t result; + result = PyNumber_AsSsize_t(key, PyExc_IndexError); + if (result == -1 && PyErr_Occurred()) + return NULL; + return memory_item(self, result); + } + else if (PySlice_Check(key)) { + PyMemoryViewObject *sliced; + + sliced = (PyMemoryViewObject *)mbuf_add_view(self->mbuf, view); + if (sliced == NULL) + return NULL; + + if (init_slice(&sliced->view, key, 0) < 0) { + Py_DECREF(sliced); + return NULL; + } + init_len(&sliced->view); + init_flags(sliced); + + return (PyObject *)sliced; + } + else if (is_multislice(key)) { + PyErr_SetString(PyExc_NotImplementedError, + "multi-dimensional slicing is not implemented"); + return NULL; + } + + PyErr_SetString(PyExc_TypeError, "memoryview: invalid slice key"); + return NULL; +} + +/* This function can be used as the base case for recursive multi-dimensional + copying, hence the name. It has a fast path for the speed critical case + of copying contiguous arrays. */ +static int +copy_base(const Py_ssize_t *shape, Py_ssize_t itemsize, + char *dptr, const Py_ssize_t *dstrides, const Py_ssize_t *dsuboffsets, + char *sptr, const Py_ssize_t *sstrides, const Py_ssize_t *ssuboffsets, + char *mem) +{ + if (!HAVE_PTR(dsuboffsets) && !HAVE_PTR(ssuboffsets) && + dstrides[0] == itemsize && sstrides[0] == itemsize) { /* contiguous */ + Py_ssize_t size = shape[0] * itemsize; + if (dptr + size < sptr || sptr + size < dptr) + memcpy(dptr, sptr, size); /* no overlapping */ + else + memmove(dptr, sptr, size); + } + else { + /* non-contiguous */ + char *p, *xmem = mem; + Py_ssize_t i; + if (mem == NULL) { + xmem = PyMem_Malloc(shape[0] * itemsize); + if (xmem == NULL) { + PyErr_NoMemory(); + return -1; + } + } + for (i=0, p=xmem; i < shape[0]; p+=itemsize, sptr+=sstrides[0], i++) { + char *xsptr = ADJUST_PTR(sptr, ssuboffsets); + memcpy(p, xsptr, itemsize); + } + for (i=0, p=xmem; i < shape[0]; p+=itemsize, dptr+=dstrides[0], i++) { + char *xdptr = ADJUST_PTR(dptr, dsuboffsets); + memcpy(xdptr, p, itemsize); + } + if (mem == NULL) + PyMem_Free(xmem); + } + return 0; +} + +static int +memory_ass_sub(PyMemoryViewObject *self, PyObject *key, PyObject *value) +{ + Py_buffer *view = &(self->view); + Py_buffer src; + const char *fmt; + char *ptr; + + CHECK_RELEASED_INT(self); + + fmt = adjust_fmt(view); + if (fmt == NULL) + return -1; + + if (view->readonly) { + PyErr_SetString(PyExc_TypeError, "cannot modify read-only memory"); + return -1; + } + if (value == NULL) { + PyErr_SetString(PyExc_TypeError, "cannot delete memory"); + return -1; + } + if (view->ndim == 0) { + if (key == Py_Ellipsis || + (PyTuple_Check(key) && PyTuple_GET_SIZE(key)==0)) { + ptr = (char *)view->buf; + return pack_single(ptr, value, fmt); + } + else { + PyErr_SetString(PyExc_TypeError, + "invalid indexing of 0-dim memory"); + return -1; + } + } + if (view->ndim != 1) { + PyErr_SetString(PyExc_NotImplementedError, + "memoryview assignments are currently restricted to ndim = 1"); + return -1; + } + + if (PyIndex_Check(key)) { + Py_ssize_t index = PyNumber_AsSsize_t(key, PyExc_IndexError); + if (index == -1 && PyErr_Occurred()) + return -1; + else { + ptr = ptr_from_index(view, index); + if (ptr == NULL) + return -1; + } + return pack_single(ptr, value, fmt); + } + /* one-dimensional: fast path */ + if (PySlice_Check(key) && view->ndim == 1) { + Py_buffer dest; /* sliced view */ + Py_ssize_t arrays[3]; + int ret = -1; + + /* rvalue must be an exporter */ + if (PyObject_GetBuffer(value, &src, PyBUF_FULL_RO) == -1) + return ret; + + dest = *view; + dest.shape = &arrays[0]; dest.shape[0] = view->shape[0]; + dest.strides = &arrays[1]; dest.strides[0] = view->strides[0]; + if (view->suboffsets) { + dest.suboffsets = &arrays[2]; dest.suboffsets[0] = view->suboffsets[0]; + } + + if (init_slice(&dest, key, 0) < 0) + goto end_block; + init_len(&dest); + + if (strcmp(dest.format, src.format) != 0 || + dest.itemsize != src.itemsize || + dest.ndim != src.ndim || + dest.shape[0] != src.shape[0]) { + PyErr_SetString(PyExc_ValueError, + "memoryview assignment: lvalue and rvalue have " + "different structures"); + goto end_block; + } + + ret = 0; + if (dest.shape[0] > 0) { /* shape[0] == 0: nothing to copy */ + ret = copy_base(dest.shape, dest.itemsize, + dest.buf, dest.strides, dest.suboffsets, + src.buf, src.strides, src.suboffsets, NULL); + } + end_block: + PyBuffer_Release(&src); + return ret; + } + else if (PySlice_Check(key) || is_multislice(key)) { + /* Call memory_subscript() to produce a sliced lvalue, then copy + rvalue into lvalue. This is already implemented in _testbuffer.c. */ + PyErr_SetString(PyExc_NotImplementedError, + "memoryview slice assignments are currently restricted " + "to ndim = 1"); + return -1; + } + + PyErr_SetString(PyExc_TypeError, "memoryview: invalid slice key"); + return -1; +} + +static Py_ssize_t +memory_length(PyMemoryViewObject *self) +{ + CHECK_RELEASED_INT(self); + return self->view.ndim == 0 ? 1 : self->view.shape[0]; +} + +/* As mapping */ +static PyMappingMethods memory_as_mapping = { + (lenfunc)memory_length, /* mp_length */ + (binaryfunc)memory_subscript, /* mp_subscript */ + (objobjargproc)memory_ass_sub, /* mp_ass_subscript */ +}; + +/* As sequence */ +static PySequenceMethods memory_as_sequence = { + 0, /* sq_length */ + 0, /* sq_concat */ + 0, /* sq_repeat */ + (ssizeargfunc)memory_item, /* sq_item */ +}; + + +/**************************************************************************/ +/* Comparisons */ +/**************************************************************************/ + +Py_LOCAL_INLINE(int) +unpack_cmp(const char *p, const char *q, const char *fmt) +{ + switch (fmt[0]) { + + /* signed integers and fast path for 'B' */ + case 'B': return *((unsigned char *)p) == *((unsigned char *)q); + case 'b': return *((signed char *)p) == *((signed char *)q); + case 'h': return *((short *)p) == *((short *)q); + case 'i': return *((int *)p) == *((int *)q); + case 'l': return *((long *)p) == *((long *)q); + + /* boolean */ + #ifdef HAVE_C99_BOOL + case '?': return *((_Bool *)p) == *((_Bool *)q); + #else + case '?': return *((char *)p) == *((char *)q); + #endif + + /* unsigned integers */ + case 'H': return *((unsigned short *)p) == *((unsigned short *)q); + case 'I': return *((unsigned int *)p) == *((unsigned int *)q); + case 'L': return *((unsigned long *)p) == *((unsigned long *)q); + + /* native 64-bit */ + #ifdef HAVE_LONG_LONG + case 'q': return *((PY_LONG_LONG *)p) == *((PY_LONG_LONG *)q); + case 'Q': return *((unsigned PY_LONG_LONG *)p) == *((unsigned PY_LONG_LONG *)q); + #endif + + /* floats */ + #ifndef PY_NO_SHORT_FLOAT_REPR + /* XXX DBL_EPSILON? */ + case 'f': return *((float *)p) == *((float *)q); + case 'd': return *((double *)p) == *((double *)q); + #endif + + /* bytes object */ + case 'c': return *p == *q; + /* pointer */ + case 'P': return *((void **)p) == *((void **)q); + + /* Py_NotImplemented */ + default: return -1; + } +} + +static PyObject * +memory_richcompare(PyObject *v, PyObject *w, int op) +{ + PyObject *res; + Py_buffer wbuf, *vv, *ww = NULL; + const char *vfmt, *wfmt; + int equal = -1; /* Py_NotImplemented */ + + if (op != Py_EQ && op != Py_NE) + goto _result; /* Py_NotImplemented */ + + assert(PyMemoryView_Check(v)); + if (BASE_INACCESSIBLE(v)) { + equal = (v == w); + goto _result; + } + vv = VIEW_ADDR(v); + + if (PyMemoryView_Check(w)) { + if (BASE_INACCESSIBLE(w)) { + equal = (v == w); + goto _result; + } + ww = VIEW_ADDR(w); + } + else { + if (PyObject_GetBuffer(w, &wbuf, PyBUF_FULL_RO) == -1) { + PyErr_Clear(); + goto _result; /* Py_NotImplemented */ + } + ww = &wbuf; + } + + vfmt = adjust_fmt(vv); + wfmt = adjust_fmt(ww); + if (vfmt == NULL || wfmt == NULL) { + PyErr_Clear(); + goto _result; /* Py_NotImplemented */ + } + + if (vv->ndim > 1 || ww->ndim > 1) + goto _result; /* Py_NotImplemented */ + + equal = 0; + if (vv->len != ww->len || + vfmt[0] != wfmt[0] || + vv->itemsize != ww->itemsize || + vv->ndim != ww->ndim) + goto _result; /* not equal */ + + /* Since shape[0] = len / itemsize, the shapes are equal. */ + if (vv->ndim == 0) { + equal = unpack_cmp(vv->buf, ww->buf, vfmt); + } + else { + char *p, *q; + Py_ssize_t i; + + equal = 1; /* cover shape[0] == 0 */ + for (i=0, p=vv->buf, q=ww->buf; i < vv->shape[0]; + p+=vv->strides[0], q+=ww->strides[0], i++) { + char *xp = ADJUST_PTR(p, vv->suboffsets); + char *xq = ADJUST_PTR(q, ww->suboffsets); + equal = unpack_cmp(xp, xq, vfmt); + if (equal <= 0) + break; + } + } + +_result: + if (equal < 0) + res = Py_NotImplemented; + else if ((equal && op == Py_EQ) || (!equal && op == Py_NE)) + res = Py_True; + else + res = Py_False; + + if (ww == &wbuf) + PyBuffer_Release(ww); + Py_INCREF(res); + return res; +} + + +/**************************************************************************/ +/* Get a contiguous view */ +/**************************************************************************/ + static void _strided_copy_nd(char *dest, char *src, int nd, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fort) @@ -247,6 +1886,7 @@ PyBUF_READ buffer only needs to be read-only PyBUF_WRITE buffer needs to be writable (give error if not contiguous) + XXX PyBUF_SHADOW has been removed: issue #10293 PyBUF_SHADOW buffer needs to be writable so shadow it with a contiguous buffer if it is not. The view will point to the shadow buffer which can be written to and then @@ -262,7 +1902,6 @@ PyMemoryViewObject *mem; PyObject *bytes; Py_buffer *view; - int flags; char *dest; if (!PyObject_CheckBuffer(obj)) { @@ -271,26 +1910,20 @@ return NULL; } - mem = PyObject_GC_New(PyMemoryViewObject, &PyMemoryView_Type); + mem = (PyMemoryViewObject *)PyMemoryView_FromObject(obj); if (mem == NULL) return NULL; view = &mem->view; - flags = PyBUF_FULL_RO; - switch(buffertype) { - case PyBUF_WRITE: - flags = PyBUF_FULL; - break; - } - - if (PyObject_GetBuffer(obj, view, flags) != 0) { + if (buffertype == PyBUF_WRITE && view->readonly) { + PyErr_SetString(PyExc_BufferError, + "underlying buffer is not writable"); Py_DECREF(mem); return NULL; } if (PyBuffer_IsContiguous(view, fort)) { /* no copy needed */ - _PyObject_GC_TRACK(mem); return (PyObject *)mem; } /* otherwise a copy is needed */ @@ -298,9 +1931,18 @@ Py_DECREF(mem); PyErr_SetString(PyExc_BufferError, "writable contiguous buffer requested " - "for a non-contiguousobject."); + "for a non-contiguous object."); return NULL; } + + PyErr_SetString(PyExc_NotImplementedError, + "feature is not implemented"); + Py_DECREF(mem); + return NULL; + /* XXX NOT REACHED: The following lines need a test case. Currently + 'dest' is filled in but not used: 'mem' is returned unchanged. + */ + bytes = PyBytes_FromStringAndSize(NULL, view->len); if (bytes == NULL) { Py_DECREF(mem); @@ -323,11 +1965,58 @@ } PyBuffer_Release(view); /* XXX ? */ } - _PyObject_GC_TRACK(mem); return (PyObject *)mem; } +/**************************************************************************/ +/* getters */ +/**************************************************************************/ + +static PyObject * +_IntTupleFromSsizet(int len, Py_ssize_t *vals) +{ + int i; + PyObject *o; + PyObject *intTuple; + + if (vals == NULL) + return PyTuple_New(0); + + intTuple = PyTuple_New(len); + if (!intTuple) + return NULL; + for (i=0; iview; + + CHECK_RELEASED(self); + if (view->obj == NULL) { + Py_RETURN_NONE; + } + Py_INCREF(view->obj); + return view->obj; +} + +static PyObject * +memory_len_get(PyMemoryViewObject *self) +{ + CHECK_RELEASED(self); + return PyLong_FromSsize_t(self->view.len); +} + static PyObject * memory_format_get(PyMemoryViewObject *self) { @@ -343,31 +2032,6 @@ } static PyObject * -_IntTupleFromSsizet(int len, Py_ssize_t *vals) -{ - int i; - PyObject *o; - PyObject *intTuple; - - if (vals == NULL) { - Py_INCREF(Py_None); - return Py_None; - } - intTuple = PyTuple_New(len); - if (!intTuple) - return NULL; - for (i=0; iview.ndim); } -static PyGetSetDef memory_getsetlist[] ={ - {"format", (getter)memory_format_get, NULL, NULL}, +static PyObject * +memory_c_contiguous(PyMemoryViewObject *self, PyObject *dummy) +{ + CHECK_RELEASED(self); + return PyBool_FromLong(MV_C_CONTIGUOUS(self->flags)); +} + +static PyObject * +memory_f_contiguous(PyMemoryViewObject *self, PyObject *dummy) +{ + CHECK_RELEASED(self); + return PyBool_FromLong(MV_F_CONTIGUOUS(self->flags)); +} + +static PyObject * +memory_contiguous(PyMemoryViewObject *self, PyObject *dummy) +{ + CHECK_RELEASED(self); + return PyBool_FromLong(MV_ANY_CONTIGUOUS(self->flags)); +} + +static PyGetSetDef memory_getsetlist[] = { + {"obj", (getter)memory_obj_get, NULL, NULL}, + {"len", (getter)memory_len_get, NULL, NULL}, + {"readonly", (getter)memory_readonly_get, NULL, NULL}, {"itemsize", (getter)memory_itemsize_get, NULL, NULL}, + {"format", (getter)memory_format_get, NULL, NULL}, + {"ndim", (getter)memory_ndim_get, NULL, NULL}, {"shape", (getter)memory_shape_get, NULL, NULL}, {"strides", (getter)memory_strides_get, NULL, NULL}, {"suboffsets", (getter)memory_suboffsets_get, NULL, NULL}, - {"readonly", (getter)memory_readonly_get, NULL, NULL}, - {"ndim", (getter)memory_ndim_get, NULL, NULL}, + {"c_contiguous", (getter)memory_c_contiguous, NULL, NULL}, + {"f_contiguous", (getter)memory_f_contiguous, NULL, NULL}, + {"contiguous", (getter)memory_contiguous, NULL, NULL}, {NULL, NULL, NULL, NULL}, }; -static PyObject * -memory_tobytes(PyMemoryViewObject *mem, PyObject *noargs) -{ - CHECK_RELEASED(mem); - return PyObject_CallFunctionObjArgs( - (PyObject *) &PyBytes_Type, mem, NULL); -} - -/* TODO: rewrite this function using the struct module to unpack - each buffer item */ - -static PyObject * -memory_tolist(PyMemoryViewObject *mem, PyObject *noargs) -{ - Py_buffer *view = &(mem->view); - Py_ssize_t i; - PyObject *res, *item; - char *buf; - - CHECK_RELEASED(mem); - if (strcmp(view->format, "B") || view->itemsize != 1) { - PyErr_SetString(PyExc_NotImplementedError, - "tolist() only supports byte views"); - return NULL; - } - if (view->ndim != 1) { - PyErr_SetString(PyExc_NotImplementedError, - "tolist() only supports one-dimensional objects"); - return NULL; - } - res = PyList_New(view->len); - if (res == NULL) - return NULL; - buf = view->buf; - for (i = 0; i < view->len; i++) { - item = PyLong_FromUnsignedLong((unsigned char) *buf); - if (item == NULL) { - Py_DECREF(res); - return NULL; - } - PyList_SET_ITEM(res, i, item); - buf++; - } - return res; -} - -static void -do_release(PyMemoryViewObject *self) -{ - if (self->view.obj != NULL) { - PyBuffer_Release(&(self->view)); - } - self->view.obj = NULL; - self->view.buf = NULL; -} - -static PyObject * -memory_enter(PyObject *self, PyObject *args) -{ - CHECK_RELEASED(self); - Py_INCREF(self); - return self; -} - -static PyObject * -memory_exit(PyObject *self, PyObject *args) -{ - do_release((PyMemoryViewObject *) self); - Py_RETURN_NONE; -} - static PyMethodDef memory_methods[] = { - {"release", memory_exit, METH_NOARGS}, - {"tobytes", (PyCFunction)memory_tobytes, METH_NOARGS, NULL}, - {"tolist", (PyCFunction)memory_tolist, METH_NOARGS, NULL}, - {"__enter__", memory_enter, METH_NOARGS}, - {"__exit__", memory_exit, METH_VARARGS}, - {NULL, NULL} /* sentinel */ -}; - - -static void -memory_dealloc(PyMemoryViewObject *self) -{ - _PyObject_GC_UNTRACK(self); - do_release(self); - PyObject_GC_Del(self); -} - -static PyObject * -memory_repr(PyMemoryViewObject *self) -{ - if (IS_RELEASED(self)) - return PyUnicode_FromFormat("", self); - else - return PyUnicode_FromFormat("", self); -} - -/* Sequence methods */ -static Py_ssize_t -memory_length(PyMemoryViewObject *self) -{ - CHECK_RELEASED_INT(self); - return get_shape0(&self->view); -} - -/* Alternate version of memory_subcript that only accepts indices. - Used by PySeqIter_New(). -*/ -static PyObject * -memory_item(PyMemoryViewObject *self, Py_ssize_t result) -{ - Py_buffer *view = &(self->view); - - CHECK_RELEASED(self); - if (view->ndim == 0) { - PyErr_SetString(PyExc_IndexError, - "invalid indexing of 0-dim memory"); - return NULL; - } - if (view->ndim == 1) { - /* Return a bytes object */ - char *ptr; - ptr = (char *)view->buf; - if (result < 0) { - result += get_shape0(view); - } - if ((result < 0) || (result >= get_shape0(view))) { - PyErr_SetString(PyExc_IndexError, - "index out of bounds"); - return NULL; - } - if (view->strides == NULL) - ptr += view->itemsize * result; - else - ptr += view->strides[0] * result; - if (view->suboffsets != NULL && - view->suboffsets[0] >= 0) { - ptr = *((char **)ptr) + view->suboffsets[0]; - } - return PyBytes_FromStringAndSize(ptr, view->itemsize); - } else { - /* Return a new memory-view object */ - Py_buffer newview; - memset(&newview, 0, sizeof(newview)); - /* XXX: This needs to be fixed so it actually returns a sub-view */ - return PyMemoryView_FromBuffer(&newview); - } -} - -/* - mem[obj] returns a bytes object holding the data for one element if - obj fully indexes the memory view or another memory-view object - if it does not. - - 0-d memory-view objects can be referenced using ... or () but - not with anything else. - */ -static PyObject * -memory_subscript(PyMemoryViewObject *self, PyObject *key) -{ - Py_buffer *view; - view = &(self->view); - - CHECK_RELEASED(self); - if (view->ndim == 0) { - if (key == Py_Ellipsis || - (PyTuple_Check(key) && PyTuple_GET_SIZE(key)==0)) { - Py_INCREF(self); - return (PyObject *)self; - } - else { - PyErr_SetString(PyExc_IndexError, - "invalid indexing of 0-dim memory"); - return NULL; - } - } - if (PyIndex_Check(key)) { - Py_ssize_t result; - result = PyNumber_AsSsize_t(key, NULL); - if (result == -1 && PyErr_Occurred()) - return NULL; - return memory_item(self, result); - } - else if (PySlice_Check(key)) { - Py_ssize_t start, stop, step, slicelength; - - if (PySlice_GetIndicesEx(key, get_shape0(view), - &start, &stop, &step, &slicelength) < 0) { - return NULL; - } - - if (step == 1 && view->ndim == 1) { - Py_buffer newview; - void *newbuf = (char *) view->buf - + start * view->itemsize; - int newflags = view->readonly - ? PyBUF_CONTIG_RO : PyBUF_CONTIG; - - /* XXX There should be an API to create a subbuffer */ - if (view->obj != NULL) { - if (PyObject_GetBuffer(view->obj, &newview, newflags) == -1) - return NULL; - } - else { - newview = *view; - } - newview.buf = newbuf; - newview.len = slicelength * newview.itemsize; - newview.format = view->format; - newview.shape = &(newview.smalltable[0]); - newview.shape[0] = slicelength; - newview.strides = &(newview.itemsize); - return PyMemoryView_FromBuffer(&newview); - } - PyErr_SetNone(PyExc_NotImplementedError); - return NULL; - } - PyErr_Format(PyExc_TypeError, - "cannot index memory using \"%.200s\"", - key->ob_type->tp_name); - return NULL; -} - - -/* Need to support assigning memory if we can */ -static int -memory_ass_sub(PyMemoryViewObject *self, PyObject *key, PyObject *value) -{ - Py_ssize_t start, len, bytelen; - Py_buffer srcview; - Py_buffer *view = &(self->view); - char *srcbuf, *destbuf; - - CHECK_RELEASED_INT(self); - if (view->readonly) { - PyErr_SetString(PyExc_TypeError, - "cannot modify read-only memory"); - return -1; - } - if (value == NULL) { - PyErr_SetString(PyExc_TypeError, - "cannot delete memory"); - return -1; - } - if (view->ndim != 1) { - PyErr_SetNone(PyExc_NotImplementedError); - return -1; - } - if (PyIndex_Check(key)) { - start = PyNumber_AsSsize_t(key, NULL); - if (start == -1 && PyErr_Occurred()) - return -1; - if (start < 0) { - start += get_shape0(view); - } - if ((start < 0) || (start >= get_shape0(view))) { - PyErr_SetString(PyExc_IndexError, - "index out of bounds"); - return -1; - } - len = 1; - } - else if (PySlice_Check(key)) { - Py_ssize_t stop, step; - - if (PySlice_GetIndicesEx(key, get_shape0(view), - &start, &stop, &step, &len) < 0) { - return -1; - } - if (step != 1) { - PyErr_SetNone(PyExc_NotImplementedError); - return -1; - } - } - else { - PyErr_Format(PyExc_TypeError, - "cannot index memory using \"%.200s\"", - key->ob_type->tp_name); - return -1; - } - if (PyObject_GetBuffer(value, &srcview, PyBUF_CONTIG_RO) == -1) { - return -1; - } - /* XXX should we allow assignment of different item sizes - as long as the byte length is the same? - (e.g. assign 2 shorts to a 4-byte slice) */ - if (srcview.itemsize != view->itemsize) { - PyErr_Format(PyExc_TypeError, - "mismatching item sizes for \"%.200s\" and \"%.200s\"", - view->obj->ob_type->tp_name, srcview.obj->ob_type->tp_name); - goto _error; - } - bytelen = len * view->itemsize; - if (bytelen != srcview.len) { - PyErr_SetString(PyExc_ValueError, - "cannot modify size of memoryview object"); - goto _error; - } - /* Do the actual copy */ - destbuf = (char *) view->buf + start * view->itemsize; - srcbuf = (char *) srcview.buf; - if (destbuf + bytelen < srcbuf || srcbuf + bytelen < destbuf) - /* No overlapping */ - memcpy(destbuf, srcbuf, bytelen); - else - memmove(destbuf, srcbuf, bytelen); - - PyBuffer_Release(&srcview); - return 0; - -_error: - PyBuffer_Release(&srcview); - return -1; -} - -static PyObject * -memory_richcompare(PyObject *v, PyObject *w, int op) -{ - Py_buffer vv, ww; - int equal = 0; - PyObject *res; - - vv.obj = NULL; - ww.obj = NULL; - if (op != Py_EQ && op != Py_NE) - goto _notimpl; - if ((PyMemoryView_Check(v) && IS_RELEASED(v)) || - (PyMemoryView_Check(w) && IS_RELEASED(w))) { - equal = (v == w); - goto _end; - } - if (PyObject_GetBuffer(v, &vv, PyBUF_CONTIG_RO) == -1) { - PyErr_Clear(); - goto _notimpl; - } - if (PyObject_GetBuffer(w, &ww, PyBUF_CONTIG_RO) == -1) { - PyErr_Clear(); - goto _notimpl; - } - - if (vv.itemsize != ww.itemsize || vv.len != ww.len) - goto _end; - - equal = !memcmp(vv.buf, ww.buf, vv.len); - -_end: - PyBuffer_Release(&vv); - PyBuffer_Release(&ww); - if ((equal && op == Py_EQ) || (!equal && op == Py_NE)) - res = Py_True; - else - res = Py_False; - Py_INCREF(res); - return res; - -_notimpl: - PyBuffer_Release(&vv); - PyBuffer_Release(&ww); - Py_RETURN_NOTIMPLEMENTED; -} - - -static int -memory_traverse(PyMemoryViewObject *self, visitproc visit, void *arg) -{ - if (self->view.obj != NULL) - Py_VISIT(self->view.obj); - return 0; -} - -static int -memory_clear(PyMemoryViewObject *self) -{ - PyBuffer_Release(&self->view); - return 0; -} - - -/* As mapping */ -static PyMappingMethods memory_as_mapping = { - (lenfunc)memory_length, /* mp_length */ - (binaryfunc)memory_subscript, /* mp_subscript */ - (objobjargproc)memory_ass_sub, /* mp_ass_subscript */ -}; - -static PySequenceMethods memory_as_sequence = { - 0, /* sq_length */ - 0, /* sq_concat */ - 0, /* sq_repeat */ - (ssizeargfunc)memory_item, /* sq_item */ -}; - -/* Buffer methods */ - -static PyBufferProcs memory_as_buffer = { - (getbufferproc)memory_getbuf, /* bf_getbuffer */ - (releasebufferproc)memory_releasebuf, /* bf_releasebuffer */ + {"release", (PyCFunction)memory_release, METH_NOARGS}, + {"tobytes", (PyCFunction)memory_tobytes, METH_NOARGS, NULL}, + {"tolist", (PyCFunction)memory_tolist, METH_NOARGS, NULL}, + {"cast", (PyCFunction)memory_cast, METH_VARARGS|METH_KEYWORDS, NULL}, + {"__enter__", memory_enter, METH_NOARGS}, + {"__exit__", memory_exit, METH_VARARGS}, + {NULL, NULL} }; PyTypeObject PyMemoryView_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) - "memoryview", - sizeof(PyMemoryViewObject), - 0, + "memoryview", /* tp_name */ + offsetof(PyMemoryViewObject, ob_array), /* tp_basicsize */ + sizeof(Py_ssize_t), /* tp_itemsize */ (destructor)memory_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ diff -r 92842e347d98 -r 51cedae5acfc Objects/object.c --- a/Objects/object.c Thu Sep 08 00:56:17 2011 +0200 +++ b/Objects/object.c Thu Sep 08 15:02:41 2011 +0200 @@ -1516,6 +1516,9 @@ if (PyType_Ready(&PyProperty_Type) < 0) Py_FatalError("Can't initialize property type"); + if (PyType_Ready(&PyManagedBuffer_Type) < 0) + Py_FatalError("Can't initialize managed buffer type"); + if (PyType_Ready(&PyMemoryView_Type) < 0) Py_FatalError("Can't initialize memoryview type"); diff -r 92842e347d98 -r 51cedae5acfc PCbuild/_testbuffer.vcproj --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/PCbuild/_testbuffer.vcproj Thu Sep 08 15:02:41 2011 +0200 @@ -0,0 +1,521 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff -r 92842e347d98 -r 51cedae5acfc PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln Thu Sep 08 00:56:17 2011 +0200 +++ b/PCbuild/pcbuild.sln Thu Sep 08 15:02:41 2011 +0200 @@ -137,6 +137,11 @@ EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "xxlimited", "xxlimited.vcproj", "{F749B822-B489-4CA5-A3AD-CE078F5F338A}" EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testbuffer", "_testbuffer.vcproj", "{A2697BD3-28C1-4AEC-9106-8B748639FD16}" + ProjectSection(ProjectDependencies) = postProject + {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} + EndProjectSection +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Win32 = Debug|Win32 @@ -588,6 +593,22 @@ {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|Win32.Build.0 = Release|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|x64.ActiveCfg = Release|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|x64.Build.0 = Release|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|Win32.ActiveCfg = Debug|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|Win32.Build.0 = Debug|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|x64.ActiveCfg = Debug|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Debug|x64.Build.0 = Debug|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|Win32.ActiveCfg = Release|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|Win32.Build.0 = Release|Win32 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|x64.ActiveCfg = Release|x64 + {A2697BD3-28C1-4AEC-9106-8B748639FD16}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff -r 92842e347d98 -r 51cedae5acfc PCbuild/readme.txt --- a/PCbuild/readme.txt Thu Sep 08 00:56:17 2011 +0200 +++ b/PCbuild/readme.txt Thu Sep 08 15:02:41 2011 +0200 @@ -92,6 +92,9 @@ _testcapi tests of the Python C API, run via Lib/test/test_capi.py, and implemented by module Modules/_testcapimodule.c +_testbuffer + buffer protocol tests, run via Lib/test/test_buffer.py, and + implemented by module Modules/_testbuffer.c pyexpat Python wrapper for accelerated XML parsing, which incorporates stable code from the Expat project: http://sourceforge.net/projects/expat/ diff -r 92842e347d98 -r 51cedae5acfc setup.py --- a/setup.py Thu Sep 08 00:56:17 2011 +0200 +++ b/setup.py Thu Sep 08 15:02:41 2011 +0200 @@ -524,6 +524,8 @@ # Python C API test module exts.append( Extension('_testcapi', ['_testcapimodule.c'], depends=['testcapi_long.h']) ) + # Python PEP-3118 (buffer protocol) test module + exts.append( Extension('_testbuffer', ['_testbuffer.c']) ) # profiler (_lsprof is for cProfile.py) exts.append( Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c']) ) # static Unicode character database