classification
Title: memoryview & ctypes: incorrect itemsize for empty array
Type: behavior Stage: patch review
Components: ctypes, Interpreter Core Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Eric Wieser, Eric.Wieser, amaury.forgeotdarc, belopolsky, meador.inge, skrah, teoliphant, terry.reedy
Priority: normal Keywords: patch

Created on 2018-02-06 18:53 by Eric.Wieser, last changed 2019-07-16 05:53 by Eric Wieser.

Pull Requests
URL Status Linked Edit
PR 5576 open Eric.Wieser, 2018-02-07 07:19
Messages (9)
msg311740 - (view) Author: Eric Wieser (Eric.Wieser) * Date: 2018-02-06 18:53
Take the following simple structure:

    class Foo(ctypes.Structure):
        _fields_ = [('f', ctypes.uint32_t)]

And construct some arrays with it:

    def get_array_view(N):
        return memoryview((Foo * N)())

In most cases, this works as expected, returning the size of one item:

    >>> get_array_view(10).itemsize
    4
    >>> get_array_view(1).itemsize
    4

But when N=0, it returns the wrong result

    >>> get_array_view(0).itemsize
    0

Which contradicts its `.format`, which still describes a 4-byte struct

    >>> get_array_view(0).format
    'T{>I:one:}'

This causes a downstream problem in numpy:

    >>> np.array(get_array_view(0))
    RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
msg321290 - (view) Author: Eric Wieser (Eric.Wieser) * Date: 2018-07-08 21:48
Pinging, as recommended by https://devguide.python.org/pullrequest/#reviewing. Ideally this and https://bugs.python.org/issue32780 would make the same patch release.
msg324705 - (view) Author: Eric Wieser (Eric.Wieser) * Date: 2018-09-06 17:03
Pinging again, for lack of a clearer path forward
msg340230 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-04-14 20:04
This issue is about the itemsize attribute of instances of the built-in memoryview class.  Ctypes in only involved in providing format information.  Hence the nosy additions.

On Win10 with 3.8, ctypes has no uint attributes.  Using 'c_int32' instead, I see the same behavior (itemsize 0 for empty structure).

About itemsize, https://www.python.org/dev/peps/pep-3118/ says

"This is a storage for the itemsize (in bytes) of each element of the shared memory. It is technically un-necessary as it can be obtained using PyBuffer_SizeFromFormat, however an exporter may know this information without parsing the format string and it is necessary to know the itemsize for proper interpretation of striding. Therefore, storing it is more convenient and faster."

The first line could be seen as implying that itemsize is undefined if there are no items (and as justifying numbytes/numitems otherwise).  The 0 return could be seen as equivalent to a None return from a python-coded function.  If so, it is not a bug, and there might be code that would break if it is changed.

On the other hand, the next lines imply that itemsize is *usually*, though not necessarily, a cache for PyBuffer_SizeFromFormat.  This could be seen as implying that in the absence of other information, the itemsize should be calculated from the format, making 0 a bug.
msg340231 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-04-14 20:14
This is actually about memoryview.itemsize within ctypes.
msg340232 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-04-14 20:25
https://docs.python.org/3/library/stdtypes.html#typememoryview
says "itemsize The size in bytes of each element of the memoryview"
Revising the code example to use an empty array:
>>> import array, struct
>>> m = memoryview(array.array('H', [0])
>>> m.itemsize
2
I agree that itemsize should also be non-zero for ctype formats.
msg340235 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2019-04-14 21:12
I agree that it is a ctypes issue, itemsize should be equal to struct.calcsize(fmt), which is never 0 for normal PEP-3118 types like the one in the example.



[Pedantically, I think that the grammar would allow for an empty record "T{}" that would have itemsize 0 but is of little use inside numpy.]
msg340236 - (view) Author: Eric Wieser (Eric Wieser) Date: 2019-04-14 21:29
> Revising the code example to use an empty array

I think you mean

>>> import array
>>> memoryview(array.array('H', [])).itemsize
2

Your example is an array containing 0, not an empty array - but the conclusion is the same.

> It is technically un-necessary as it can be obtained using PyBuffer_SizeFromFormat

This obviously predicates on `PyBuffer_SizeFromFormat` being implemented, which according to the docs it is not.

> I think that the grammar would allow for an empty record "T{}" that would have itemsize 0 but is of little use inside numpy.

It also allows for records of empty arrays, "T{(0)d:data:}". While these are of little use, they _are_ supported by both ctypes and numpy, so we should support them in the PEP3118 interface used between them.
msg348004 - (view) Author: Eric Wieser (Eric Wieser) Date: 2019-07-16 05:53
Pinging again, now that the patch has undergone a revision with some cleanup thanks to @skrah
History
Date User Action Args
2019-07-16 05:53:15Eric Wiesersetmessages: + msg348004
2019-04-14 21:29:15Eric Wiesersetnosy: + Eric Wieser
messages: + msg340236
2019-04-14 21:12:56skrahsetmessages: + msg340235
2019-04-14 20:25:23terry.reedysetmessages: + msg340232
title: memoryview & ctypes: incorrect PEP3118 itemsize for empty array -> memoryview & ctypes: incorrect itemsize for empty array
2019-04-14 20:14:41terry.reedysetmessages: + msg340231
components: + ctypes
title: memoryview gives incorrect PEP3118 itemsize for empty array -> memoryview & ctypes: incorrect PEP3118 itemsize for empty array
2019-04-14 20:05:19terry.reedysettitle: ctypes: memoryview gives incorrect PEP3118 itemsize for array of length zero -> memoryview gives incorrect PEP3118 itemsize for empty array
2019-04-14 20:04:27terry.reedysetversions: + Python 3.8, - Python 3.6
nosy: + skrah, terry.reedy, teoliphant

messages: + msg340230

components: + Interpreter Core, - ctypes
2018-09-06 17:03:02Eric.Wiesersetmessages: + msg324705
2018-07-08 21:48:58Eric.Wiesersetmessages: + msg321290
2018-02-10 03:40:05terry.reedysetnosy: + amaury.forgeotdarc, belopolsky, meador.inge
2018-02-07 07:19:38Eric.Wiesersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5394
2018-02-06 18:53:21Eric.Wiesercreate