classification
Title: memoryview complain ctypes byte array are not native single character
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.3
process
Status: closed Resolution: duplicate
Dependencies: Superseder: implement PEP 3118 struct changes
View: 3132
Assigned To: Nosy List: hct, skrah
Priority: normal Keywords:

Created on 2013-11-26 20:54 by hct, last changed 2013-12-02 18:33 by skrah. This issue is now closed.

Messages (7)
msg204536 - (view) Author: HCT (hct) Date: 2013-11-26 20:54
I'm trying to create ctypes buffer to be used back and forth with C libraries, but I keep getting errors. I need to slice the buffer to cut out different pieces to work on, so I try to get a memoryview of the buffer. does the error message imply that c_ubyte, c_uint8, c_byte and c_char are not native single character types?

>>> memoryview(c_ubyte()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_ubyte*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview(c_uint8()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_uint8*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_byte*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_char*1024)()).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> memoryview((c_char*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<c
>>> memoryview((c_byte*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<b
>>> memoryview((c_ubyte*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<B
>>> memoryview((c_uint8*1024)())[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format (1024)<B
>>>
msg204572 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-11-27 10:53
Memoryview currently only knows the types from the struct module.
msg204613 - (view) Author: HCT (hct) Date: 2013-11-27 21:09
this seems to disagree with the statement of "Memoryview currently only knows the types from the struct module."

why is memoryview aware of _pack_, a implementation detail of ctypes structures. is memoryview a collection of implementation for different types or a unique type on its own?


>>> import ctypes
>>> class B1(ctypes.Structure):
...     _fields_ = [( "data", c_uint8 * 256 ), ]
...     _pack_ = 1
...
>>> class B2(ctypes.Structure):
...     _fields_ = [( "data", c_uint8 * 256 ), ]
...
>>>
>>> a = B1()
>>> b = B2()
>>> memoryview( a ).cast( 'B' )
<memory at 0x01FFCDC0>
>>> memoryview( b ).cast( 'B' )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>>
msg204614 - (view) Author: HCT (hct) Date: 2013-11-27 21:12
more examples (using 64-bit integer vs 8-bit integer in the above example) to show that ctypes aren't being translated for memoryview properly. _pack_ is the only way to make memoryview handle ctypes properly


>>> import ctypes
>>> class B1(ctypes.Structure):
...     _fields_ = [( "data", ctypes.c_uint64 * 256 ), ]
...     _pack_ = 1
...
>>> class B2(ctypes.Structure):
...     _fields_ = [( "data", ctypes.c_uint64 * 256 ), ]
...
>>>
>>> a = B1()
>>> b = B2()
>>> memoryview( a ).cast( 'B' )
<memory at 0x01FFB030>
>>> memoryview( b ).cast( 'B' )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>>
msg204719 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-11-29 11:07
>>> class B1(ctypes.Structure):
...     _fields_ = [("data", ctypes.c_uint8 * 256), ]
...     _pack_ = 1 
... 
>>> a= B1()
>>> x = memoryview(a)
>>> x.format
'B'

In the first case the format is 'B', in the second case the
format is:

>>> x = memoryview(b)
>>> x.format
'T{(256)<B:data:}'

While the latter is probably a valid PEP 3118 format, it's
not implemented anywhere outside ctypes See #3132.
msg205036 - (view) Author: HCT (hct) Date: 2013-12-02 18:13
I wonder why "_pack_ = 1" has significant impact on the behaviour of memoryview if memoryview is not a sub class of ctypes structure. unless memoryview was specifically coded to understand ctypes structure, I don't see why "_pack_ = 1" should make any difference.
msg205037 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-12-02 18:33
Memoryview sends a *request* to ctypes to fill in a Py_buffer struct according
to the buffer *protocol*.  IOW, memoryview knows nothing about ctypes.

For some reason ctypes fills in 'B' as the format if _pack_ is set.  I do not
know if that is intended, it could be a ctypes issue.
History
Date User Action Args
2013-12-02 18:33:53skrahsetmessages: + msg205037
2013-12-02 18:13:37hctsetmessages: + msg205036
2013-11-29 11:07:39skrahsetstatus: open -> closed

superseder: implement PEP 3118 struct changes
components: + Interpreter Core, - Library (Lib), ctypes
messages: + msg204719
type: behavior
resolution: duplicate
stage: resolved
2013-11-27 21:12:36hctsetmessages: + msg204614
2013-11-27 21:09:45hctsetmessages: + msg204613
2013-11-27 10:53:51skrahsetnosy: + skrah
messages: + msg204572
2013-11-26 20:54:09hctcreate