There's probably a bigger discussion about memoryviews for a rainy day.  However, the number one thing that would save all of this in my book would be to make sure cast('B') is universally supported regardless of format including endianness--especially in the standard library. For example, being able to do this:

>>> a = array.array('d',[1.0, 2.0, 3.0, 4.0])
>>> m = memoryview(a).cast('B')
>>> m[0:4] = b'\x00\x01\x02\x03'
>>> a
array('d', [1.0000000112050316, 2.0, 3.0, 4.0])
>>> 

Right now, it doesn't work for ctypes.  For example:

>>> import ctypes
>>> a = (ctypes.c_double * 4)(1,2,3,4)
>>> a
<__main__.c_double_Array_4 object at 0x1006a7cb0>
>>> m = memoryview(a).cast('B')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> 

As some background, being able to work with a "byte" view of memory is important for a lot of problems involving I/O, data interchange, and related problems where being able to accurately construct/deconstruct the underlying memory buffers is more useful than actually interpreting their contents.