Memoryview objects provide a structured view over a memory area, meaning
the length, indexing and slicing operations respect the itemsize:

>>> import array
>>> a = array.array('i', [1,2,3])
>>> m = memoryview(a)
>>> len(a)
3
>>> m.itemsize
4
>>> m.format
'i'

However, in some cases, you want the memoryview to behave as a chunk of
pure bytes regardless of the original object *and without making a
copy*. Therefore, it would be handy to be able to change the format of
the memoryview, or ask for a new memoryview with another format.

An example of use could be:
>>> a = array.array('i', [1,2,3])
>>> m = memoryview(a).with_format('B')
>>> len(a), m.itemsize, m.format
(12, 1, 'B')

(Another way to see it is as supplying a Python equivalent to the C
buffer API, with access to the raw Py_buffer)

Agreed, this would be useful.  See http://codereview.appspot.com/12470/show if anyone doesn't believe us. 
;)

Is this issue from 2 years ago still open? I checked the docs and it seems to be.

If it is, I would like to work on a patch and submit it soon.

It is, but keep issue 10181 in mind (since that may lead to some restructuring of the memoryview code, potentially leading to a need to update your patch).

In the mean time I had to resort to dirty hacks in 1ac03e071d65 (such as using io.BytesIO.write(), which I know is implemented in C and doesn't care about item size).

At the minimum, a memoryview.getflatview() function would be nice (and probably easier to code than the generic version). Or a "flat" optional argument in the memoryview constructor.

Read a int32 array as a raw byte string is useful, but the opposite is also useful.

Unassigning.  Sorry;  no time for this at the moment.

I think this would be useful and I'll try it out in features/pep-3118#memoryview.

Syntax options that I'd prefer:

a = array.array('i', [1,2,3])
m = memoryview(a, 'B')


Or go all the way and make memoryview take any flag:

a = array.array('i', [1,2,3])
m = memoryview(a, getbuf=PyBUF_SIMPLE)


This is what I currently do in _testbuffer.c:


>>> from _testbuffer import *
>>> import array
>>> a = array.array('i', [1,2,3])
>>> nd = ndarray(a, getbuf=PyBUF_SIMPLE)
>>> nd.format
''
>>> nd.len
12
>>> nd.shape
()
>>> nd.strides
()
>>> nd.itemsize # XXX array_getbuf should set this to 1.
4



We would need to fix various getbuffer() methods to adhere to
strict rules that I've posed here:

http://mail.scipy.org/pipermail/numpy-discussion/2011-August/058189.html

> Or go all the way and make memoryview take any flag:
> 
> a = array.array('i', [1,2,3])
> m = memoryview(a, getbuf=PyBUF_SIMPLE)

This is good for testing, but Python developers shouldn't have to know
about the low-level flags.

Antoine Pitrou <report@bugs.python.org> wrote:
> > Or go all the way and make memoryview take any flag:
> > 
> > a = array.array('i', [1,2,3])
> > m = memoryview(a, getbuf=PyBUF_SIMPLE)
> 
> This is good for testing, but Python developers shouldn't have to know
> about the low-level flags.

Hmm, indeed. How about:

1) memoryview(a, format='B')

Shadows a builtin function; annoying syntax highlighting in current Vim.

2) memoryview(a, fmt='B')

I'm fully expecting a comment about 'strpbrk' again, but I like it. :)

Also, we've to see about speed implications. My current version of memoryview
(not pushed yet to the public repo) also solves #10227, but is pretty sensitive
even to small changes.

> Hmm, indeed. How about:
> 
> 1) memoryview(a, format='B')
> 
> Shadows a builtin function; annoying syntax highlighting in current Vim.
> 
> 2) memoryview(a, fmt='B')
> 
> I'm fully expecting a comment about 'strpbrk' again, but I like it. :)

I really prefer "format", it's the natural word to use there.
I don't think this the only place where we shadow a builtin function.
There are probably variables named "dict" in many places.

> Also, we've to see about speed implications. My current version of memoryview
> (not pushed yet to the public repo) also solves #10227, but is pretty sensitive
> even to small changes.

Well, solving #10227 would be nice, but I don't think it's critical
either.

Good, I'll use 'format'. I was mainly worried about the shadowing
issue.

Rethinking a bit: Casting to arbitrary formats might go a bit far.

Currently, the combination (format=NULL, shape=NULL) can serve as
a warning "This buffer has been cast to unsigned bytes".

If we allow casts from bytes to int32, we'll have (format="i", shape=x)
and consumers of that buffer have no indication that the original
exporter had a different format.

If you know what you are doing, fine. On the other hand following
the buffer paths in #12817 quickly turned into a very complex
maze of getbuffer requests.


So, an option would be to try out the cast to bytes first and
disallow other casts.

Casting to a flat 1-D array of bytes is reasonable (it's essentially saying 'look, just give me the raw data, it's on my own head if I stuff up the formatting').

However, requiring an explicit two-step process for any other casting (i.e. take a 1-D view, then a shaped view of that flat 1-D view) also sounds reasonable.

So I agree with Victor that 1-D bytes -> any shape/format and any shape/format -> 1-D bytes should be allowed, but I think we should hold off on allowing arbitrary transformations in a single step.

> However, requiring an explicit two-step process for any other casting
> (i.e. take a 1-D view, then a shaped view of that flat 1-D view) also
> sounds reasonable.
> 
> So I agree with Victor that 1-D bytes -> any shape/format and any
> shape/format -> 1-D bytes should be allowed, but I think we should
> hold off on allowing arbitrary transformations in a single step.

Converting to 1-D bytes is my main motivation for this feature request,
so I'm fine with such a limitation.

The point is to be able to do in Python what we can do in C, take an
arbitrary buffer and handle it as pure bytes (for I/O or cryptography
purposes, for example).

Nick Coghlan <report@bugs.python.org> wrote:
> So I agree with Victor that 1-D bytes -> any shape/format and any
> shape/format -> 1-D bytes should be allowed, but I think we should
> hold off on allowing arbitrary transformations in a single step.

1-D bytes -> any shape/format would work if everyone agrees on the
Numpy mailing list post that I linked to in an earlier message.

[Summary: PyBUF_SIMPLE may downcast any C-contiguous array to unsigned bytes.]

Otherwise a PyBUF_SIMPLE getbuffer call to the newly shaped memoryview
would be required to fail, and these calls are almost certain to occur
somewhere, e.g. in PyObject_AsWriteBuffer().

But then memoryview would also need a 'shape' parameter:

m = memoryview(x, format='L', shape=[3, 4])

In that case, making it a method might indeed be more clear to underline
that something extraordinary is going on:

m = memoryview(x).cast(format='L', shape=[3, 4])

It also takes away a potential speed loss for regular uses.

1-D bytes would then be defined as 'b', 'B' and 'c', I presume? Being able
to cast to 'c' would also solve certain memoryview index assignment problems
that arise if we opt for strict typing as the struct module does.

The cast method is completely implemented over at #10181.

Shouldn't this be closed in favour of #10181?

Yes, it's really superseded by #10181 now. I'm closing as 'duplicate',
since technically it'll be fixed once the patch for #10181 is committed.

New changeset 3f9b3b6f7ff0 by Stefan Krah in branch 'default':
- Issue #10181: New memoryview implementation fixes multiple ownership
http://hg.python.org/cpython/rev/3f9b3b6f7ff0