I'm unsure of the expected behavior in this case but it seems odd. The
bytearray in the following example can be resized to a length of 5-10
bytes without throwing an exception.


Python 3.0rc3 (r30rc3:67312, Dec  3 2008, 10:38:14) 

[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2

>>> b = bytearray(b'x' * 10)

>>> v = memoryview(b)

>>> b[:] = b'y' * 11

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

BufferError: Existing exports of data: object cannot be re-sized

>>> b[:] = b'y' * 5

>>> b

bytearray(b'yyyyy')

>>> v.tobytes()

b'yyyyy\x00xxxx'

>>> v2 = memoryview(b)

>>> v2.tobytes()

b'yyyyy'

It's not a memoryview bug, but a bytearray oddity. The bytearray uses a
variable-sized buffer underneath, and it tries to minimize the number of
reallocations when changing the object length through some simple
heuristics. Therefore, a bytearray has both a logical size (the one
which is seen from Python, e.g. len()) and a physical size (which can be
greater than the logical size, due to those heuristics).

The bug here is that the bytearray only prohibits changing the physical
size, not the logical size when there is a buffer pointing to it. This
also explains the "\00" you are witnessing at the 5th byte when calling
tobytes() on the memoryview object: it is the end-of-string marker which
has been inserted when changing the logical size.

Please see patch at http://codereview.appspot.com/10049

I found another related bug. In bytes_setslice, when the buffer is
resized to a smaller size, a memmove happens regardless of whether the
resize is successful or not.

>>> b = bytearray(range(10))
>>> m = memoryview(b)
>>> b[1:8] = b'X'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BufferError: Existing exports of data: object cannot be re-sized
>>> b
bytearray(b'\x00\x01\x08\t\x04\x05\x06\x07\x08\t')

The same problem also applies to bytes_remove:

>>> b
bytearray(b'\x02\x03\x04\x05\x06\x07\x08\t')
>>> b.remove(2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
BufferError: Existing exports of data: object cannot be re-sized
>>> b
bytearray(b'\x03\x04\x05\x06\x07\x08\t\x00')

There may be other places this can happen but I haven't checked yet.

Sorry, forgot to give this issue a more accurate title earlier.

I've found that arrays from the array module have similar issues:

>>> a = array.array('i', range(2))
>>> m = memoryview(a)
>>> bytes(m)
b'\x00\x00\x00\x00\x01\x00\x00\x00'
>>> a.pop(0)
0
>>> bytes(m)
b'\x01\x00\x00\x00\x01\x00\x00\x00'

> There may be other places this can happen but I haven't checked yet.

PyByteArray_Resize() is called in various places in bytearrayobject.c,
some of them where it is mandatory to mutate the underlying storage
before reallocating it. The solution would be to have a separate
function to check whether resizing is allowed (but it would not solve
the problem in the face of MemoryErrors).

New bytearray patch at http://codereview.appspot.com/10049. I think I've
covered all bases. array.array will need another patch (I must admit I
care a bit less about it, since it's not a builtin type).

The patch will have to be backported for 2.6/2.7 as well, but memoryview
doesn't exist there, so the tests will have to be disabled.

It turns out the problems in array are more serious than I thought and
allow writing to unallocated memory through a memoryview leading to
memory corruption, segfaults and possibly exploits. The following
example extends an array enough to trigger a realloc of the array's buffer.

Python 3.0 (r30:67503, Dec  4 2008, 13:30:57) 
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from array import array
>>> a = array('i', range(16))
>>> m = memoryview(a)
>>> a.extend(array('i', range(48))
... )
>>> m[:] = array('i', [0] * (len(m) // m.itemsize))
*** glibc detected *** python3.0: corrupted double-linked list:
0x0822c1f8 ***

The segfault happens even when the array is not being resized, I've
opened a separate bug for it: #4509.

The bytearray patch has been committed to 2.6, 2.7, 3.0, 3.1. Now the
array.array problem remains.

Sorry, typo: the segfault issue is #4569.

Found another bug with memoryview and arrays in #4580.

I've opened a new memoryview/array segfault issue since #4569 was
closed: #4583

Fixed the array bug in r73850. Is there any bug left to fixed that were
reported in this issue?

Closing as I don't see any other bugs in this issue to fix.