mmap, buffer, bytearray, string and unicode objects set the char buffer callback (bf_getcharbuffer). The bytearray object sets also the release buffer callback (bf_releasebuffer).

In Python 2.7, PyObject_AsCharBuffer() accepts bytearray objects, whereas the "t#" format of PyArg_Parse functions rejects byte bytearray objects (expect an "pinned buffer object").

In Python 3.2, PyObject_AsCharBuffer() releases the buffer.

PyObject_AsCharBuffer() documentation (in 2.7 and 3.2) says that the function only accepts read-only objects.

Something is wrong here. If the caller doesn't hold a kind of lock, the object cannot be protected against futher modifications. The caller has to ensure that the object is not modifiable until it finishs to use the char* pointer.

I think that it would be safer to respect the documentation: PyObject_AsCharBuffer() should only accept read-only objects. The most important change is that functions using PyObject_AsCharBuffer() will not accept bytearray objects anymore.

Attached patch (for Python 2.7) changes PyObject_AsCharBuffer() to reject modifiable objects. It removes also the character buffer callback from the bytearray type. To avoid breaking compatibility too much, I patched int(), long() and float() to still support bytearray objects.

Examples of functions rejecting bytearray with the patch:
 - int(), long(), float(), complex()
 - many str methods: split, partition, rpartition, rsplit, index, find, count, translate, replace, startswith, endswith
 - writelines() of file objects (eg. sys.stdout.writelines)
 - writelines() method of a bz2 file

--

My patch breaks backward compatibility, and I don't know that it is acceptable in Python 2.7.

I will write a similar patch for Python 3.2.

I don't think we should change anything in 2.7 at this point. It risks breaking compatibility while we are at the end of the 2.x line, for little added benefit (the old buffer API has always been insecure with mutable buffers).

As for 3.2, PyObject_AsCharBuffer() should IMO be deprecated, as well as any other functions meant to query the old buffer API.

STINNER Victor wrote:
> 
> New submission from STINNER Victor <victor.stinner@haypocalc.com>:
> 
> mmap, buffer, bytearray, string and unicode objects set the char buffer callback (bf_getcharbuffer). The bytearray object sets also the release buffer callback (bf_releasebuffer).
> 
> In Python 2.7, PyObject_AsCharBuffer() accepts bytearray objects, whereas the "t#" format of PyArg_Parse functions rejects byte bytearray objects (expect an "pinned buffer object").
> 
> In Python 3.2, PyObject_AsCharBuffer() releases the buffer.
> 
> PyObject_AsCharBuffer() documentation (in 2.7 and 3.2) says that the function only accepts read-only objects.
> 
> Something is wrong here. If the caller doesn't hold a kind of lock, the object cannot be protected against futher modifications. The caller has to ensure that the object is not modifiable until it finishs to use the char* pointer.
> 
> I think that it would be safer to respect the documentation: PyObject_AsCharBuffer() should only accept read-only objects. The most important change is that functions using PyObject_AsCharBuffer() will not accept bytearray objects anymore.
> 
> Attached patch (for Python 2.7) changes PyObject_AsCharBuffer() to reject modifiable objects. It removes also the character buffer callback from the bytearray type. To avoid breaking compatibility too much, I patched int(), long() and float() to still support bytearray objects.
> 
> Examples of functions rejecting bytearray with the patch:
>  - int(), long(), float(), complex()
>  - many str methods: split, partition, rpartition, rsplit, index, find, count, translate, replace, startswith, endswith
>  - writelines() of file objects (eg. sys.stdout.writelines)
>  - writelines() method of a bz2 file
> 
> --
> 
> My patch breaks backward compatibility, and I don't know that it is acceptable in Python 2.7.

Simple answer: no.

Long answer: The caller is responsible for making sure that the object
is not modified while in use.

> I will write a similar patch for Python 3.2.

Restricting the API to read-only buffers would seriously limit
it's functionality. I'm -1 on doing that.

Instead, it's better to clarify the documentation and mention the
fact that the used object may not change during use.

Note that the buffer interface release API is meant to protect
against such modifications, so I don't see why rejecting objects
that do implement this API should be rejected. It's object that
don't implement the release buffer slot which you'd have to worry
about. Then again, this has never really been an issue in practice
during the 10 years of the 2.x branch, so I wouldn't call this
a serious issue.

See PEP 3118... "All that is specifically required by the exporter, however, is to ensure that any
memory shared through the bufferinfo structure remains valid until releasebuffer is called on the
bufferinfo structure exporting that memory."

> Note that the buffer interface release API is meant to protect
> against such modifications, so I don't see why rejecting objects
> that do implement this API should be rejected.

As I explained, the release API is *not* used by PyObject_AsCharBuffer() in Python 2.7 and 3.2.

Pseudo-code example:
---
PyObject_AsCharBuffer(obj, &str, &size)
... modify or destroy obj ...
str is no more valid here
---

> Restricting the API to read-only buffers would seriously limit
> it's functionality. I'm -1 on doing that.

PyObject_AsCharBuffer() is dangerous because the caller has to ensure that the object is not modified or destroyed. Antoine proposes to deprecated PyObject_AsCharBuffer().

PyObject_GetBuffer() can replace PyObject_AsCharBuffer(): it's easy to get the pointer to the buffer content (view.buf) and the size (view.len) using PyObject_GetBuffer(), and it does protect the buffer against modification or destuction thanks to the release API (PyBuffer_Release). But PyObject_GetBuffer() is maybe a little bit to low level, eg. it doesn't check that the buffer is contiguous, and it requires a flag argument. A new function is maybe needed to replace PyObject_AsCharBuffer(). Eg. PyObject_GetCharBuffer() which will call PyObject_GetBuffer() (the caller will then have to call PyBuffer_Release() to release the buffer).

Example:
---
PyObject_GetCharBuffer(obj, &view, &str, &size)
... use str and size ...
PyBuffer_Release(view);
---
or just
---
PyObject_GetCharBuffer(obj, &view)
... use view.buf and view.len ...
PyBuffer_Release(view);
---

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> Note that the buffer interface release API is meant to protect
>> against such modifications, so I don't see why rejecting objects
>> that do implement this API should be rejected.
> 
> As I explained, the release API is *not* used by PyObject_AsCharBuffer() in Python 2.7 and 3.2.
> 
> Pseudo-code example:
> ---
> PyObject_AsCharBuffer(obj, &str, &size)
> ... modify or destroy obj ...
> str is no more valid here
> ---

Right, but as I explained before: this doesn't really happen in
practice, otherwise we would have had issues with these APIs
long before the Py_buffers were introduced.

Note that the same comment applies to PyObject_AsReadBuffer() and
PyObject_AsWriteBuffer().

For Python 2.7 you can't change anything anymore. For Python 3.2
you could start a deprecation process as outlined in PEP 4, if you
feel this really is such a big issue.

>> Restricting the API to read-only buffers would seriously limit
>> it's functionality. I'm -1 on doing that.
> 
> PyObject_AsCharBuffer() is dangerous because the caller has to ensure that the object is not modified or destroyed. Antoine proposes to deprecated PyObject_AsCharBuffer().
> 
> PyObject_GetBuffer() can replace PyObject_AsCharBuffer(): it's easy to get the pointer to the buffer content (view.buf) and the size (view.len) using PyObject_GetBuffer(), and it does protect the buffer against modification or destuction thanks to the release API (PyBuffer_Release). But PyObject_GetBuffer() is maybe a little bit to low level, eg. it doesn't check that the buffer is contiguous, and it requires a flag argument. A new function is maybe needed to replace PyObject_AsCharBuffer(). Eg. PyObject_GetCharBuffer() which will call PyObject_GetBuffer() (the caller will then have to call PyBuffer_Release() to release the buffer).
> 
> Example:
> ---
> PyObject_GetCharBuffer(obj, &view, &str, &size)
> ... use str and size ...
> PyBuffer_Release(view);
> ---
> or just
> ---
> PyObject_GetCharBuffer(obj, &view)
> ... use view.buf and view.len ...
> PyBuffer_Release(view);
> ---

Why add a new function ? Python 3.2 doesn't provide a way to access
a *character* buffer version of an object anymore. Not that this is
good, or shouldn't be readded at some point, but right now, we don't
have a need for such an API.

PEP 3118

"""
Rationale

...

3) There is no way for a consumer to tell the buffer-API-exporting object it is "finished" with its view of the memory and therefore no way for the exporting object to be sure that it is safe to reallocate the pointer to the memory that it owns (for example, the array object reallocating its memory after sharing it with the buffer object which held the original pointer led to the infamous buffer-object problem).
"""


Let's consider Pygame, and the SDL surface it wraps as a pygame.Surface. Pygame exposes a surface's data through the buffer protocol for manipulation by a NumPy array. Now some SDL surfaces need locking before accessing, so Pygame will lock the surface for the lifetime of the consumer array. How is the surface unlocked when the array is reclaimed? By a buffer proxy that is owned by the array. The proxy unlocks the surface when the proxy is garbage collected. The new buffer protocol suggests a simpler way, equivalent to contexts, but only if the buffer release mechanism works as advertised. This promise is broken when a consumer calls PyBuffer_Release before it is done with the buffer.

All the questionable buffer api functions use the PyBUF_SIMPLE access flag. So I suggest a PyBUF_SIMPLE access request requires a returned buffer to remain valid for the lifetime of the exporter, not the Py_buffer view. If an exporter cannot honor this request, due to locking requirements for instance, then it raises an exception. All other access request flags promise that PyBuffer_Release, and therefore bf_releasebuffer, is called only when the buffer is released, not before.

Le mardi 28 septembre 2010 à 03:11 +0000, Lenard Lindstrom a écrit :
> 
> Let's consider Pygame, and the SDL surface it wraps as a
> pygame.Surface. Pygame exposes a surface's data through the buffer
> protocol for manipulation by a NumPy array. Now some SDL surfaces need
> locking before accessing, so Pygame will lock the surface for the
> lifetime of the consumer array. How is the surface unlocked when the
> array is reclaimed? By a buffer proxy that is owned by the array. The
> proxy unlocks the surface when the proxy is garbage collected. The new
> buffer protocol suggests a simpler way, equivalent to contexts, but
> only if the buffer release mechanism works as advertised. This promise
> is broken when a consumer calls PyBuffer_Release before it is done
> with the buffer.

I don't know why you're saying that. The purpose of PyBuffer_Release is
precisely to solve these kinds of use cases (where you want timely
release of a resource rather than rely on the garbage collector).

(you shouldn't even need a proxy anymore; just make the surface object
expose the buffer API, with bf_getbuffer() acquiring the lock and
bf_releasebuffer() releasing the lock. If you want to support several
buffer exports at once, just maintain some internal count of them)

>I don't know why you're saying that. The purpose of PyBuffer_Release is
>precisely to solve these kinds of use cases (where you want timely
>release of a resource rather than rely on the garbage collector).

Yes, I was unclear. This refers to Python 3.2, not the 2.x series. PyObject_AsReadBuffer (defined at line 270 in abstract.c, code of routine attached) calls bf_getbuffer with the PyBUF_SIMPLE flag to retrieve a bytes buffer. It then calls bf_releasebuffer before returning the buffer to the caller. PyObject_AsCharBuffer and PyObject_AsWriteBuffer do the same. It is not be exactly the same issue discussed so far, but is closely related.

Deprecating PyObject_AsReadBuffer is extreme, and doesn't solve the early release problem. Redefining the PyBUF_SIMPLE flag to be like the old buffer protocol warns those implementing a new buffer interface to  avoid processing PyBUF_SIMPLE requests when locking is required.

> Yes, I was unclear. This refers to Python 3.2, not the 2.x series.
> PyObject_AsReadBuffer (defined at line 270 in abstract.c, code of
> routine attached) calls bf_getbuffer with the PyBUF_SIMPLE flag to
> retrieve a bytes buffer. It then calls bf_releasebuffer before
> returning the buffer to the caller. PyObject_AsCharBuffer and
> PyObject_AsWriteBuffer do the same. It is not be exactly the same
> issue discussed so far, but is closely related.
> 
> Deprecating PyObject_AsReadBuffer is extreme,

I disagree. PyObject_As*Buffer functions are remnants of the old buffer
API in Python 2.x. They are here only to ease porting of existing C
code, but carefully written 3.x code should switch to
PyObject_GetBuffer() (or one of the dedicated typecodes in
PyArg_ParseTuple: "y*", "w*", etc.).

> Redefining the PyBUF_SIMPLE flag to be like the old buffer protocol
> warns those implementing a new buffer interface to  avoid processing
> PyBUF_SIMPLE requests when locking is required.

Well, the new buffer API was designed precisely because the old API
wasn't appropriate, so your proposal to revive the old API doesn't sound
very compelling, to say the least.

To restate things a bit more clearly: you should use
PyObject_GetBuffer(), not any of the PyObject_As*Buffer functions.
Perhaps you should explain why you care about the latter rather than the
former.

Le mardi 28 septembre 2010 22:24:56, vous avez écrit :
> I disagree. PyObject_As*Buffer functions are remnants of the old buffer
> API in Python 2.x. They are here only to ease porting of existing C
> code, but carefully written 3.x code should switch to
> PyObject_GetBuffer() (or one of the dedicated typecodes in
> PyArg_ParseTuple: "y*", "w*", etc.).

Do you mean that PyObject_As*Buffer functions are deprecated? The documentation 
doesn't tell that the new buffer API should be use instead.

> Le mardi 28 septembre 2010 22:24:56, vous avez écrit :
> > I disagree. PyObject_As*Buffer functions are remnants of the old buffer
> > API in Python 2.x. They are here only to ease porting of existing C
> > code, but carefully written 3.x code should switch to
> > PyObject_GetBuffer() (or one of the dedicated typecodes in
> > PyArg_ParseTuple: "y*", "w*", etc.).
> 
> Do you mean that PyObject_As*Buffer functions are deprecated? The documentation 
> doesn't tell that the new buffer API should be use instead.

Well, the documentation should be updated then. Given that the old
buffer API doesn't exist anymore, the old functions don't have any
practical purpose except for ease of porting code.

Ok, 3.x documentation is fixed to recommend the new buffer API funcs over the old compatibility funcs. On Victor's private suggestion, I'm now closing the issue.

That is perfectly fine with me.