This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ctypes from_buffer no longer accepts bytes
Type: behavior Stage: needs patch
Components: ctypes Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: theller Nosy List: benrg, eryksun, martin.panter, meador.inge, santoso.wijaya, theller, vstinner
Priority: normal Keywords: 3.2regression

Created on 2011-03-07 07:11 by benrg, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (10)
msg130229 - (view) Author: (benrg) Date: 2011-03-07 07:11
In Python 3.1.3, (c_char*5).from_buffer(b'abcde') worked. In 3.2 it fails with "TypeError: expected an object with a writable buffer interface".

This seems to represent a significant decrease in the functionality of ctypes, since, if I understand correctly, it has no notion of a const array or a const char. I used from_buffer with a bytes argument in 3.1 and it was far from obvious how to port to 3.2 without introducing expensive copying. I understand the motivation behind requiring a writable buffer, but I think it's a bad idea. If you take this to its logical conclusion, it should not be possible to pass bytes or str values directly to C functions, since there's no way to be sure they won't write through the pointer.
msg147331 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-11-09 03:33
I agree this is a regression.  I am looking more into it now.
msg147364 - (view) Author: Santoso Wijaya (santoso.wijaya) * Date: 2011-11-09 19:07
From what I understand, even in 2.7 the exception is expected. The doc for `from_buffer` says:

> The source object must support the writeable buffer interface.

Is there any reason why `from_buffer_copy` cannot be used, instead? The latter only requires readable buffer interface.
msg147392 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-11-10 03:10
Hmmm... I take back what I said before (not sure what I was thinking), I 
don't think this is a regression.  If this ever did work, it must have 
been a bug.  For one reason, Python byte strings are now immutable.  
Allowing their memory to be indirectly mutated in Python via buffers 
created from 'from_buffer' would be very bad.

Also, I can't reproduce the reported behavior with 3.1.3:

Python 3.1.3 (unknown, Nov  9 2011, 19:57:58) 
[GCC 4.6.1 20110908 (Red Hat 4.6.1-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> (ctypes.c_char*5).from_buffer(b'abcde')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: expected an object with a writable buffer interface

Are you sure it was 3.1.3?  At first I thought you might be using
3.0a, since bytes are mutable in that version, but 'from_buffer'
didn't exist in those times.
msg148535 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-11-29 02:38
benrg, any further interest in this?  If so, please comment on my last reply.  Otherwise, I am closing this issue.
msg148561 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-11-29 13:19
.from_buffer() expects a modifiable buffer, bytes type is immutable. If .from_buffer() accepted an immutable buffer, it was a bug and the bug was fixed.

If you want to use bytes, use .from_buffer_copy() instead of .from_buffer().
msg148586 - (view) Author: (benrg) Date: 2011-11-29 16:55
I am still interested in this for the same reason I was interested in this in the first place; nothing has changed. I guess I will reiterate, and try to expand.

The problem is that ctypes tries to enforce const correctness (inconsistently), but it has no way to declare its objects as const, and assumes they are all non-const. This is the worst possible combination. It would make sense to have no notion of const and never try to enforce it (like K&R C), and it would make sense to have constness and try to enforce it (like C++). But the present situation is like a compiler that treats string literals as const char[], and doesn't allow casting away const, but doesn't allow the use of the const keyword in user code, so there's no way to pass a string literal as a function argument because the necessary type can't be expressed. Instead you have to copy the literal to a char array and pass that.

Calling C functions is inherently unsafe. Calling C functions could always corrupt the Python heap or otherwise screw around with state that the Python environment thinks that it owns. All I want is some means to assert that my code is not going to do that, as a way of getting around the limited type system that can't provide those sorts of guarantees. More broadly, what I want from ctypes is a way to do the sorts of things that I can do in C. If I can call foo(&bar + 17) in C, I want to be able to make that call in Python.

I wasn't using (c_type*N).from_buffer because I wanted to. I was using it after wasting literally hours trying to find some other way to get ctypes to agree to pass a pointer into the buffer of a Python object to an external function (which was not going to alter it, I promise). This should be easy; instead it's nearly impossible. I don't want to wrestle with random un-overridable attempts to enforce correctness when calling a language where that can never be enforced. I just want to call my C function.

I'm pretty sure that I verified that this code worked in 3.1.3 before opening this bug, but it's been a while. I could try to reproduce it, but I think this functionality should be present regardless. You can call it a feature request instead of a regression if you want.
msg148588 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-11-29 17:57
> I just want to call my C function.

You can use something else than ctypes_array.from_buffer(buffer). ctypes_array.from_buffer() creates a read-write object which can be used to modify the buffer. You cannot pass a read-only object to ctypes_array.from_buffer().

> I'm pretty sure that I verified that this code worked in 3.1.3 before
> opening this bug, but it's been a while.

If it was possible, it was a bug. You cannot modify a read-only object, like bytes, because it would lead to inconsistent object and may lead your program to a crash, even if you don't call C functions.

Example in pure Python:

>>> import ctypes
>>> T=(ctypes.c_uint8*4)
>>> B=bytearray(4)
>>> x=T.from_buffer(B)
>>> B
bytearray(b'\x00\x00\x00\x00')
>>> x[0]=9
>>> x[2]=100
>>> B
bytearray(b'\t\x00d\x00')
>>> list(x)                                                                                                                                             
[9, 0, 100, 0]

B is modified when x[index] is modified.

--

A bug tracker is not the right place to ask help with ctypes.
msg224395 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-07-31 11:30
For the record, I have been (ab?)using the “c_char_p” type to get the address of immutable byte strings without copying memory:

>>> from ctypes import *
>>> data = b"123\x00abc"
>>> pubyte = cast(c_char_p(data), POINTER(c_ubyte))
>>> address = addressof(pubyte.contents)
>>> string_at(address, 7)
b'123\x00abc'
>>> # 2nd instance has same address, rather than memory copy:
... cast(c_char_p(data), c_void_p).value == address
True

In to the documentation, “c_char_p” is only defined for zero-terminated strings, but it seems to also work for strings with embedded zero bytes. However it does not work for “bytearray” objects (“from_buffer” should work instead), nor memory views.
msg224420 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2014-07-31 15:41
cast calls ctypes._cast(obj, obj, typ). _cast is a ctypes function pointer defined as follows:

    _cast = PYFUNCTYPE(py_object, 
                       c_void_p, py_object, py_object)(_cast_addr)

Since cast makes an FFI call that converts the first arg to c_void_p, you can directly cast bytes to a pointer type:

    >>> from ctypes import *
    >>> data = b'123\x00abc'

    >>> ptr = cast(data, c_void_p)

string_at is defined similarly using c_void_p for the first arg:

    _string_at = PYFUNCTYPE(py_object, c_void_p, c_int)(_string_at_addr)
 
    >>> string_at(ptr, 8)
    b'123\x00abc\x00'

Get around the from_buffer mutability requirement by casting to an array (i.e. to an array pointer followed by a dereference):

    >>> arr = cast(data, POINTER(c_char * len(data)))[0]
    >>> arr[:]
    b'123\x00abc'

Then use byref to pass an offset into the array:

    >>> from ctypes.util import find_library
    >>> printf = CDLL(find_library('c')).printf

    >>> printf(b'%s\n', byref(arr, 4))
    abc
    4

Since this doesn't copy the buffer, take care to ensure the function call won't modify the contents.
History
Date User Action Args
2022-04-11 14:57:14adminsetgithub: 55636
2014-07-31 15:41:14eryksunsetnosy: + eryksun
messages: + msg224420
2014-07-31 11:30:47martin.pantersetnosy: + martin.panter
messages: + msg224395
2011-11-29 17:57:48vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg148588
2011-11-29 16:55:31benrgsetstatus: closed -> open
resolution: not a bug -> (no value)
messages: + msg148586
2011-11-29 13:19:47vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg148561
2011-11-29 02:38:48meador.ingesetmessages: + msg148535
2011-11-10 03:10:49meador.ingesetmessages: + msg147392
2011-11-09 19:07:25santoso.wijayasetmessages: + msg147364
2011-11-09 19:02:17santoso.wijayasetnosy: + santoso.wijaya

versions: + Python 3.3
2011-11-09 17:59:15vstinnersetnosy: + vstinner
2011-11-09 03:33:06meador.ingesetmessages: + msg147331
stage: needs patch
2011-11-08 22:42:27ezio.melottisetnosy: + meador.inge
2011-03-07 07:36:17georg.brandlsetkeywords: + 3.2regression
nosy: theller, benrg
2011-03-07 07:11:24benrgcreate