classification
Title: ctypes arrays have incorrect buffer information (PEP-3118)
Type: behavior Stage: resolved
Components: ctypes Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Tilka, amaury.forgeotdarc, belopolsky, eryksun, haypo, mark.dickinson, mattip, meador.inge, pitrou, pv, python-dev, ronaldoussoren, skrah, theller
Priority: normal Keywords: patch

Created on 2010-12-20 22:06 by pv, last changed 2014-05-17 22:07 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
001-ctypes-fix-pep-3118-format-strings-for-arrays.patch pv, 2010-12-20 22:06 Suggested fix (on py3k branch)
hg-python2.7-ctypes-fix-pep3118-format-strings-for-arrays-updated.patch mattip, 2014-05-11 19:57 python 2.7 patch review
hg-default-ctypes-fix-pep3118-format-strings-for-arrays-update.patch mattip, 2014-05-11 19:58 patch against HEAD review
hg-default-ctypes-fix-pep3118-format-strings-for-arrays-update.patch mattip, 2014-05-12 14:54 review
Messages (11)
msg124406 - (view) Author: Pauli Virtanen (pv) Date: 2010-12-20 22:06
Ctypes arrays have invalid buffer interface information (on Python 3.1.2):

>>> import ctypes
>>> x = (ctypes.c_double*2)()
>>> y = memoryview(x)
>>> y.shape
(2,)
>>> y.format
'(2)<d'

This implies that the array contains 2 items, each consisting of 2 floats, which is incorrect -- the shape information is duplicated.

A suggested fix is attached.

(From http://projects.scipy.org/numpy/ticket/1699)
msg142081 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-08-14 20:28
Thanks for the patch. I agree with the interpretation of the format string.
One thing is unclear though: Using this interpretation the multi-dimensional 
array notation in format strings only seems useful for pointers to arrays.

The PEP isn't so clear on that, would you agree?


I'm not done reviewing the patch, just a couple of nitpicks:

  - We need a function declaration of _ctypes_alloc_format_string_with_shape()
    in ctypes.h.

  - prefix_len = 32*(ndim+1) + 3: This is surely sufficient, but (ndim+1)
    is not obvious to me. I think we need (20 + 1) * ndim + 3.

  - I'd use "%zd" for Py_ssize_t (I know that in other parts of the
    code "%ld" is used, too).
msg142192 - (view) Author: Pauli Virtanen (pv) Date: 2011-08-16 13:27
The array notation is useful for arrays inside structs, such as "T{(4)i(2,3)f}".
msg191562 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-21 10:30
@skrah: What is the status of this issue?
msg191575 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-06-21 12:57
Basically someone has to review the patch and commit it. I'm not really
using ctypes, so for me a decent review would take a while.
msg191583 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-21 14:23
The correct format string for Py_ssize_t with sprintf is 
"%"PY_FORMAT_SIZE_T"d" (defined and explained in pyport.h).
msg218292 - (view) Author: mattip (mattip) * Date: 2014-05-11 19:57
Here are updated patches for 2.7 and HEAD. 

I've updated the patches for both python 2 and python 3 and addressed some of the issues here:
1. Added _ctypes_alloc_format_string_with_shape() to ctypes.h
2. Changed the buffer size calculation to 32 *ndim + 3 (for the question asked before, the 32 size is just because this was the size used before the patch....)
3. Changed the sprintf to use the macro: sprintf(buf, "%"PY_FORMAT_SIZE_T"d,", shape[k]);

I really hope this patch makes it to 2.7.7 and python 3 versions, as currently the situation is broken and code that once worked with older numpy versions does not work today (and we have legacy code that stopped working).
msg218293 - (view) Author: mattip (mattip) * Date: 2014-05-11 19:58
Here is the patch for HEAD
msg218298 - (view) Author: eryksun (eryksun) Date: 2014-05-11 21:00
The shape for the StructWithArrays test should be () in 3.x; it's None in 2.x.
msg218322 - (view) Author: mattip (mattip) * Date: 2014-05-12 14:54
Updated patch for default for empty shape, thanks eryksun
msg218716 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-17 22:07
New changeset 44dac2e7fcb8 by Benjamin Peterson in branch '2.7':
support pep 3118 format strings for ctypes objects with nontrivial shapes (closes #10744)
http://hg.python.org/cpython/rev/44dac2e7fcb8

New changeset 22938bf57161 by Benjamin Peterson in branch '3.4':
support pep 3118 format strings for ctypes objects with nontrivial shapes (closes #10744)
http://hg.python.org/cpython/rev/22938bf57161

New changeset ea9193340d6c by Benjamin Peterson in branch 'default':
merge 3.4 (#10744)
http://hg.python.org/cpython/rev/ea9193340d6c
History
Date User Action Args
2014-05-17 22:07:43python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg218716

resolution: fixed
stage: patch review -> resolved
2014-05-12 14:54:40mattipsetfiles: + hg-default-ctypes-fix-pep3118-format-strings-for-arrays-update.patch

messages: + msg218322
2014-05-11 21:00:07eryksunsetnosy: + eryksun

messages: + msg218298
versions: + Python 3.3, - Python 3.5
2014-05-11 19:59:53pitrousetversions: + Python 2.7
2014-05-11 19:59:43pitrousetnosy: + amaury.forgeotdarc, belopolsky, meador.inge

versions: + Python 3.5, - Python 2.7, Python 3.3
2014-05-11 19:58:12mattipsetfiles: + hg-default-ctypes-fix-pep3118-format-strings-for-arrays-update.patch

messages: + msg218293
2014-05-11 19:57:34mattipsetfiles: + hg-python2.7-ctypes-fix-pep3118-format-strings-for-arrays-updated.patch
versions: + Python 2.7
nosy: + mattip

messages: + msg218292
2013-06-21 14:23:22ronaldoussorensetnosy: + ronaldoussoren
messages: + msg191583
2013-06-21 12:57:59skrahsetmessages: + msg191575
2013-06-21 10:30:21hayposetmessages: + msg191562
2013-06-21 10:29:53hayposetversions: + Python 3.4, - Python 3.1, Python 3.2
2013-06-21 08:52:59Tilkasetnosy: + Tilka
2011-08-16 13:30:30hayposetnosy: + haypo
2011-08-16 13:27:20pvsetmessages: + msg142192
2011-08-14 20:28:36skrahsetassignee: theller ->
messages: + msg142081
stage: patch review
2011-08-10 08:22:17skrahsetnosy: + skrah
2010-12-21 18:46:51georg.brandlsetnosy: + pitrou
2010-12-21 18:31:24r.david.murraysetnosy: + mark.dickinson
2010-12-20 22:06:47pvcreate