Title: slice-object support for ctypes Pointer/Array
Type: Stage:
Components: Extension Modules Versions: Python 2.6
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: twouters Nosy List: theller, twouters
Priority: normal Keywords: patch

Created on 2006-12-18 04:28 by twouters, last changed 2007-08-30 23:49 by twouters. This issue is now closed.

File name Uploaded Description Edit
noslice.ctypes.diff twouters, 2007-08-29 21:26
Messages (14)
msg51567 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2006-12-18 04:28
Support for slicing ctypes' Pointer and Array types with slice objects, although only for step=1 case.
(Backported from p3yk-noslice branch.)
msg51568 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2006-12-20 18:45
Unfortunately I'm unable to review or work on this patch *this year*.  I
will definitely take a look in January.  Sorry.
msg51569 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2007-01-12 20:48
Thomas, a question:  Since steps != 1 are not supported, does this patch have any value?
IIUC, array[x:y] returns exactly the same as array[x:y:1] for all x and y values.

Formally, the patch is missing unittests and documentation ;-).
msg51570 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-01-15 11:48
The point is that simple slicing will go away, and extended slices (with sliceobjects) are used in various places, and currently can't be passed on to ctypes arrays and pointers. That is to say, a Python class defining __getitem__ but not __getslice__ still supports slice syntax, but it can't do 'pointer[sliceobj]' -- it would have to do 'pointer[sliceobj.start:sliceobj.end]'. Also, because simple slices will go away, this code will have to be added to the p3yk branch in any case; having it in the trunk just makes for easier maintenance.

Oh, and the non-support for steps other than 1 is not a fundamental issue, I just couldn't bear to write the code for that if you didn't think it would be useful, as I'd already written the same logic and arithmetic for array, tupleseq, mmap and I forget what else :P You can consider this code half-done, if you wish; I'll get to it again soon enough.
msg51571 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-07-12 07:54
Thomas, do you consider non-step-1 slicing important? I'd really like to get the noslice branch merged into p3yk soon-ish, and I need to know which bits I need to commit to the trunk first (to avoid merge conflicts for the rest of the lifetime of the 2.x codebase.)
msg51572 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2007-07-12 10:17
I think that full slicing support for pointers/arrays would be great.  Presonally I have a use case for step = -1, in other words, I need to iterate backwards over a ctypes array.
msg55374 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-08-28 15:37
I'd like to check this into the trunk, without the non-step-1 support
for now, so that we can remove simple slicing from the py3k branch. We
can always add non-step-1 support later (all the sooner if someone who
isn't me volunteers to do the painful bits of that support, probably by
copy-pasting from the array module ;-)
msg55419 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-08-29 13:43
Added tests (by duplicating any slicing operations in the test suite
with extended slice syntax, to force the use of slice-objects ;)
msg55442 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-08-29 21:26
Implemented actual extended slicing support for the Array and Pointer
types. The semantics of negative and ommitted indices in the Pointer
case are somewhat muddy, but I took the approach the simple slicing code
took: pretend the programmer knows what they're doing, and that the
'sequence' that is the pointer is actually PY_SSIZE_T_MAX elements long.
I'm not sure I like that approach at all, but the actual problem is the
unbounded nature of the pointers, and this is the only approach I can
think of that makes sense, short of completely disallowing negative or
missing indices.

This patch is pretty much done now, I think. Note that it's entirely
backward compatible, the incompatible bits (removing simple slicing)
happen in Py3K.
msg55497 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2007-08-30 18:15
Set to accepted.  As pointed out in private email, please apply it to
the trunk.

Your thoughts about the 'length' of pointers make sense, and are very
similar to what I had in mind when I implemented pointer indexing.

For indexing pointers, negative indices (in the C sense, not the usual
Python sense) absolutely are needed, IMO.  For slicing, missing indices 
do not really have a meaning - would it be possible to disallow them?
msg55506 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-08-30 20:10
Well, that's not quite how I implemented the slicing, and it's also not
how the existing simple-slicing was implemented: A negative start index
is taken to mean 0, and a stop index below the start index is taken to
mean 'the start index' (leading to an empty slice.)

However, it isn't too hard to do what I think you want done: a negative
index means indexing before the pointer, not from the end of the
pointer, and missing indices are only okay if they clearly mean '0'
('start' when step > 0, 'stop' when step < 0.)

 P[5:10] would slice from P[5] up to but not including P[10],
 P[-5:5] would slice from P[-5] up to but not including P[5],
 P[:5] would slice from P[0] up to but not including P[5],
 P[5::-1] would slice from P[5] down to *and including* P[0]
but the following would all be errors:

Does that sound like what you wanted?
msg55507 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2007-08-30 20:19

But looking at your examples I think it would be better to forbid
missing indices completely instead of allowing them only where they
clearly mean 0.

Writing (and reading!) a 0 is faster than thinking about if a missing
index is allowed or what it means.
msg55510 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-08-30 20:36
Hmmm.... Well, that's fine by me, but it changes current behaviour, and
in a way that ctypes own testsuite was testing, even ;) (it does, e.g.,
'p[:4]' in a couple of places.) Requiring the start always would
possibly break a lot of code. We could make only the start (and step)
optional, and the start only if the step is positive, perhaps? That
would change no existing, sane behaviour.
msg55518 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2007-08-30 23:49
Checked in a slightly newer version.
Date User Action Args
2007-08-30 23:49:52twouterssetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg55518
2007-08-30 20:36:11twouterssetmessages: + msg55510
2007-08-30 20:19:10thellersetmessages: + msg55507
2007-08-30 20:10:18twouterssetmessages: + msg55506
2007-08-30 18:15:47thellersetassignee: theller -> twouters
2007-08-30 18:15:18thellersetresolution: accepted
messages: + msg55497
2007-08-29 21:26:37twouterssetfiles: - noslice.ctypes.diff
2007-08-29 21:26:28twouterssetfiles: + noslice.ctypes.diff
messages: + msg55442
2007-08-29 13:43:51twouterssetfiles: - noslice.ctypes.diff
2007-08-29 13:43:44twouterssetfiles: + noslice.ctypes.diff
messages: + msg55419
2007-08-28 15:37:44twouterssetmessages: + msg55374
2006-12-18 04:28:01twouterscreate