classification
Title: Fix range slicing and indexing to handle lengths > sys.maxsize
Type: Stage: resolved
Components: Versions: Python 3.2
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: georg.brandl, mark.dickinson, ncoghlan
Priority: normal Keywords: patch

Created on 2011-01-11 15:59 by ncoghlan, last changed 2011-01-12 07:53 by mark.dickinson. This issue is now closed.

Files
File name Uploaded Description Edit
issue10889_range_subscripts.diff ncoghlan, 2011-01-11 18:19 Indexing and slicing for large ranges
Messages (6)
msg126017 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-11 15:59
Enhancement to range to correctly handle indexing and slicing when len(x) raises OverflowError.

Note that this enables correct calculation of the length of such ranges via:

        def _range_len(x):
            try:
                length = len(x)
            except OverflowError:
                step = x[1] - x[0]
                length = 1 + ((x[-1] - x[0]) // step)
            return length
msg126023 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-11 17:14
Having started work on this, the code changes are probably too significant to consider adding it to 3.2 at this late stage.

Writing my own slice interpretation support which avoids the ssize_t limit is an interesting exercise :)
msg126027 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-11 18:19
Attached patch moves range indexing and slicing over to PyLong and updates the tests accordingly.

Georg, I think this really makes the large range story far more usable - if you're OK with it, I would like to check it in this week so it lands in 3.2.
msg126029 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-11 18:34
Oh, and to explain my negative comment from earlier: that was my reaction when I realised I also needed to write PyLong versions of _PyEval_SliceIndex and PySlice_GetIndicesEx to make range slicing with large integers work properly.

As it turned out, the end result wasn't as scary as I initially feared (while compute_slice_indices is quite long, most of that is just the verbosity of PyLong arithmetic).
msg126030 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-01-11 18:44
It's a moderate chunk of code, but lots of new tests... I'd say go for it.
msg126069 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-12 03:21
Committed as r87948.

I added a few large_range tests to those in the patch. I checked that IndexError is raised when appropriate, as well as a specific test for the combination of a large range with a large negative step.
History
Date User Action Args
2011-01-12 07:53:56mark.dickinsonsetnosy: + mark.dickinson
2011-01-12 03:22:45ncoghlansetstatus: open -> closed
resolution: accepted
stage: resolved
2011-01-12 03:21:52ncoghlansetmessages: + msg126069
2011-01-11 18:44:06georg.brandlsetmessages: + msg126030
2011-01-11 18:34:53ncoghlansetmessages: + msg126029
2011-01-11 18:19:39ncoghlansetfiles: + issue10889_range_subscripts.diff

assignee: ncoghlan -> georg.brandl
versions: + Python 3.2, - Python 3.3
keywords: + patch
nosy: + georg.brandl

messages: + msg126027
2011-01-11 17:14:34ncoghlansetmessages: + msg126023
versions: + Python 3.3
2011-01-11 15:59:27ncoghlancreate