classification
Title: Bug in range() function for large values
Type: crash Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 3.0, Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: akitada, belopolsky, christian.heimes, josm, loewis, rhettinger, robertwb, zanella (8)
Priority: Keywords patch

Created on 2007-12-01 00:28 by robertwb, last changed 2009-01-02 21:50 by rhettinger.

Files
File name Uploaded Description Edit Remove
bad_range.py robertwb, 2007-12-01 00:28
bltinmodule.diff belopolsky, 2008-02-24 23:08
Messages (17)
msg58034 - (view) Author: Robert Bradshaw (robertwb) Date: 2007-12-01 00:28
Range accepts arguments coerce-able into ints via __int__, but rejects
arguments coerce-able into longs but to large to fit into an int. 

The problem is in handle_range_longs in bltinmodule.c:1527-1541. If they
type is not an int or long, it should try to make it so before failing
(for consistency with smaller values at least). 

Attached is a file that reproduces this bug.
msg58039 - (view) Author: John Smith (josm) Date: 2007-12-01 01:37
Is this a bug?
  print range(MyInt(2**64), MyInt(2**64+10)) 
produced
 TypeError: range() integer start argument expected, got instance.

print range(int(MyInt(2**64)), int(MyInt(2**64+10)))
should work.
msg58040 - (view) Author: Robert Bradshaw (robertwb) Date: 2007-12-01 01:42
Yes, that is a workaround, but 

range(MyInt(n), MyInt(n+10))

should work for any valid value of n, not just some of them.
msg62617 - (view) Author: Rafael Zanella (zanella) Date: 2008-02-21 11:16
FWIW, using xrange() it seems to give the proper error message:

Traceback (most recent call last):
  File "bad_range.py", line 12, in <module>
    print xrange(MyInt(2**64), MyInt(2**64+10))
OverflowError: long int too large to convert to int
msg62631 - (view) Author: Robert Bradshaw (robertwb) Date: 2008-02-21 17:02
Yes, the error for xrange is more illustrative of the problem, but just
shows that xrange has this a too. Why should xrange be invalid for
non-word sized values (especially as range works)? Incidentally, just a
week ago I had to write my own iterator for a project because xrange
couldn't handle large values.
msg62898 - (view) Author: Rafael Zanella (zanella) Date: 2008-02-24 13:19
According to the documentation
(http://docs.python.org/dev/library/functions.html) "The arguments must
be plain integers", so I think the wrong thing here is to run the
object's __int__() under the range()'s hood. I think the right thing to
do would be to explicitly invoke int() on passing an non-int argument as
parameter.
msg62937 - (view) Author: Alexander Belopolsky (belopolsky) Date: 2008-02-24 21:03
This is related to issue1540617 and issue1546078.

issue1540617 contains a simple patch that extend acceptable range of 
argument to +/-2**63

issue1546078 contains a complete long integer support implementation and  
was accepted in Py3k.

It looks like all three issues can be closed by either accepting or 
rejecting issue1540617 patch for 2.6 and marking issue1546078 patch as 
accepted for Py3k.
msg62938 - (view) Author: Christian Heimes (christian.heimes) Date: 2008-02-24 21:07
I'm -10 on the patch in issue1540617 ( +/-2**63).

Reason: It's a good thing that the range of "range" is limited since it
returns a list of integers. range(2**32) allocates (2**32)*16 bytes +
small overhead for ints plus the space for the list (probably
(2**32)*sizeof(ptr) which is 4 or 8 bytes). So far the memory for the
ints is *never* returned to the system. I'm working on the problem.
msg62943 - (view) Author: Alexander Belopolsky (belopolsky) Date: 2008-02-24 21:44
Christian,

I was probably a bit sloppy using "range" instead of "xrange," but issue1540617 is limited to xrange only.  Are you still -10 on extending 
xrange on 64-bit platforms to +/- 2**63?  If so, what is your position 
on backporting py3k's unlimited range implementation?
msg62945 - (view) Author: Alexander Belopolsky (belopolsky) Date: 2008-02-24 22:00
> So far the memory for the ints is *never* returned
> to the system. I'm working on the problem.

Christian,

Are you working on the memory problem or on this issue? I think I have a 
solution to OP's problem, but don't want to duplicate your effort.
msg62947 - (view) Author: Christian Heimes (christian.heimes) Date: 2008-02-24 22:14
I'm working on the memory problem. See #2039 and #2013.

xrange is a totally different story. I'm +0 on changing xrange. Is
Python 3.0's range function compatible with xrange? If the answer is
yes, we may reuse the code for an unlimited xrange.
msg62950 - (view) Author: Alexander Belopolsky (belopolsky) Date: 2008-02-24 23:08
Attached patch addresses OP's issue:

$ ./python.exe bad_range.py
[8, 9, 10, 11, 12, 13, 14, 15, 16, 17]
here
[18446744073709551616L, 18446744073709551617L, 18446744073709551618L, 
18446744073709551619L, 18446744073709551620L, 18446744073709551621L, 
18446744073709551622L, 18446744073709551623L, 18446744073709551624L, 
18446744073709551625L]
[18446744073709551616L, 18446744073709551617L, 18446744073709551618L, 
18446744073709551619L, 18446744073709551620L, 18446744073709551621L, 
18446744073709551622L, 18446744073709551623L, 18446744073709551624L, 
18446744073709551625L]

The only existing test that fails is range(1e100, 1e101, 1e101) 
producing a TypeError. It will now produce

>>> range(1e100, 1e101, 1e101)
__main__:1: DeprecationWarning: integer argument expected, got float
[10000000000000000159028911097599180468360808563945281389781327557747838
772170381060813469985856815104L]

Note that range(1e100, 1e101) would still fail:
>>> range(1e100, 1e101)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: range() result has too many items

An alternative solution would be to disallow non-ints regardless of 
their value, but that is more likely to break someone's code.
msg62989 - (view) Author: Robert Bradshaw (robertwb) Date: 2008-02-25 17:52
Alexander Belopolsky's patch looks like the right fix for range() to me. 

The xrange limits still hold, but that should probably be a separate
issue/ticket.
msg76623 - (view) Author: Akira Kitada (akitada) Date: 2008-11-29 20:28
I'm just curious to know which is the right behavior.

# Python 3.0
Traceback (most recent call last):
  File "bad_range.py", line 7, in <module>
    print(range(MyInt(2**3), MyInt(2**3+10)))
TypeError: 'MyInt' object cannot be interpreted as an integer

# Python 2.7a0
[8, 9, 10, 11, 12, 13, 14, 15, 16, 17]
here
[18446744073709551616L, 18446744073709551617L, 18446744073709551618L,
18446744073709551619L, 18446744073709551620L, 18446744073709551621L,
18446744073709551622L, 18446744073709551623L, 18446744073709551624L,
18446744073709551625L]
Traceback (most recent call last):
  File "bad_range.py", line 11, in <module>
    print(range(MyInt(2**64), MyInt(2**64+10)))
TypeError: range() integer start argument expected, got instance.
msg76624 - (view) Author: Robert Bradshaw (robertwb) Date: 2008-11-29 20:45
I think *both* behaviors are wrong, the 3.0 one is backwards
incompatible, and the 2.7 one is inconsistent (accepting MyInt if it's <
32 bits, rejecting it for > 64 bits). 

For our particular use case, it is very annoying to not be able to use
non-ints. It goes against the principle duck typing by simply defining
the __int__ and __index__ methods.
msg76630 - (view) Author: Akira Kitada (akitada) Date: 2008-11-29 22:53
Updating versions.
msg77509 - (view) Author: Martin v. Löwis (loewis) Date: 2008-12-10 08:52
This change is out of scope for 2.5.3 (plus, the desired behavior still
seems to be debated)
History
Date User Action Args
2009-01-02 21:50:18rhettingersetassignee: rhettinger ->
2008-12-10 08:52:59loewissetnosy: + loewis
messages: + msg77509
versions: - Python 2.5, Python 2.5.3
2008-11-29 22:53:58akitadasetmessages: + msg76630
versions: + Python 2.6, Python 3.0, Python 2.7, Python 2.5.3
2008-11-29 20:45:36robertwbsetmessages: + msg76624
2008-11-29 20:28:15akitadasetnosy: + akitada
messages: + msg76623
2008-02-25 17:52:53robertwbsetmessages: + msg62989
2008-02-24 23:08:45belopolskysetfiles: + bltinmodule.diff
keywords: + patch
messages: + msg62950
2008-02-24 22:14:51christian.heimessetmessages: + msg62947
2008-02-24 22:00:36belopolskysetmessages: + msg62945
2008-02-24 21:44:36belopolskysetmessages: + msg62943
2008-02-24 21:07:59christian.heimessetnosy: + christian.heimes
messages: + msg62938
2008-02-24 21:03:21belopolskysetnosy: + belopolsky
messages: + msg62937
2008-02-24 13:19:42zanellasetmessages: + msg62898
2008-02-21 17:02:08robertwbsetmessages: + msg62631
2008-02-21 11:16:28zanellasetnosy: + zanella
messages: + msg62617
2007-12-01 13:25:53rhettingersetassignee: rhettinger
nosy: + rhettinger
2007-12-01 01:42:03robertwbsetmessages: + msg58040
2007-12-01 01:37:18josmsetnosy: + josm
messages: + msg58039
2007-12-01 00:28:10robertwbcreate