classification
Title: pickle.dumps(xrange(sys.maxsize)) produces xrange(0)
Type: behavior Stage: commit review
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: mark.dickinson Nosy List: akira, mark.dickinson, python-dev
Priority: normal Keywords: patch

Created on 2012-09-24 20:28 by akira, last changed 2012-09-28 19:50 by mark.dickinson. This issue is now closed.

Files
File name Uploaded Description Edit
test_pickle_dumps_xrange.py akira, 2012-09-24 20:28
issue16029.patch mark.dickinson, 2012-09-24 21:04 review
xrange_reduce_repr.patch mark.dickinson, 2012-09-25 07:16 review
xrange_reduce_repr_2.patch mark.dickinson, 2012-09-25 13:12 review
xrange_reduce_repr_3.patch mark.dickinson, 2012-09-25 13:20 review
xrange_reduce_repr_4.patch mark.dickinson, 2012-09-25 19:12 review
Messages (12)
msg171187 - (view) Author: Akira Li (akira) * Date: 2012-09-24 20:28
>>> import sys
  >>> from pickle import dumps, loads
  >>> r = xrange(sys.maxsize)
  >>> len(r) == sys.maxsize
  True
  >>> pr = loads(dumps(r))
  >>> len(pr) == len(r)
  False
  >>> pr
  xrange(0)
  >>> r
  xrange(9223372036854775807)

It breaks multiprocessing module:
http://stackoverflow.com/questions/12569977/python-large-iterations-number-fail

It fails on 2.6.6, 2.7.3. It works correctly on 3.1-3.3, pypy 1.7-1.9  x86_64 Linux.
msg171188 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-24 20:59
The bug is (not surprisingly) in range_reduce in Objects/rangeobject.c, where 

    return Py_BuildValue("(O(iii))", Py_TYPE(r),

should be

    return Py_BuildValue("(O(lll))", Py_TYPE(r),

But in writing tests for this bug, I fell over another one:


>>> import sys
>>> xrange(0, sys.maxint, sys.maxint-1)
xrange(0, -4, 2147483646)
msg171189 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-24 21:04
Here's the fix.  There's a commented out test, which fails because of the second xrange bug (or something closely related to it).
msg171190 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-24 21:04
Removing 2.6:  this isn't a security issue.
msg171192 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-24 21:14
Okay, the xrange stop for both its pickle and its repr is computed as:

   r->start + r->len * r->step

If this overflows, it gives a bad value.  It would suffice to replace it with sys.maxint or -sys.maxint - 1 on overflow.

I'll look at this shortly.
msg171195 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-24 21:24
Opened issue #16030 for the repr issue.  The patch for this issue still lacks a fix for the stop value.
msg171228 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-25 07:16
Updated patch, which also fixes issue 16030.  It needs more tests.
msg171265 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-25 13:12
Patch with tests.
msg171269 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-25 13:20
Whoops; there's no need to iterate over pickle protocols in test_repr.  New patch.
msg171310 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-25 19:12
Updated patch:  rename range_stop, as suggested in Rietveld review.
msg171533 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-09-28 19:48
New changeset bff269ee7288 by Mark Dickinson in branch '2.7':
Issues #16029, #16030: Fix pickling and repr of large xranges.
http://hg.python.org/cpython/rev/bff269ee7288
msg171535 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-28 19:50
Now fixed.  Thanks for the report!
History
Date User Action Args
2012-09-28 19:50:33mark.dickinsonsetstatus: open -> closed
resolution: fixed
messages: + msg171535
2012-09-28 19:48:59python-devsetnosy: + python-dev
messages: + msg171533
2012-09-25 19:12:31mark.dickinsonsetfiles: + xrange_reduce_repr_4.patch

messages: + msg171310
2012-09-25 13:20:30mark.dickinsonsetfiles: + xrange_reduce_repr_3.patch

messages: + msg171269
2012-09-25 13:13:11mark.dickinsonlinkissue16030 dependencies
2012-09-25 13:12:12mark.dickinsonsetfiles: + xrange_reduce_repr_2.patch

messages: + msg171265
components: + Interpreter Core, - Library (Lib)
stage: needs patch -> commit review
2012-09-25 07:16:40mark.dickinsonsetfiles: + xrange_reduce_repr.patch

messages: + msg171228
2012-09-24 21:24:46mark.dickinsonsetmessages: + msg171195
2012-09-24 21:16:03mark.dickinsonsetstage: patch review -> needs patch
2012-09-24 21:14:57mark.dickinsonsetassignee: mark.dickinson
messages: + msg171192
2012-09-24 21:04:43mark.dickinsonsetstage: patch review
messages: + msg171190
versions: - Python 2.6
2012-09-24 21:04:08mark.dickinsonsetfiles: + issue16029.patch
keywords: + patch
messages: + msg171189
2012-09-24 20:59:09mark.dickinsonsetnosy: + mark.dickinson
messages: + msg171188
2012-09-24 20:28:59akiracreate