Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pickle.dumps(xrange(sys.maxsize)) produces xrange(0) #60233

Closed
4kir4 mannequin opened this issue Sep 24, 2012 · 12 comments
Closed

pickle.dumps(xrange(sys.maxsize)) produces xrange(0) #60233

4kir4 mannequin opened this issue Sep 24, 2012 · 12 comments
Assignees
@mdickinson
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@4kir4
Copy link
Mannequin

4kir4 mannequin commented Sep 24, 2012

BPO 16029
Nosy @mdickinson, @4kir4
Files
  • test_pickle_dumps_xrange.py
  • issue16029.patch
  • xrange_reduce_repr.patch
  • xrange_reduce_repr_2.patch
  • xrange_reduce_repr_3.patch
  • xrange_reduce_repr_4.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/mdickinson'
closed_at = <Date 2012-09-28.19:50:33.917>
created_at = <Date 2012-09-24.20:28:59.620>
labels = ['interpreter-core', 'type-bug']
title = 'pickle.dumps(xrange(sys.maxsize)) produces xrange(0)'
updated_at = <Date 2012-09-28.19:50:33.917>
user = 'https://github.com/4kir4'

    bugs.python.org fields:

    activity = <Date 2012-09-28.19:50:33.917>
actor = 'mark.dickinson'
assignee = 'mark.dickinson'
closed = True
closed_date = <Date 2012-09-28.19:50:33.917>
closer = 'mark.dickinson'
components = ['Interpreter Core']
creation = <Date 2012-09-24.20:28:59.620>
creator = 'akira'
dependencies = []
files = ['27281', '27282', '27289', '27295', '27296', '27302']
hgrepos = []
issue_num = 16029
keywords = ['patch']
message_count = 12.0
messages = ['171187', '171188', '171189', '171190', '171192', '171195', '171228', '171265', '171269', '171310', '171533', '171535']
nosy_count = 3.0
nosy_names = ['mark.dickinson', 'akira', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16029'
versions = ['Python 2.7']
    @4kir4
    Copy link
    Mannequin Author

    4kir4 mannequin commented Sep 24, 2012 

    >>> import sys
  >>> from pickle import dumps, loads
  >>> r = xrange(sys.maxsize)
  >>> len(r) == sys.maxsize
  True
  >>> pr = loads(dumps(r))
  >>> len(pr) == len(r)
  False
  >>> pr
  xrange(0)
  >>> r
  xrange(9223372036854775807)

    It breaks multiprocessing module:
    http://stackoverflow.com/questions/12569977/python-large-iterations-number-fail

    It fails on 2.6.6, 2.7.3. It works correctly on 3.1-3.3, pypy 1.7-1.9 x86_64 Linux.

    @4kir4 4kir4 mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Sep 24, 2012
    @mdickinson
    Copy link
    Member

    The bug is (not surprisingly) in range_reduce in Objects/rangeobject.c, where

    return Py_BuildValue("(O(iii))", Py_TYPE(r),

    should be

    return Py_BuildValue("(O(lll))", Py_TYPE(r),

    But in writing tests for this bug, I fell over another one:

    >>> import sys
>>> xrange(0, sys.maxint, sys.maxint-1)
xrange(0, -4, 2147483646)

    @mdickinson
    Copy link
    Member

    Here's the fix. There's a commented out test, which fails because of the second xrange bug (or something closely related to it).

    @mdickinson
    Copy link
    Member

    Removing 2.6: this isn't a security issue.

    @mdickinson
    Copy link
    Member

    Okay, the xrange stop for both its pickle and its repr is computed as:

    r->start + r->len * r->step

    If this overflows, it gives a bad value. It would suffice to replace it with sys.maxint or -sys.maxint - 1 on overflow.

    I'll look at this shortly.

    @mdickinson mdickinson self-assigned this Sep 24, 2012
    @mdickinson
    Copy link
    Member

    Opened issue bpo-16030 for the repr issue. The patch for this issue still lacks a fix for the stop value.

    @mdickinson
    Copy link
    Member

    Updated patch, which also fixes bpo-16030. It needs more tests.

    @mdickinson
    Copy link
    Member

    Patch with tests.

    @mdickinson mdickinson added interpreter-core (Objects, Python, Grammar, and Parser dirs) and removed stdlib Python modules in the Lib dir labels Sep 25, 2012
    @mdickinson
    Copy link
    Member

    Whoops; there's no need to iterate over pickle protocols in test_repr. New patch.

    @mdickinson
    Copy link
    Member

    Updated patch: rename range_stop, as suggested in Rietveld review.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 28, 2012

    New changeset bff269ee7288 by Mark Dickinson in branch '2.7':
    Issues bpo-16029, bpo-16030: Fix pickling and repr of large xranges.
    http://hg.python.org/cpython/rev/bff269ee7288

    @mdickinson
    Copy link
    Member

    Now fixed. Thanks for the report!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @mdickinson mdickinson

    Labels
    interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @mdickinson