Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange Exception from copying an iterator #58558

Closed
JakobBowyer mannequin opened this issue Mar 17, 2012 · 9 comments
Closed

Strange Exception from copying an iterator #58558

JakobBowyer mannequin opened this issue Mar 17, 2012 · 9 comments
Labels
type-bug An unexpected behavior, bug, or error

Comments

@JakobBowyer
Copy link
Mannequin

JakobBowyer mannequin commented Mar 17, 2012

BPO 14350
Nosy @rhettinger, @terryjreedy, @avassalotti, @bitdancer, @serhiy-storchaka
Superseder
  • bpo-22995: Restrict default pickleability
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2015-11-12.10:05:47.873>
    created_at = <Date 2012-03-17.19:04:25.627>
    labels = ['type-bug']
    title = 'Strange Exception from copying an iterator'
    updated_at = <Date 2015-11-12.10:05:47.872>
    user = 'https://bugs.python.org/JakobBowyer'

    bugs.python.org fields:

    activity = <Date 2015-11-12.10:05:47.872>
    actor = 'serhiy.storchaka'
    assignee = 'none'
    closed = True
    closed_date = <Date 2015-11-12.10:05:47.873>
    closer = 'serhiy.storchaka'
    components = ['None']
    creation = <Date 2012-03-17.19:04:25.627>
    creator = 'Jakob.Bowyer'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 14350
    keywords = []
    message_count = 9.0
    messages = ['156189', '156227', '156229', '156239', '156329', '156721', '156746', '156779', '249560']
    nosy_count = 7.0
    nosy_names = ['rhettinger', 'terry.reedy', 'alexandre.vassalotti', 'r.david.murray', 'Jakob.Bowyer', 'Ramchandra Apte', 'serhiy.storchaka']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '22995'
    type = 'behavior'
    url = 'https://bugs.python.org/issue14350'
    versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

    @JakobBowyer
    Copy link
    Mannequin Author

    JakobBowyer mannequin commented Mar 17, 2012

    Running:
    Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
    Code:
    import copy
    copy.copy(iter([1,2,3]))
    Exception:

    ---------------------------------------------------------------------------

    TypeError                                 Traceback (most recent call last)
    T:\languages\Python27\Scripts\<ipython-input-2-4b0069a09ded> in <module>()
    ----> 1 copy.copy(iter([1,2,3]))

    T:\languages\Python27\lib\copy.pyc in copy(x)
    94 raise Error("un(shallow)copyable object of type %s" % cl
    s)
    95
    ---> 96 return _reconstruct(x, rv, 0)
    97
    98

    T:\languages\Python27\lib\copy.pyc in _reconstruct(x, info, deep, memo)
    327 if deep:
    328 args = deepcopy(args, memo)
    --> 329 y = callable(*args)
    330 memo[id(x)] = y
    331

    T:\languages\Python27\lib\copy_reg.pyc in __newobj__(cls, *args)
    91
    92 def __newobj__(cls, *args):
    ---> 93 return cls.__new__(cls, *args)
    94
    95 def _slotnames(cls):

    TypeError: object.__new__(listiterator) is not safe, use listiterator.__new__()

    Either this is a bug or not a clear error message in the exception

    @JakobBowyer JakobBowyer mannequin added the type-bug An unexpected behavior, bug, or error label Mar 17, 2012
    @RamchandraApte
    Copy link
    Mannequin

    RamchandraApte mannequin commented Mar 18, 2012

    I get a normal exception.
    I see ipython at the top level in 'T:\languages\Python27\Scripts\<ipython-input-2-4b0069a09ded> in <module>()'
    Perhaps you ran ipython accidentally?

    @bitdancer
    Copy link
    Member

    @ramchandra: I think you referring to the traceback format (which is indeed less useful than a normal Python traceback in the context of this tracker). The OP, however, is referring to the exception itself:

    TypeError: object.__new__(listiterator) is not safe, use listiterator.__new__()

    This is indeed a bit unexpected, though I don't know that copying iterators is actually supported.

    @JakobBowyer
    Copy link
    Mannequin Author

    JakobBowyer mannequin commented Mar 18, 2012

    C:\Users\Jakob>python -c "import copy; copy.copy(iter([1,2,3]))"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "T:\languages\Python27\lib\copy.py", line 96, in copy
        return _reconstruct(x, rv, 0)
      File "T:\languages\Python27\lib\copy.py", line 329, in _reconstruct
        y = callable(*args)
      File "T:\languages\Python27\lib\copy_reg.py", line 93, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: object.__new__(listiterator) is not safe, use listiterator.__new__()
    
    C:\Users\Jakob>python3 -c "import copy; copy.copy(iter([1,2,3]))"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "T:\languages\Python32\lib\copy.py", line 97, in copy
        return _reconstruct(x, rv, 0)
      File "T:\languages\Python32\lib\copy.py", line 285, in _reconstruct
        y = callable(*args)
      File "T:\languages\Python32\lib\copyreg.py", line 88, in __newobj__
        return cls.__new__(cls, *args)
    TypeError: object.__new__(list_iterator) is not safe, use list_iterator.__new__()

    Pure python traceback. Just for clarity.

    @RamchandraApte
    Copy link
    Mannequin

    RamchandraApte mannequin commented Mar 19, 2012

    BTW, can we add support for copying iterators by using itertools.tee

    @terryjreedy
    Copy link
    Member

    Title corrected. Non-iterator iterables are usually easy to copy.
    I think this is in the category of "don't do that" ;-)

    I believe the idea of making iterators copyable has been rejected on one of the lists because it is probably not possible in general.
    Tee consumes indefinite and unbounded space. Its use should be explicit. If a function needs to iterate more than once with an input iterable, it should explicitly do list(iterator) for iterator inputs.
    See Raymond's comments, such as msg156720 in bpo-14288.

    So I suspect that raising an error for copy.copy(iter([])) is correct. The error message amounts to saying make a new instance with the class constructor. That is usually correct when copy is not possible, but not in this case. ListIterator.__new__ is the inherited object.__new__. It appears that ListIterators can only be constructed by list.__iter__. But I doubt that the code that raises the error message can know that.
    But perhaps the error message can be improved anyway.

    @terryjreedy terryjreedy changed the title Strange Exception from copying an iterable Strange Exception from copying an iterator Mar 24, 2012
    @JakobBowyer
    Copy link
    Mannequin Author

    JakobBowyer mannequin commented Mar 25, 2012

    Before anyone else rushes of to do this, can I? I really want to break into python-dev and this might be my chance.

    @bitdancer
    Copy link
    Member

    Please, give it a try. But also be prepared for it being harder than it looks; the problem is that there may be very limited knowledge available where the error is generated. (I don't know; I haven't looked at the code and am not likely to...I usually leave reviews of C code to our C experts.)

    @serhiy-storchaka
    Copy link
    Member

    Actually the tp_new field of list iterator class is NULL. Unpickler raises other error in such case (see bpo-24900 for example).

    UnpicklingError: NEWOBJ class argument has NULL tp_new
    

    Backported to 2.7 the part of the patch for bpo-22995 fixes this issue.

    >>> copy.copy(iter([]))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/serhiy/py/cpython-2.7/Lib/copy.py", line 88, in copy
        rv = reductor(2)
    TypeError: can't pickle listiterator objects

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants