classification
Title: Strange Exception from copying an iterator
Type: behavior Stage: resolved
Components: None Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Restrict default pickleability
View: 22995
Assigned To: Nosy List: Jakob.Bowyer, Ramchandra Apte, alexandre.vassalotti, r.david.murray, rhettinger, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2012-03-17 19:04 by Jakob.Bowyer, last changed 2015-11-12 10:05 by serhiy.storchaka. This issue is now closed.

Messages (9)
msg156189 - (view) Author: Jakob Bowyer (Jakob.Bowyer) Date: 2012-03-17 19:04
Running:
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)]
Code:
import copy
copy.copy(iter([1,2,3]))
Exception:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
T:\languages\Python27\Scripts\<ipython-input-2-4b0069a09ded> in <module>()
----> 1 copy.copy(iter([1,2,3]))

T:\languages\Python27\lib\copy.pyc in copy(x)
     94                 raise Error("un(shallow)copyable object of type %s" % cl
s)
     95
---> 96     return _reconstruct(x, rv, 0)
     97
     98

T:\languages\Python27\lib\copy.pyc in _reconstruct(x, info, deep, memo)
    327     if deep:
    328         args = deepcopy(args, memo)
--> 329     y = callable(*args)
    330     memo[id(x)] = y
    331

T:\languages\Python27\lib\copy_reg.pyc in __newobj__(cls, *args)
     91
     92 def __newobj__(cls, *args):
---> 93     return cls.__new__(cls, *args)
     94
     95 def _slotnames(cls):

TypeError: object.__new__(listiterator) is not safe, use listiterator.__new__()

Either this is a bug or not a clear error message in the exception
msg156227 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-03-18 02:39
I get a normal exception.
I see ipython at the top level in 'T:\languages\Python27\Scripts\<ipython-input-2-4b0069a09ded> in <module>()'
Perhaps you ran ipython accidentally?
msg156229 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-18 03:26
@Ramchandra: I think you referring to the traceback format (which is indeed less useful than a normal Python traceback in the context of this tracker).  The OP, however, is referring to the exception itself:

  TypeError: object.__new__(listiterator) is not safe, use listiterator.__new__()

This is indeed a bit unexpected, though I don't know that copying iterators is actually supported.
msg156239 - (view) Author: Jakob Bowyer (Jakob.Bowyer) Date: 2012-03-18 09:57
C:\Users\Jakob>python -c "import copy; copy.copy(iter([1,2,3]))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "T:\languages\Python27\lib\copy.py", line 96, in copy
    return _reconstruct(x, rv, 0)
  File "T:\languages\Python27\lib\copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "T:\languages\Python27\lib\copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(listiterator) is not safe, use listiterator.__new__()

C:\Users\Jakob>python3 -c "import copy; copy.copy(iter([1,2,3]))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "T:\languages\Python32\lib\copy.py", line 97, in copy
    return _reconstruct(x, rv, 0)
  File "T:\languages\Python32\lib\copy.py", line 285, in _reconstruct
    y = callable(*args)
  File "T:\languages\Python32\lib\copyreg.py", line 88, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(list_iterator) is not safe, use list_iterator.__new__()

Pure python traceback. Just for clarity.
msg156329 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2012-03-19 11:59
BTW, can we add support for copying iterators by using itertools.tee
msg156721 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-03-24 23:23
Title corrected. Non-iterator iterables are usually easy to copy.
I think this is in the category of "don't do that" ;-)

I believe the idea of making iterators copyable has been rejected on one of the lists because it is probably not possible in general.
Tee consumes indefinite and unbounded space. Its use should be explicit. If a function needs to iterate more than once with an input iterable, it should explicitly do list(iterator) for iterator inputs.
See Raymond's comments, such as msg156720 in #14288.

So I suspect that raising an error for copy.copy(iter([])) is correct. The error message amounts to saying make a new instance with the class constructor. That is usually correct when copy is not possible, but not in this case. ListIterator.__new__ is the inherited object.__new__. It appears that ListIterators can only be constructed by list.__iter__. But I doubt that the code that raises the error message can know that.
But perhaps the error message can be improved anyway.
msg156746 - (view) Author: Jakob Bowyer (Jakob.Bowyer) Date: 2012-03-25 09:09
Before anyone else rushes of to do this, can I? I really want to break into python-dev and this might be my chance.
msg156779 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-25 22:58
Please, give it a try.  But also be prepared for it being harder than it looks; the problem is that there may be very limited knowledge available where the error is generated. (I don't know; I haven't looked at the code and am not likely to...I usually leave reviews of C code to our C experts.)
msg249560 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-09-02 18:59
Actually the tp_new field of list iterator class is NULL. Unpickler raises other error in such case (see issue24900 for example).

    UnpicklingError: NEWOBJ class argument has NULL tp_new

Backported to 2.7 the part of the patch for issue22995 fixes this issue.

>>> copy.copy(iter([]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-2.7/Lib/copy.py", line 88, in copy
    rv = reductor(2)
TypeError: can't pickle listiterator objects
History
Date User Action Args
2015-11-12 10:05:47serhiy.storchakasetstatus: open -> closed
superseder: Restrict default pickleability
dependencies: - Restrict default pickleability
resolution: duplicate
stage: needs patch -> resolved
2015-09-02 18:59:18serhiy.storchakasetnosy: + serhiy.storchaka
dependencies: + Restrict default pickleability
messages: + msg249560
2012-03-25 22:58:17r.david.murraysetmessages: + msg156779
2012-03-25 09:09:38Jakob.Bowyersetmessages: + msg156746
2012-03-24 23:23:26terry.reedysetnosy: + terry.reedy, alexandre.vassalotti, rhettinger
title: Strange Exception from copying an iterable -> Strange Exception from copying an iterator
messages: + msg156721

stage: needs patch
2012-03-19 11:59:39Ramchandra Aptesetmessages: + msg156329
2012-03-18 09:57:04Jakob.Bowyersetmessages: + msg156239
2012-03-18 03:26:47r.david.murraysetversions: + Python 3.2, Python 3.3
2012-03-18 03:26:34r.david.murraysetnosy: + r.david.murray
messages: + msg156229
2012-03-18 02:39:44Ramchandra Aptesetnosy: + Ramchandra Apte
messages: + msg156227
2012-03-17 19:04:25Jakob.Bowyercreate