This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author serhiy.storchaka
Recipients alexandre.vassalotti, pitrou, rhettinger, serhiy.storchaka
Date 2015-08-16.08:44:10
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1439714650.99.0.526029405196.issue24874@psf.upfronthosting.co.za>
In-reply-to
Content
Original Raymonds reason in msg248662 is not valid. Pickling a cycle object that fully consumed its input iterable is already space-inefficient.

>>> import itertools, pickle, pickletools
>>> c = itertools.cycle(iter('abcde'))
>>> [next(c) for i in range(8)]
['a', 'b', 'c', 'd', 'e', 'a', 'b', 'c']
>>> pickle.dumps(c)
b'\x80\x03citertools\ncycle\nq\x00cbuiltins\niter\nq\x01]q\x02(X\x01\x00\x00\x00aq\x03X\x01\x00\x00\x00bq\x04X\x01\x00\x00\x00cq\x05X\x01\x00\x00\x00dq\x06X\x01\x00\x00\x00eq\x07e\x85q\x08Rq\tK\x03b\x85q\nRq\x0bh\x02K\x01\x86q\x0cb.'
>>> pickletools.dis(pickle.dumps(c))
    0: \x80 PROTO      3
    2: c    GLOBAL     'itertools cycle'
   19: q    BINPUT     0
   21: c    GLOBAL     'builtins iter'
   36: q    BINPUT     1
   38: ]    EMPTY_LIST
   39: q    BINPUT     2
   41: (    MARK
   42: X        BINUNICODE 'a'
   48: q        BINPUT     3
   50: X        BINUNICODE 'b'
   56: q        BINPUT     4
   58: X        BINUNICODE 'c'
   64: q        BINPUT     5
   66: X        BINUNICODE 'd'
   72: q        BINPUT     6
   74: X        BINUNICODE 'e'
   80: q        BINPUT     7
   82: e        APPENDS    (MARK at 41)
   83: \x85 TUPLE1
   84: q    BINPUT     8
   86: R    REDUCE
   87: q    BINPUT     9
   89: K    BININT1    3
   91: b    BUILD
   92: \x85 TUPLE1
   93: q    BINPUT     10
   95: R    REDUCE
   96: q    BINPUT     11
   98: h    BINGET     2
  100: K    BININT1    1
  102: \x86 TUPLE2
  103: q    BINPUT     12
  105: b    BUILD
  106: .    STOP
highest protocol among opcodes = 2

An internal iterator is not pickled as iter("de"), but as an iterator of the list ["a", "b", "c", "d", "e"] with 3 items consumed. This list also saved as a part of a cycle object state, but not as a copy, but as a reference.

There are two alternative patches. Both keep Raymonds optimization of cycle iterating, but have advantages. cycle_reduce_2.patch makes __reduce__ faster and more memory efficient than Raymonds variant. cycle_reduce_3.patch makes unpickled cycle object so optimized as original.
History
Date User Action Args
2015-08-16 08:44:11serhiy.storchakasetrecipients: + serhiy.storchaka, rhettinger, pitrou, alexandre.vassalotti
2015-08-16 08:44:10serhiy.storchakasetmessageid: <1439714650.99.0.526029405196.issue24874@psf.upfronthosting.co.za>
2015-08-16 08:44:10serhiy.storchakalinkissue24874 messages
2015-08-16 08:44:10serhiy.storchakacreate