Message248677
Original Raymonds reason in msg248662 is not valid. Pickling a cycle object that fully consumed its input iterable is already space-inefficient.
>>> import itertools, pickle, pickletools
>>> c = itertools.cycle(iter('abcde'))
>>> [next(c) for i in range(8)]
['a', 'b', 'c', 'd', 'e', 'a', 'b', 'c']
>>> pickle.dumps(c)
b'\x80\x03citertools\ncycle\nq\x00cbuiltins\niter\nq\x01]q\x02(X\x01\x00\x00\x00aq\x03X\x01\x00\x00\x00bq\x04X\x01\x00\x00\x00cq\x05X\x01\x00\x00\x00dq\x06X\x01\x00\x00\x00eq\x07e\x85q\x08Rq\tK\x03b\x85q\nRq\x0bh\x02K\x01\x86q\x0cb.'
>>> pickletools.dis(pickle.dumps(c))
0: \x80 PROTO 3
2: c GLOBAL 'itertools cycle'
19: q BINPUT 0
21: c GLOBAL 'builtins iter'
36: q BINPUT 1
38: ] EMPTY_LIST
39: q BINPUT 2
41: ( MARK
42: X BINUNICODE 'a'
48: q BINPUT 3
50: X BINUNICODE 'b'
56: q BINPUT 4
58: X BINUNICODE 'c'
64: q BINPUT 5
66: X BINUNICODE 'd'
72: q BINPUT 6
74: X BINUNICODE 'e'
80: q BINPUT 7
82: e APPENDS (MARK at 41)
83: \x85 TUPLE1
84: q BINPUT 8
86: R REDUCE
87: q BINPUT 9
89: K BININT1 3
91: b BUILD
92: \x85 TUPLE1
93: q BINPUT 10
95: R REDUCE
96: q BINPUT 11
98: h BINGET 2
100: K BININT1 1
102: \x86 TUPLE2
103: q BINPUT 12
105: b BUILD
106: . STOP
highest protocol among opcodes = 2
An internal iterator is not pickled as iter("de"), but as an iterator of the list ["a", "b", "c", "d", "e"] with 3 items consumed. This list also saved as a part of a cycle object state, but not as a copy, but as a reference.
There are two alternative patches. Both keep Raymonds optimization of cycle iterating, but have advantages. cycle_reduce_2.patch makes __reduce__ faster and more memory efficient than Raymonds variant. cycle_reduce_3.patch makes unpickled cycle object so optimized as original. |
|
Date |
User |
Action |
Args |
2015-08-16 08:44:11 | serhiy.storchaka | set | recipients:
+ serhiy.storchaka, rhettinger, pitrou, alexandre.vassalotti |
2015-08-16 08:44:10 | serhiy.storchaka | set | messageid: <1439714650.99.0.526029405196.issue24874@psf.upfronthosting.co.za> |
2015-08-16 08:44:10 | serhiy.storchaka | link | issue24874 messages |
2015-08-16 08:44:10 | serhiy.storchaka | create | |
|