New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
itertools.accumulate __reduce__/__setstate__ bug #69904
Comments
itertools.accumulate() has got methods __reduce__() and __setstate__() which fail at saving/restoring the state in all cases. Example: >>> a = itertools.accumulate([0.0, 42])
>>> next(a)
0.0
>>> b = copy.deepcopy(a)
>>> next(a)
42.0
>>> next(b)
42 # should have been the same as the previous result More precisely, the problem occurs when the C-level "total" field has some value whose truth-value is false. Then __reduce__/setstate will not restore the value of this field, but keep it NULL. That's why in the example above the intermediate total of 0.0 is forgotten, and the next(b) call will again just pick the next value without doing any addition. (The problem can hurt more than in this example if we use a custom function instead of addition.) A fix is easy but, as far as I can tell, it requires breaking the pickled format of accumulate() iterators that have already started. |
__reduce__() and __setstate__() methods of itertools.accumulate() look correct. The problem is not in itertools.accumulate(), but in the copy module. It uses the same __reduce__ protocol as pickle, but in different way. In the pickle module (in both Python and C implementations) state value is ignored only if it is None. In the copy module any state with boolean value is False is ignored. Proposed patch fixes the copy module. |
Ok, then with pickle you can have the same problem but only with None-vs-no-total. Here is an artificial example: import itertools, pickle
def foo(a, b):
print(a, b)
a = itertools.accumulate([3, 4, 5], foo)
next(a)
next(a) # prints: 3, 4
b = pickle.loads(pickle.dumps(a))
next(a) # prints: None, 5
next(b) # foo() is not called at all |
In this case yes, you are unlucky. But this is an artificial example, and I don't think this happens in any real code. However specific meaning of None should be documented (there are other wrong words about __getstate__). |
New changeset 095d21df9374 by Serhiy Storchaka in branch '3.4': New changeset 3d39e1e2dcb3 by Serhiy Storchaka in branch '2.7': New changeset fecb07050aae by Serhiy Storchaka in branch '3.5': New changeset 962086677306 by Serhiy Storchaka in branch 'default': |
The bug in the copy module is fixed. As for accumulate's __reduce__, I don't think we can do something. |
This could be fixed by saving the accumulate state in a tuple. It would break protocol, though. I don't recall the rules for backwards compatibility of pickles. |
Well, we can fix this without breaking the protocol. The accumulate() object with state is None can be reconstructed as islice(accumulate(chain([None], iterator)), 1, None). Proposed patch implements this solution. |
What would you say about the patch Kristján? |
Ping. |
Ping again. |
Raymond, could you please look at the patch? |
+1 |
New changeset f3c54cbac3de by Serhiy Storchaka in branch '3.5': New changeset 49c035b30e18 by Serhiy Storchaka in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: