classification
Title: OrderedDict iterators are exhausted during pickling
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: josh.r, miss-islington, serhiy.storchaka, sir-sigurd, xtreak
Priority: normal Keywords: patch

Created on 2018-09-04 04:19 by sir-sigurd, last changed 2018-10-20 06:06 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9051 merged sir-sigurd, 2018-09-04 04:38
PR 9996 merged miss-islington, 2018-10-20 05:21
PR 9997 merged miss-islington, 2018-10-20 05:21
Messages (7)
msg324551 - (view) Author: Sergey Fedoseev (sir-sigurd) * Date: 2018-09-04 04:19
In [1]: from collections import OrderedDict

In [2]: od = OrderedDict.fromkeys(range(10))

In [3]: it = iter(od)

In [4]: it.__reduce__()
Out[4]: (<function iter>, ([0, 1, 2, 3, 4, 5, 6, 7, 8, 9],))

In [5]: list(it)
Out[5]: []
msg324650 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2018-09-05 20:05
This would presumably be a side-effect of all generic pickling operations of iterators; figuring out what the iterator produces requires running out the iterator. You could special case it case-by-case, but that just makes the behavior unreliable/confusing; now some iterators pickle without being mutated, and others don't. Do you have a proposal to fix it? Is it something that needs to be fixed at all, when the option to pickle the original OrderedDict directly is there?
msg324664 - (view) Author: Sergey Fedoseev (sir-sigurd) * Date: 2018-09-06 06:40
Other iterators either don't support pickling or aren't mutated during pickling:

In [10]: it = iter({1})

In [11]: pickle.dumps(it)
Out[11]: b'\x80\x04\x95\x1e\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x04iter\x94\x93\x94]\x94K\x01a\x85\x94R\x94.'

In [12]: list(it)
Out[12]: [1]

In [13]: it = (x for x in {1})

In [14]: pickle.dumps(it)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-7e86a183a31c> in <module>()
----> 1 pickle.dumps(it)

TypeError: can't pickle generator objects
msg328124 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-20 05:20
New changeset a5259fb05d03f4871837c14fed704541a20896c0 by Serhiy Storchaka (Sergey Fedoseev) in branch 'master':
bpo-34574: Prevent OrderedDict iterators from exhaustion during pickling. (GH-9051)
https://github.com/python/cpython/commit/a5259fb05d03f4871837c14fed704541a20896c0
msg328126 - (view) Author: miss-islington (miss-islington) Date: 2018-10-20 05:54
New changeset dcd56f615e89d4920a0598a9c3d3301701f238a6 by Miss Islington (bot) in branch '3.7':
bpo-34574: Prevent OrderedDict iterators from exhaustion during pickling. (GH-9051)
https://github.com/python/cpython/commit/dcd56f615e89d4920a0598a9c3d3301701f238a6
msg328127 - (view) Author: miss-islington (miss-islington) Date: 2018-10-20 06:05
New changeset 0d3dd9fe0d2565f09f70d8ea7341dfd88e6bd380 by Miss Islington (bot) in branch '3.6':
bpo-34574: Prevent OrderedDict iterators from exhaustion during pickling. (GH-9051)
https://github.com/python/cpython/commit/0d3dd9fe0d2565f09f70d8ea7341dfd88e6bd380
msg328128 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-20 06:06
Thank your Sergey for your report and fix.
History
Date User Action Args
2018-10-20 06:06:46serhiy.storchakasetstatus: open -> closed
versions: + Python 3.6, Python 3.7, Python 3.8
messages: + msg328128

components: + Extension Modules
resolution: fixed
stage: patch review -> resolved
2018-10-20 06:05:24miss-islingtonsetmessages: + msg328127
2018-10-20 05:54:13miss-islingtonsetnosy: + miss-islington
messages: + msg328126
2018-10-20 05:21:11miss-islingtonsetpull_requests: + pull_request9339
2018-10-20 05:21:00miss-islingtonsetpull_requests: + pull_request9338
2018-10-20 05:20:46serhiy.storchakasetmessages: + msg328124
2018-09-08 12:32:21serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
2018-09-06 06:51:16xtreaksetnosy: + xtreak
2018-09-06 06:40:38sir-sigurdsetmessages: + msg324664
2018-09-05 20:05:08josh.rsetnosy: + josh.r
messages: + msg324650
2018-09-04 04:38:03sir-sigurdsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8513
2018-09-04 04:19:39sir-sigurdcreate