Issue38770
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-11-11 21:10 by Saim Raza, last changed 2022-04-11 14:59 by admin.
Messages (3) | |||
---|---|---|---|
msg356388 - (view) | Author: Saim Raza (Saim Raza) | Date: 2019-11-11 21:10 | |
If the __qualname__ of a class is set to have a circular reference to itself, pickle behaves differently based on protocol. Following script demonstrates the issue: ====================================================== from __future__ import print_function import pickle, sys class Foo: __name__ = __qualname__ = "Foo.ref" pass Foo.ref = Foo print(sys.version_info) for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): print("{}:".format(proto), end=" ") try: pkl = pickle.dumps(Foo, proto) print("Dump OK,", end=" ") assert(pickle.loads(pkl) is Foo) print("Load OK,") except Exception as err: print(repr(err)) ====================================================== OUTPUT: Python2.7: sys.version_info(major=2, minor=7, micro=16, releaselevel='final', serial=0) 0: Dump OK, Load OK, 1: Dump OK, Load OK, 2: Dump OK, Load OK, Python3.7: sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0) 0: RecursionError('maximum recursion depth exceeded while pickling an object') 1: RecursionError('maximum recursion depth exceeded while pickling an object') 2: RecursionError('maximum recursion depth exceeded while pickling an object') 3: RecursionError('maximum recursion depth exceeded while pickling an object') 4: Dump OK, Load OK, ====================================================== This was introduced as a side effect of issue#23611 (?). I can think of the following approaches to fix the issue and make the behavior consistent: 1. Check if the class has a self-reference and raise an error for all protocols. 2. Use memoization to handle self-references. I am not sure what should be dumped in this case. In the example above `Foo` will exist in the namespace but not `Foo.ref`. 3. Dump such classes similar to Python 2 pickle and Python 3 pickle protocol >= 4. I had a stab at pickle.py and had a bit of success in doing point 3 above. Posting this issue for discussions. I would be happy to submit a PR for this issue. Thanks, Saim Raza |
|||
msg366951 - (view) | Author: Furkan Onder (furkanonder) * | Date: 2020-04-21 23:19 | |
I ran your script and didn't get RecursionError. The issue seems to be fixed. Python 3.8.2 (default, Apr 8 2020, 14:31:25) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from __future__ import print_function >>> >>> import pickle, sys >>> >>> class Foo: ... __name__ = __qualname__ = "Foo.ref" ... pass ... >>> Foo.ref = Foo >>> >>> print(sys.version_info) sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0) >>> for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): ... print("{}:".format(proto), end=" ") ... try: ... pkl = pickle.dumps(Foo, proto) ... print("Dump OK,", end=" ") ... assert(pickle.loads(pkl) is Foo) ... print("Load OK,") ... except Exception as err: ... print(repr(err)) ... 0: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed") 1: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed") 2: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed") 3: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed") 4: Dump OK, Load OK, 5: Dump OK, Load OK, >>> |
|||
msg366956 - (view) | Author: Furkan Onder (furkanonder) * | Date: 2020-04-21 23:57 | |
Ahh. I misunderstood the problem. Pickle behaves differently when it is a circular reference. If you have a solution, I am waiting with curiosity. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:23 | admin | set | github: 82951 |
2020-04-21 23:57:29 | furkanonder | set | messages: + msg366956 |
2020-04-21 23:19:10 | furkanonder | set | nosy:
+ furkanonder messages: + msg366951 |
2019-11-11 21:36:07 | serhiy.storchaka | set | assignee: serhiy.storchaka |
2019-11-11 21:10:17 | Saim Raza | create |