This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Pickle handle self references in classes
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Saim Raza, furkanonder, serhiy.storchaka
Priority: normal Keywords:

Created on 2019-11-11 21:10 by Saim Raza, last changed 2022-04-11 14:59 by admin.

Messages (3)
msg356388 - (view) Author: Saim Raza (Saim Raza) Date: 2019-11-11 21:10
If the __qualname__ of a class is set to have a circular reference to itself, pickle behaves differently based on protocol. Following script demonstrates the issue:

======================================================
from __future__ import print_function

import pickle, sys

class Foo:
    __name__ = __qualname__ = "Foo.ref"
    pass

Foo.ref = Foo

print(sys.version_info)

for proto in range(0, pickle.HIGHEST_PROTOCOL + 1):
    print("{}:".format(proto), end=" ")
    try:
        pkl = pickle.dumps(Foo, proto)
        print("Dump OK,", end=" ")
        assert(pickle.loads(pkl) is Foo)
        print("Load OK,")
    except Exception as err:
        print(repr(err))
======================================================
OUTPUT:
Python2.7:
sys.version_info(major=2, minor=7, micro=16, releaselevel='final', serial=0)
0: Dump OK, Load OK,
1: Dump OK, Load OK,
2: Dump OK, Load OK,

Python3.7:
sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
0: RecursionError('maximum recursion depth exceeded while pickling an object')
1: RecursionError('maximum recursion depth exceeded while pickling an object')
2: RecursionError('maximum recursion depth exceeded while pickling an object')
3: RecursionError('maximum recursion depth exceeded while pickling an object')
4: Dump OK, Load OK,
======================================================

This was introduced as a side effect of issue#23611 (?). I can think of the following approaches to fix the issue and make the behavior consistent:

1. Check if the class has a self-reference and raise an error for all protocols.
2. Use memoization to handle self-references. I am not sure what should be dumped in this case. In the example above `Foo` will exist in the namespace but not `Foo.ref`. 
3. Dump such classes similar to Python 2 pickle and Python 3 pickle protocol >= 4.

I had a stab at pickle.py and had a bit of success in doing point 3 above. Posting this issue for discussions. I would be happy to submit a PR for this issue.

Thanks,
Saim Raza
msg366951 - (view) Author: Furkan Onder (furkanonder) * Date: 2020-04-21 23:19
I ran your script and didn't get RecursionError. The issue seems to be fixed.

Python 3.8.2 (default, Apr  8 2020, 14:31:25) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> 
>>> import pickle, sys
>>> 
>>> class Foo:
...     __name__ = __qualname__ = "Foo.ref"
...     pass
... 
>>> Foo.ref = Foo
>>> 
>>> print(sys.version_info)
sys.version_info(major=3, minor=8, micro=2, releaselevel='final', serial=0)
>>> for proto in range(0, pickle.HIGHEST_PROTOCOL + 1):
...     print("{}:".format(proto), end=" ")
...     try:
...         pkl = pickle.dumps(Foo, proto)
...         print("Dump OK,", end=" ")
...         assert(pickle.loads(pkl) is Foo)
...         print("Load OK,")
...     except Exception as err:
...         print(repr(err))
... 
0: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
1: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
2: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
3: PicklingError("Can't pickle <class '__main__.Foo.ref'>: import of module '__main__' failed")
4: Dump OK, Load OK,
5: Dump OK, Load OK,
>>>
msg366956 - (view) Author: Furkan Onder (furkanonder) * Date: 2020-04-21 23:57
Ahh. I misunderstood the problem. Pickle behaves differently when it is a circular reference. If you have a solution, I am waiting with curiosity.
History
Date User Action Args
2022-04-11 14:59:23adminsetgithub: 82951
2020-04-21 23:57:29furkanondersetmessages: + msg366956
2020-04-21 23:19:10furkanondersetnosy: + furkanonder
messages: + msg366951
2019-11-11 21:36:07serhiy.storchakasetassignee: serhiy.storchaka
2019-11-11 21:10:17Saim Razacreate