classification
Title: Pickle failure is raising AttributeError and not PicklingError
Type: behavior Stage: patch review
Components: Extension Modules Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Matt.Dodge, alexandre.vassalotti, iritkatriel, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-01-06 23:06 by Matt.Dodge, last changed 2021-02-17 19:39 by iritkatriel.

Messages (3)
msg284869 - (view) Author: Matt Dodge (Matt.Dodge) Date: 2017-01-06 23:06
When failing to pickle something (like a locally scoped class) the documentation indicates that a PicklingError should be raised.
Doc links:
 - https://docs.python.org/3/library/pickle.html?highlight=pickle#pickle-picklable
 - https://docs.python.org/3.5/library/pickle.html#what-can-be-pickled-and-unpickled

However, instead I'm seeing AttributeError get raised instead, starting in Python 3.5. In Python 3.4 PicklingErrror was raised, as expected.

To reproduce, use the following file <pickletest.py>
    def func():
        class C: pass
        return C
    import pickle
    pickle.dumps(func()())

In Python 3.4 you see:
Traceback (most recent call last):
  File "pickletest.py", line 5, in <module>
    pickle.dumps(func()())
_pickle.PicklingError: Can't pickle <class '__main__.func.<locals>.C'>: attribute lookup C on __main__ failed

But in 3.5/3.6 you see:
Traceback (most recent call last):
  File "pickletest.py", line 5, in <module>
    pickle.dumps(func()())
AttributeError: Can't pickle local object 'func.<locals>.C'

I don't necessarily mind that a different exception is being raised, but how should we be handling exceptions while pickling? Catch all exceptions? That doesn't feel right to me, but if we're trying to pickle data out of our control I'm not sure what else to do.

FYI, the UnpicklingError documentation (https://docs.python.org/3/library/pickle.html?highlight=pickle#pickle.UnpicklingError) indicates that other exceptions can possibly be raised during unpickling. I assume that is more related to the fact that the pickled data may not fit into the current class/module structure though, so I think it's unrelated.
msg284896 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-07 07:15
Python implementation of pickle still raises PicklingError. Seems this was not intentional change.

>>> pickle._dumps(func()())
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 918, in save_global
    obj2, parent = _getattribute(module, name)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 266, in _getattribute
    .format(name, obj))
AttributeError: Can't get local attribute 'func.<locals>.C' on <function func at 0xb7118d1c>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 1544, in _dumps
    _Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 409, in dump
    self.save(obj)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 605, in save_reduce
    save(cls)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 978, in save_type
    return self.save_global(obj)
  File "/home/serhiy/py/cpython/Lib/pickle.py", line 922, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class '__main__.func.<locals>.C'>: it's not found as __main__.func.<locals>.C
msg387185 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-02-17 19:39
Still the same in 3.10:

Python 3.10.0a5+ (heads/bpo-43146-dirty:8f5cf4d381, Feb 17 2021, 14:51:27) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
...     def func():
...         class C: pass
...         return C
...     import pickle
...     pickle.dumps(func()())
... except BaseException as e:
...   exc = e
...
>>> exc
AttributeError("Can't pickle local object 'func.<locals>.C'")
>>>
History
Date User Action Args
2021-02-17 19:39:09iritkatrielsetnosy: + iritkatriel

messages: + msg387185
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.5, Python 3.6, Python 3.7
2017-01-07 07:15:10serhiy.storchakasettype: behavior
components: + Extension Modules, - Library (Lib)
versions: + Python 3.7, - Python 3.4
nosy: + alexandre.vassalotti, serhiy.storchaka

messages: + msg284896
stage: patch review
2017-01-06 23:06:54Matt.Dodgecreate