This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Pickle can't import builtins at exit
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Wicken, christian.heimes, gvanrossum
Priority: normal Keywords:

Created on 2021-01-15 12:05 by Wicken, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (4)
msg385110 - (view) Author: Pete Wicken (Wicken) * Date: 2021-01-15 12:05
Originally found as an issue in Lib/shelve.py; if we attempt to pickle a builtin as the program is exiting then Modules/_pickle.c will fail at the point of the PyImport_Import in save_global. In CPython3.8 this causes a segfault, in CPython3.9 a PicklingError is raised.

This is especially problematic in shelve.py as object pickling is attempted by the __del__ method's call stack when writeback=True. Therefore if the program exits before an explicit sync is called; in 3.8 the data will not be written to disk and a segfault occurs; in 3.9 the data is written to disk, but with an uncaught exception being raised.


Exception demonstrated via shelve on 3.9.1 on MacOS with Clang:

Python 3.9.1 (default, Dec 10 2020, 11:11:14)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> s = shelve.open('testing', writeback=True)
>>> s['a'] = Exception
>>> exit()
Exception ignored in: <function Shelf.__del__ at 0x10ed67790>
Traceback (most recent call last):
  File ".../3.9/lib/python3.9/shelve.py", line 162, in __del__
  File ".../3.9/lib/python3.9/shelve.py", line 144, in close
  File ".../3.9/lib/python3.9/shelve.py", line 168, in sync
  File ".../3.9/lib/python3.9/shelve.py", line 124, in __setitem__
_pickle.PicklingError: Can't pickle <class 'Exception'>: import of module 'builtins' failed


Segfault demonstrated via shelve on 3.8.5 on MacOS with Clang (different system from above):

Python 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> s = shelve.open('testing', writeback=True)
>>> s['a'] = Exception
>>> exit()
[1]    10040 segmentation fault  python3.8


Exception demonstrated via shelve on 3.9.1 on RHEL with GCC:

Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> s = shelve.open("thing", writeback=True)
>>> s['a'] = Exception
>>> 
Exception ignored in: <function Shelf.__del__ at 0x7f3f30b7c820>
Traceback (most recent call last):
  File "/usr/lib64/python3.9/shelve.py", line 162, in __del__
  File "/usr/lib64/python3.9/shelve.py", line 144, in close
  File "/usr/lib64/python3.9/shelve.py", line 168, in sync
  File "/usr/lib64/python3.9/shelve.py", line 124, in __setitem__
_pickle.PicklingError: Can't pickle <class 'Exception'>: import of module 'builtins' failed


Code example to reproduce using Pickle in the class __del__, demonstrated on a RHEL system:

Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pickle import DEFAULT_PROTOCOL, Pickler
>>> from io import BytesIO
>>> class T:
...   def __del__(self):
...     f = BytesIO()
...     p = Pickler(f, DEFAULT_PROTOCOL)
...     p.dump(sum)
... 
>>> t = T()
>>> exit()
Exception ignored in: <function T.__del__ at 0x7f5f04d9ef70>
Traceback (most recent call last):
  File "<stdin>", line 5, in __del__
_pickle.PicklingError: Can't pickle <built-in function sum>: import of module 'builtins' failed


Have not tested on 3.6, 3.7 or 3.10.
msg385130 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-01-16 00:41
I don’t know there is much we can do about this. I recommend using ‘with’ to close the shelf earlier.
msg385140 - (view) Author: Pete Wicken (Wicken) * Date: 2021-01-16 13:38
Out of curiosity, why is there not much we can do? I'm not familiar enough with Python's C code to appreciate the difficulty of making the builtins available at the point where the pickle save is run when exiting. The fact that this segfaults in the 3.8 version tested was rather concerning.

Shelve's implementation without the context manager was how we found the original bug; but it feels the way pickle handles this should be a lot safer.

Some clarification might help me craft a warning for the documentation if there is nothing that can be done in the code base.
msg385149 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-16 20:51
At the end of a Python process, the interpreter is shut down in multiple steps. Object finalizers such as __del__ may be executed late in the interpreter shut down process. In your case, most of the interpreter is already gone. There isn't anything we can do to solve the problem.

That's why we recommend explicit resource management with the "with" statement. The atexit module is another way to execute cleanup hooks before the interpreter is shut down.
History
Date User Action Args
2022-04-11 14:59:40adminsetgithub: 87101
2021-10-18 16:28:51iritkatrielsetstatus: open -> closed
resolution: wont fix
stage: resolved
2021-01-16 20:51:41christian.heimessetnosy: + christian.heimes
messages: + msg385149
2021-01-16 13:38:40Wickensetmessages: + msg385140
2021-01-16 00:41:51gvanrossumsetnosy: + gvanrossum
messages: + msg385130
2021-01-15 12:05:38Wickencreate