This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Segfault when using pickle with exceptions and dynamic class inheritance
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: gabrielc, iritkatriel, pitrou
Priority: normal Keywords:

Created on 2019-08-31 23:39 by gabrielc, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (3)
msg350932 - (view) Author: Gabriel C (gabrielc) Date: 2019-08-31 23:39
The following results in a segfault on 3.7.4 (running on macOS high sierra) and 3.5 (running on ubuntu 16.04). It does not happen in python 2.7.

The problem seems to involve two effects. The first is the creation of a class with a dynamic type that inherits from a MemoryError (but any exception will work). The second is pickling an instance of that MemoryError (note that the dynamic type is never pickled).

If a basic (non-exception) class is used instead of the MemoryError (see uncomment case 1), then the segfault does not occur in my testing. If the exception being pickled is a different type to the one used as the base class for the dynamic type, the segfault does not occur. 

Note that unpickling the MemoryError actually raises another exception, this raised exception (see uncomment case 2) is raised by the dynamic type class __init__ method. It seems that in trying to unpickle the MemoryError, pickle attempts to create an instance of the unrelated dynamic type instead. Examining the stack trace (uncomment case 3), shows the raised exception is indeed originating from attempting to unpickle the MemoryError.

The segfault does not happen immediately, but instead after several attempts. It can happen after as few as 5 attempts or as many as 100. 

```
import pickle

def CreateDynamicClass(basetype):
    class DynamicClassImpl(basetype):
        def __init__(self):
            super(DynamicClassImpl, self).__init__()

    return DynamicClassImpl()

class TestClass(object):
    pass

N_attemps = 1000

pickle_list = []

def load_list():
    for i in range(N_attemps):
        test_exception = MemoryError("Test" + str(i))
        #test_exception = TestClass() # Uncomment case 1
        pickle_list.append(pickle.dumps(test_exception))
load_list()

def unload_list():
    for i in range(N_attemps):
        try:
            test_object_instance = pickle.loads(pickle_list.pop())
            test_dynamic_object = CreateDynamicClass(MemoryError)
            #test_dynamic_object = CreateDynamicClass(TestClass) # Uncomment case 1
        except Exception as e:
            print("Exception at iteration {}: {}".format(i, e)) # Uncomment case 2
            #raise # Uncomment case 3
            pass
unload_list()
```
msg351547 - (view) Author: Gabriel C (gabrielc) Date: 2019-09-09 19:01
Some further investigation suggests this may have nothing to do with pickle at all.

Consider the following short example:
```
def CreateDynamicClass(basetype):
    class DynamicClassImpl(basetype):
        def __init__(self):
            super(DynamicClassImpl, self).__init__()
    
    return DynamicClassImpl()

# Comment out any of the following three lines and the test passes
dynamic_memerror = CreateDynamicClass(MemoryError)
memory_error = MemoryError("Test")
dynamic_memerror = CreateDynamicClass(MemoryError)

print(MemoryError("Test2"))
```

This reliably produces the following stack trace:
```
Traceback (most recent call last):
  File "test_min2.py", line 13, in <module>
    print(MemoryError("Test2"))
TypeError: __init__() takes 1 positional argument but 2 were given
```
In addition it causes a bus error (signal 10) around 30% of the time when run on python 3.7.4 (macOS high sierra) and a segmentation fault around 10% of the time when run on python 3.5 (Ubuntu 16.04).

Now modify the short example as follows, inserting a dummy argument into the constructor of the dynamic exception type:
```
def CreateDynamicClass(basetype):
    class DynamicClassImpl(basetype):
        def __init__(self, dummy_arg):
            if dummy_arg is not None:
                raise AssertionError("Dynamic exception constructor called.")
            super(DynamicClassImpl, self).__init__()
    
    return DynamicClassImpl(None)

# Comment out any of the following three lines and the test passes
dynamic_memerror = CreateDynamicClass(MemoryError)
memory_error = MemoryError("Test")
dynamic_memerror = CreateDynamicClass(MemoryError)

print(MemoryError("Test2"))
```

This produces the following stack trace:
```
Traceback (most recent call last):
  File "test_min2.py", line 15, in <module>
    print(MemoryError("Test2"))
  File "test_min2.py", line 5, in __init__
    raise AssertionError("Dynamic exception constructor called.")
AssertionError: Dynamic exception constructor called.
```
showing that the user-defined type constructor is actually called when the MemoryError object is created.
msg401240 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-09-07 11:19
I am unable to reproduce the crash with either example on 3.9 or 3.11 on a Mac. Please create a new issue if you still see a problem on a current version (>= 3.9).
History
Date User Action Args
2022-04-11 14:59:19adminsetgithub: 82178
2021-09-07 11:19:11iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg401240

resolution: out of date
stage: resolved
2019-09-09 19:01:56gabrielcsetmessages: + msg351547
2019-09-01 02:29:05xtreaksetnosy: + pitrou
2019-08-31 23:39:17gabrielccreate