If a unittest is written which accesses a module written in C++ (I used Pybind11 to allow Python access) which uses malloc for a string, a segmentation fault is caused. This is not replicated when malloc is used for some other data types such as int, char or char*

C++ code:

#include <string>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
using namespace std;
void t(){
    string * data = (string*)malloc(100*sizeof(string));
    data[0] = "this is a test";
    cout<<data[0]<<endl;
}
PYBIND11_MODULE(TestModule, m){
    m.def("editPointerString", &t);
}

Once compiled and imported this can be run fine with editPointerString() as expected. No errors occur when running this function from any Python functions or from within a class.


However, if this is run through a unittest such as:


class TestUnit(unittest.TestCase):
    def testStringPointer(self):
        editPointerString()


A segmentation fault occurs before the cout<<data[0]<<endl; is executed. Not having an assert statement does not affect this outcome.


However, running 3 or more tests which call this function allows the tests to run and pass - instead displaying there has been a segmentation fault after the OK message from unittest

This can be fixed by using calloc instead of malloc, so may be caused by processing of data which is not cleared when malloc is used?

Also, malloc can be used in functions in tests after a test running calloc has been run.


System info:
Python 3.7.3
Ubuntu 19.04
pybind11 2.3.0
g++ 8.3.0
C++ 8.3.0

The issue looks like a bug in third party code, rather than a bug in Python itself. You can try to set PYTHONMALLOC=debug environment variable, it might help to detect a memory corruption. I strongly advice you to test a debug build of Python. Since Python 3.8, the ABI is now compatible in debug and release builds.

See also https://pythondev.readthedocs.io/debug_tools.html to debug a crash.

I close the issue.