classification
Title: unittest causing segfaults with string malloc in c++ module
Type: crash Stage: resolved
Components: Extension Modules, Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: vstinner, ymerej
Priority: normal Keywords:

Created on 2019-09-06 10:11 by ymerej, last changed 2020-03-26 00:41 by vstinner. This issue is now closed.

Messages (2)
msg351243 - (view) Author: Jeremy (ymerej) Date: 2019-09-06 10:11
If a unittest is written which accesses a module written in C++ (I used Pybind11 to allow Python access) which uses malloc for a string, a segmentation fault is caused. This is not replicated when malloc is used for some other data types such as int, char or char*

C++ code:

#include <string>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
using namespace std;
void t(){
    string * data = (string*)malloc(100*sizeof(string));
    data[0] = "this is a test";
    cout<<data[0]<<endl;
}
PYBIND11_MODULE(TestModule, m){
    m.def("editPointerString", &t);
}

Once compiled and imported this can be run fine with editPointerString() as expected. No errors occur when running this function from any Python functions or from within a class.


However, if this is run through a unittest such as:


class TestUnit(unittest.TestCase):
    def testStringPointer(self):
        editPointerString()


A segmentation fault occurs before the cout<<data[0]<<endl; is executed. Not having an assert statement does not affect this outcome.


However, running 3 or more tests which call this function allows the tests to run and pass - instead displaying there has been a segmentation fault after the OK message from unittest

This can be fixed by using calloc instead of malloc, so may be caused by processing of data which is not cleared when malloc is used?

Also, malloc can be used in functions in tests after a test running calloc has been run.


System info:
Python 3.7.3
Ubuntu 19.04
pybind11 2.3.0
g++ 8.3.0
C++ 8.3.0
msg365039 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-26 00:41
The issue looks like a bug in third party code, rather than a bug in Python itself. You can try to set PYTHONMALLOC=debug environment variable, it might help to detect a memory corruption. I strongly advice you to test a debug build of Python. Since Python 3.8, the ABI is now compatible in debug and release builds.

See also https://pythondev.readthedocs.io/debug_tools.html to debug a crash.

I close the issue.
History
Date User Action Args
2020-03-26 00:41:14vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg365039

resolution: third party
stage: resolved
2019-09-06 10:11:18ymerejcreate