This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Leak of locks from multiprocessing.Process
Type: resource usage Stage:
Components: C API Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: YannickJadoul, bstaletic, davin, pitrou, shihai1991
Priority: normal Keywords:

Created on 2021-02-06 13:02 by bstaletic, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
log bstaletic, 2021-02-06 13:02 valgrind output
valgrind output.txt shihai1991, 2021-07-24 07:52
Messages (4)
msg386558 - (view) Author: Boris Staletic (bstaletic) Date: 2021-02-06 13:02
The following C code leaks 7 locks allocated with PyThread_allocate_lock:

#include <Python.h>

int main()
{
	Py_Initialize();
	PyObject* multiprocessing = PyImport_ImportModule("multiprocessing");
	PyObject* Process = PyObject_GetAttrString(multiprocessing, "Process");
	PyObject* args = PyTuple_New(0);
	PyObject* kw = PyDict_New();
	PyDict_SetItemString(kw, "target", Process);

	PyObject* p = PyObject_Call(Process, args, kw);
	PyObject* start = PyObject_GetAttrString(p, "start");
	PyObject* join = PyObject_GetAttrString(p, "join");

	PyObject_CallNoArgs(start);
	PyObject_CallNoArgs(join);

	Py_DECREF(join);
	Py_DECREF(start);
	Py_DECREF(p);
	Py_DECREF(kw);
	Py_DECREF(args);
	Py_DECREF(Process);
	Py_DECREF(multiprocessing);
	Py_Finalize();
}



The following locks are leaked:

1. https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L78
2. https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L84
3. https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L90
4. https://github.com/python/cpython/blob/master/Python/ceval.c#L810
5. https://github.com/python/cpython/blob/master/Python/import.c#L126
6. and 7. https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L597

In the attachment is valgrind's output.
msg386559 - (view) Author: Boris Staletic (bstaletic) Date: 2021-02-06 13:34
Slightly simpler C example:

#include <Python.h>

int main()
{
	Py_Initialize();
	PyObject* multiprocessing = PyImport_ImportModule("multiprocessing");
	PyObject* Process = PyObject_GetAttrString(multiprocessing, "Process");

	PyObject* p = PyObject_CallNoArgs(Process);
	PyObject* start = PyObject_GetAttrString(p, "start");
	PyObject* join = PyObject_GetAttrString(p, "join");

	Py_DECREF(PyObject_CallNoArgs(start));
	Py_DECREF(PyObject_CallNoArgs(join));

	Py_DECREF(join);
	Py_DECREF(start);
	Py_DECREF(p);
	Py_DECREF(Process);
	Py_DECREF(multiprocessing);
	Py_Finalize();
}
msg386774 - (view) Author: Boris Staletic (bstaletic) Date: 2021-02-10 11:20
The `multiprocessing.Process`, on Linux, ends up doing something like this:

pid = os.fork()
if pid == 0: os._exit()

Translated to C:

int main() {
    Py_Initialize();
    PyOS_BeforeFork();
    pid_t pid = fork();
    if (pid == 0) {
        PyOS_AfterFork_Child(); // Reinitializes stuff.
        _exit(0); // Child process exits without cleanup.
    }
    PyOS_AfterFork_Parent();
    Py_Finalize();
}

The call to `_exit()` happens in Lib/multiprocessing/popen_fork.py#L73

My attempts at cleaning this up resulted in even more problems.
msg398121 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2021-07-24 07:52
> The following locks are leaked:
> 1. https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L78
> 2. https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L84
> 3. https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L90
> 4. https://github.com/python/cpython/blob/master/Python/ceval.c#L810
> 5. https://github.com/python/cpython/blob/master/Python/import.c#L126
> 6. and 7. https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L597

Hi, Boris. Thanks for your report. We may receive the wrong alarm sometimes when we use valgrind.
 
1. https://github.com/python/cpython/blob/4512848ab92c8ed6aafb54d6e1908b1074558c43/Python/pystate.c#L78
   free in here: https://github.com/python/cpython/blob/196d4deaf4810a0bba75ba537dd40f2d71a5a634/Python/pystate.c#L127
2. https://github.com/python/cpython/blob/main/Python/ceval.c#L741
   free in here: https://github.com/python/cpython/blob/4512848ab92c8ed6aafb54d6e1908b1074558c43/Python/pystate.c#L375
3. https://github.com/python/cpython/blob/4512848ab92c8ed6aafb54d6e1908b1074558c43/Python/import.c#L124
4. free in here: https://github.com/python/cpython/blob/4512848ab92c8ed6aafb54d6e1908b1074558c43/Python/import.c#L156
5. https://github.com/python/cpython/blob/4512848ab92c8ed6aafb54d6e1908b1074558c43/Modules/_threadmodule.c#L613
   free in here: https://github.com/python/cpython/blob/4512848ab92c8ed6aafb54d6e1908b1074558c43/Modules/_threadmodule.c#L74
   To tihs refleak, I can't get the entire traceback of newobject. But I haven't get refleak when I run `./python -m test _test_multiprocessing -R 3:3 -vv`.
History
Date User Action Args
2022-04-11 14:59:41adminsetgithub: 87311
2021-07-24 07:52:25shihai1991setfiles: + valgrind output.txt
versions: + Python 3.11, - Python 3.9, Python 3.10
nosy: + shihai1991

messages: + msg398121
2021-02-12 21:58:27terry.reedysetnosy: + pitrou, davin
2021-02-10 11:20:06bstaleticsetmessages: + msg386774
2021-02-10 09:06:51gregory.p.smithsettitle: Leak of locks in a subprocess -> Leak of locks from multiprocessing.Process
2021-02-06 23:40:03YannickJadoulsetnosy: + YannickJadoul
2021-02-06 13:34:30bstaleticsetmessages: + msg386559
2021-02-06 13:02:37bstaleticcreate